working with DataFrames as results

[ddev24]

Hello and thanks for contributing ob-julia.el.

ob-julia is using writecsv() to export data for importing into the org buffer. This works quiet well for most of the results.

When working with DataFrames the function writecsv() isn't optimal. The DataFrames Package provides an own write command: writetable().

My suggestion is to take care of DataFrames in variable org-babel-julia-write-object-command as follows (diff at the end):

(defvar org-babel-julia-write-object-command "wout(fn,o) = if try isa(typeof(o), Type{DataFrames.DataFrame}) catch; false end; DataFrames.writetable(fn,o); else; writecsv(fn,o); end; wout("%s",%s)")

try/catch will take care if Package DataFrames is already used or not. The function name wout should be optimized but I don't know which name is appropriate. An anonymous function would be even better.

Thanks and regards, Oliver

PS: Output compared (return of query() is of type DataFrame)

• with writecsv()

+begin_src elisp

(setq org-babel-julia-write-object-command "writecsv("%s",%s)")

+end_src

+BEGIN_SRC julia :results values :colnames yes

query("select * from test")

+END_SRC

+RESULTS:

| c1 | 1 | 2] |
|----+--------+--------|
| c2 | Hello | Hallo] |
| c3 | World! | Welt!] |

• With DataFrames.writetable()

+begin_src elisp

(setq org-babel-julia-write-object-command "wout(fn,o) = if try isa(typeof(o), Type{DataFrames.DataFrame}) catch; false end; DataFrames.writetable(fn,o); else; writecsv(fn,o); end; wout("%s",%s)")

+end_src

+BEGIN_SRC julia :results values :colnames yes

query("select * from test")

+END_SRC

+RESULTS:

| c1 | c2 | c3 |
|----+-------+--------|
| 1 | Hello | World! |
| 2 | Hallo | Welt! |

PPS: git diff

index 4d8deb2..5dc9f06 100644
--- a/share/emacs/elisp/org-8.2.10/contrib/lisp/ob-julia.el
+++ b/share/emacs/elisp/org-8.2.10/contrib/lisp/ob-julia.el
@@ -208,7 +208,7 @@ current code buffer."
(defvar org-babel-julia-eoe-indicator "print("org_babel_julia_eoe")")
(defvar org-babel-julia-eoe-output "org_babel_julia_eoe")

-(defvar org-babel-julia-write-object-command "writecsv("%s",%s)")
+(defvar org-babel-julia-write-object-command "wout(fn,o) = if try isa(typeof(o), Type{DataFrames.DataFrame}) catch; false end; DataFrames.writetable(fn,o); else; writecsv(fn,o); end; wout("%s",%s)")

;; The following was a very complicated write object command
;; The replacement needs to add error catching

[gjkerns]

Thank you for this. Sorry for the late reply; it's the last week of the semester and I am so busy I don't know what day it is. I haven't tested your code yet but from what I've quickly read above it looks great. I will come back to this when I get some minutes to test it out. Thanks again, and cheers.

[garrison]

My suggestion is to take care of DataFrames in variable org-babel-julia-write-object-command as follows (diff at the end):

(defvar org-babel-julia-write-object-command "wout(fn,o) = if try isa(typeof(o), Type{DataFrames.DataFrame}) catch; false end; DataFrames.writetable(fn,o); else; writecsv(fn,o); end; wout("%s",%s)")

Following a brief discussion on discourse, I believe the solution as specified is misguided. Julia is a language that supports (multiple) dispatch, and we should take advantage of this in forming a clean solution. The above solution makes the ob-julia code considerably more complex while supporting only a single additional data type. Here is my proposed solution:

1. Define a method writeorgmodebabel. This could, for instance, be in a new package called EmacsOrgmodeBabel.jl (or perhaps an existing package if there is a julia package for emacs support). The package should also implement this method for many julia Base data types. The entire CSV pass (as described here) should be eliminated in favor of outputting the format emacs expects directly. And then, ob-julia should be modified to instead call

using EmacsOrgmodeBabel
writeorgmodebabel(%s, begin
%s
end)

2. Any packages such as DataFrames.jl can REQUIRE EmacsOrgmodeBabel.jl and implement writeorgmodebabel for the data types in the package.

I believe this is a much cleaner solution.

EDIT: I think perhaps even better than having a specific package for this would be to have a more generic EditorSupport.jl package. This way, julia packages will not have to depend on N packages in order to support the special features of N editors (which they may be hesitant to do). But at the moment I am not aware of the existence of such a package, so perhaps no other editors have functionality similar to orgmode-babel? In any case, it would be good to ping some of the JuliaEditorSupport folks to see how they think this out to be structured.

[nico202]

Ehm, I'll vote against the module solution. What about "include"-ing the file defining the export functions? I usually change the JULIA_PKGS variable dir in order to keep project-specific Julia dependencies. If you want a module, else, we could keep it in Julia-ob and add it to LOAD_PATH

[nico202]

Oh, but this would be against the point 2. OK maybe having the package to include it would be better. Sorry

[gjkerns]

I haven't worked on this for some time, and you both clearly know more about this than I do. Would you like to take this and run with it? Godspeed, as far as I am concerned.

[nico202]

@garrison however, I don't expect many packages will do extra work in order to support us. Keeping code in this repo and implementing export functions ourselves seems to me just easier to maintain

[garrison]

@nico202 but on the other hand, people are often using old versions of orgmode (I myself am using a version from 2013). I think it is best if the code in ob-julia is as simple as possible such that the heavy-lifting is done in julia itself -- that way the functionality can evolve as packages decide to support babel output.

EDIT: I do agree that few packages will want to provide such support, but I still think the barriers to doing so should be as low as possible.

[garrison]

@gjkerns thank you for the offer. Mentioning here that you are no longer as interested in maintaining is one of the most important things an open source maintainer can do, so I'd like to encourage it as much as possible. On the other hand, I was hoping to make a drive-by contribution myself (after seeing the discourse thread) and am therefore hesitant to get too involved ...

[nico202]

Any news on this? It's boring that something like
#+BEGIN_SRC julia
println("ciao")
#+END_SRC

makes my julia crash and I have to add "0" or [] to make it work without reason

[gjkerns]

I am less bored today with non-Julia life than I was on Nov 10, and to reiterate: Godspeed.