Excuse me, that was almost thinking out loud writing, and probably the wrong context in which to put it.
I was thinking of a FIFO file to capture the stream from the hook point in charge of reading the block data when a new block inserts or block changes, and another FIFO file to point to the hook in charge of save block when it receives valid block data, for example. That is, before the API of the Developer Plugins layer…
On the other hand, being able to access the API from an external application would also be interesting. Isn’t it?
Yeah definitely and since most programming languages can call shell commands one could call a Logseq one, though it would seem a bit strange to call it from a code block inside Logseq itself.
I’d prefer libraries for popular languages and keep the code as much idiomatic as possible, don’t you think so?
Just for the reference (and future searching) - this idea is pretty much doing literate programming such as done in Emacs org-mode also (came here searching for that capability in logseq)…
I am thinking about this for a while now as well and interested in exploring this.
From the discussion, I see two directions:
Having a python API accessible from within logseq to access logseq itself. I think this could be done with Pyodide, loading the python interpreter within the logseq electron app and adapt to the logseq javascript API. I have tested pyodide in logseq and it works. Not sure though how hard the bridging is.
I suppose you could do the same of course with JavaScript directly, having code blocks of JavaScript that are evaluated on demand with access to the logseq API. Perhaps there is Clojure already exposed?
Connecting to a Jupyter Server and run code in a Jupyter kernel, basically replacing Jupyter Notebook with a logseq frontend. I have created an experimental plugin for that:
Still very very rudimentary though. Jupyter also has a lot of intricacies so this might be bumpy road ahead.
For both versions a lots of questions:
where to read code from? code sections? read children as well?
where to put the output? inline? child block? within a code section or plain text? replacing the code?
how to display various formats? HTML, Images, stdout and stderr (print), …
Maybe two options: render the output instead of the code (kinda like query results) and write the output in a specified block (the one below by default)?
More than a request this is supposed to be a space to discuss ideas. There is not a single solution but multiple approaches and very different levels of integration, supported languages and tools etc.
So I would like to gather here what people come up with, no matter what: just let “Logseq+Jupyter” inspire you.
There is work in progress to let plugins (and eventually user directly) exec native commands that are in a whitelist specified in config.edn.
In the attached videos you can see a code block whose content is passed to a native command.
Maybe we will be able to add a command like python in config.edn whitelist and run snippets written in code blocks, hopefully displaying the output in a block too.
There are a number of very different approaches that could be taken to implementing a feature like this. Choice of an approach has some quite consequential implications so this is something the Logseq community should probably give careful consideration if it is decided to move forward with something like this. Particularly what scope would be desirable for such a feature. Jupyter and Quatro/Rmarkdown take slightly different approaches to the problem. This article The First Notebook War - So Joel Grus doesn't like Jupyter notebooks. Here are some of my thoughts on notebooks, IDE, and R Markdown. - Yihui Xie | 谢益辉 by Yihui Xie the main developer of amongst other things knitr does a good job of contrasting them and pointing out the strengths and weakness on the ‘computational notebook’ more generally.
Jupyter
Jupyter’s approach is to have an external web server running jupyter hub to provide you with an interactive ipython REPL like session using Jupyter’s kernels for specific languages.
I view jupyter as more of an interactive scratch session for prototyping an playing around than a more serious document authoring tools or tool to perform reproducible computational analyses
In this approach the notebook is authored usually in some slight super-set of pandoc markdown in an IDE like RStudio or VScode with some plugins. Code blocks are executed in a session on the system’s install of whatever language your are using with the outputs displayed interactively in the editor.
This approach in my view scales better, by which I mean it is easier to transition from the prototyping and exploratory phase to more serious and larger scale projects either developing software or authoring technical of academic works without having to switch up your tooling.
The components in this approach are less tightly coupled than in the jupyter approach making things more modular and extensible.
WebAssembly (WASM)
Another way entirely could be to have the code you want to run compiled to web assembly and run ‘serverlessly’ with electron providing the runtime so you can keep it entirely self contained and not have any external runtimes. (see GitHub - quarto-ext/shinylive: Quarto extension to embed Shinylive for Python applications for and example of what can be done with this) Pyodide (mentioned above) also takes this approach.
A big gap in this thread so far is thinking about defining, preserving & making portable the compute environment in which any code is run. For a portable and reproducible analysis you need the data, the code and the compute environment to all be adequately defined. My current favourite tool for this is Renku https://renkulab.io/ an open source project from the swiss data science centre build on gitlab, kubernetes and docker which allows you to run RStudio, JupyterHub, or a full linux desktop from a docker container either in the cloud or locally and to store data in a shared knowledge graph.
Do we want to be able to come back to our code and re-run the analysis after making some tweaks? Do we want to freeze the output of a given run with out preserving the ability to re-run it keeping the results of running it at the time but not necessarily with the ability to re-run it add get the same results?
In addition to Renku Stencila https://stenci.la/ may be work watching their approach is very ambitious and they seem to have become less active lately but I like their interest in integrating with existing linked data formats like schema.org
Beyond the containerisation approaches which have certain weaknesses for computational reproducibility, as container image builds are not necessarily fully reproducible, the NIX & GUIX approach to package management will I think in the long run become the more popular approach to reproducible compute environments especially as people start taking software bill of materials more seriously for security reasons.
Thanks for the guidance. I’ve implemented your custom.js, custom.css and macro buttons to good effect and now have a number of custom calculators in my graphs without the need for plugins so they work on Desktop and Android.
I effectively now have Jupyter style functionailty built in including carryover of variable values from code block to code block.
