Logseq has issues when using Markdown (or Org) files as storage. To the point that the team is adding a “database version” that won’t use Markdown files, but the user will be able to export content as Markdown manually.
To my understanding, Logseq already run with a graph database that is updated in two ways:
By monitoring MD files in the graph folder, parsing them and updating the DB accordingly
While the user type the DB is updated and THEN changes are written to MD files
Is this the reason of all conflicts? The fact that the graph DB is updated in two different ways? And that MD files can be edited externally while Logseq still hasn’t written recent changes from the DB?
If so, why doesn’t Logseq just:
act as a Markdown editor: while you type the MD files are updated (automatically every few seconds or with a save button)
monitor the graph folder and update the graph DB periodically according to that only, no matter if the changes came from Logseq or external tool
What I mean is not updating the DB as the user type, because it’s what is happening now: if you edit the result of a query it could disappear while you are typing if it doesn’t match the query anymore. In my opinion it’s this type of responsiveness that is useless and does only damages.
I don’t care if things like autocompletition when typing [[ doesn’t show pages that I created a few moments ago. Same with query results, I don’t care if they don’t show blocks I just created.
I can see why Logseq has the architecture it has (it was meant to run in a browser) but why take a drastic decision like a database version, forcing the user to choose between basic stability and text files as storage?
I hope someone could provide a technical explanation and maybe @Ramses could forward this to the team.
P.S. I know a DB version enables block timestamps but for me it really isn’t something that justify giving up on Markdown files like this.
And for more structured data (blocks as values of properties) in standard Markdown I already proposed this:
I’m not super familiar with the logseq backend but it seems like it may make sense to first ship DB feature then markdown plain dir bisync. Joplin etc takes similar approach for indexings sake and for bi-links as you mention. I love the idea of a 1 to 1 map with plain markdown as it helps the PKM be “universal”
I think it also deserves mention that markdown editing is still important because the LogSeq editor isn’t good enough yet that external editors aren’t needed. For example when I want to move a header without moving the children (the header was inserted in the wrong place). And the “find in page” function finds blocks, but it doesn’t manage to locate them in the page (they have no context). And I always need to use another editor to clean up pasted content, because repetitive actions require macros.
It’s also a testament to LogSeq’s power and flexibility that other components can be used with it. It’s not necessarily because anything is wrong with LogSeq. Use the right tool for the job at hand.
The database version is still in the early days, we’re working on it because using Markdown files as the true facts brings some challenges:
Bad sync experience due to file system complexity, it can also be slow when renaming a page since all the files that refed this page will be updated and synced too.
Building proper and fast real-time collaboration on top of Markdown files is almost impossible.
Limited structure data support compared to db, for example, not every block has a persistent id, no timestamps, etc.
etc.
That said, we’d like to enable two-way sync between the database and Markdown files so that we can keep using our favorite tools such as Git/VS Code/Other Markdown editors.
Plain-text files for longevity, and database for freedom (extreme user experience)!
I am heavily using VSCodium for the REGEX Search & Replace in all files in Graph (for targeted refactoring actually). I don’t think Logseq will, even in the database version, become so complex as to allow refactoring in so many ways. I even changed the names of all pages containing spaces to names without spaces and replaced every Page Link pointing to the old name to mirror the new name. Re-Index in Logseq and done. (I did this because Logseq breaks functionality (Tooltip Preview, Shift_Click to open in Sidebar) if Commonmark Angle Bracketed Links are used -ex: [link text][<[[LogSeq Page Name with Spaces]]> , and I wanted to still use that Logseq functionality).
Also for static site generators there is a lot of modifying the actual md files that needs to be done so having those modification reflect back into Logseq DB would be crucial for some of us that don’t rely exclusively on Logseq.
So yeah, keep Markdown Compatibility and Markdown as First Class citizen in Logseq.
@tienson thank you but I am wondering if the Markdown version could be more reliable in the light of what I have said.
At the moment if I use a text editor to edit the Markdown files (with Logseq running in background) I have a more reliable experience. Logseq updates just a few moments after I hit the “Save” button in the text editor. And I can even use queries from the terminal using Babashka and logseq-query.
I am looking for an experience that looks more like file text editing, while all the other features that require a (graph) DB comes later and don’t interfere with the file editing.
Does this make sense?
About the database version, since exporting everything in Markdown can be tedious in certain cases, have you thought about exposing pages as virtual read-only files?
On Linux (and MacOS) there is FUSE for user-space virtual filesystems but I thought: what about a local WebDAV server?
AFAIK WebDAV is supported by Windows, MacOS and Linux: the user would be able to browse read-only Markdown files. This could be useful both for the database version and for query results in general. If it can help, I found this mininal WebDAV server written in Clojure.
This discussion was great to understand a bit more of Logseq and how to best use it, thank you @alex0 !
Expanding on the idea of “Logseq as Markdown editor”… The truth is that I’ve wondered as much. I even imagined that editor being (e.g.) a plugin for VSCode (which I guess would fit well with what @FlorianF commented?). That would surely make it easier to do things like opening multiple pages in any screen configuration one wants (vs the current right sidebar), work on WYSIWYG vs source mode, etc.
I’m assuming that Logseq-as-an-editor is needed (as opposed to some other Markdown editor) because Logseq would augment the rendered text with its functionalities (outliner, blocks, zooming, etc)… right?
And about the WebDAV auto-export, would that be doable as a plugin?
I really wonder what things will be like in the future. For now Logseq has been able to mold with my ever changing needs. I never had an app so flexible enough that it adapts (save for a physical notebook/bullet journal) with me.
So I always worry about things breaking or falling apart. Therefore always a few versions behind on my phone (main Logseq device) till I had a chance to fully test new versions on my PC or iPad.
Having said this, I do run into enough performance/memory problems that I hope a db version would help solve. (Or some other solution obviously), but I indeed wouldn’t want to lose the plain text files which means I will always have my data.
As a user of both Joplin and Logseq, I’m well aware of the performance gains that can come from the shift to a database-first app, and I’m very excited about this. How would the two-way sync you mention work, though? I do use git to sync now, which is convenient and great for manually resolving conflicts, but it would also be nice if a database version of Logseq had the kinds of syncing options that we have in Joplin, like built-in support for WebDAV, S3, OneDrive, etc. The database approach also allows for manually choosing which attachments to download on-demand, something that is great for saving space on mobile with Joplin.
Are there any plans to enable these kinds of sync options in the database version?
@Diego_Meza I am not from the Logseq team but by looking at GitHub commits it seems they want to make the Datascript DB persistent using SQLite (that to my knowledge can also be used as a graph DB, but I don’t know what’s the team intentions).
