Where can I learn how the plans are how proper synching of Logseq graphs in the future DB mode will work with a self hosted server (or git repo or cloud storage or file share …)?
With “proper” I mean synching that involves conflict resolution etc. Is there a design document or similar that explains it? Is this a planned user story at all?
1 Like
The conversation about this is happening here: Discord
The fact that all discussion on this is hosted on a proprietary chat app with legendarily bad searchability and a signup wall to see anything tells me everything I need to know about the outcome.
1 Like
The DB version is still under active development. It’s pretty normal for something that is yet-incomplete to not have full documentation and for those interested to have to pick up the pieces from various discussion sources. Proprietary communication apps exist, people use them and it’s fine. Whatever gets the job done. Discord is also one of the least-troubling communication platforms, in my view. If the dev team used, for example, a Matrix server or somesuch, it would be a larger barrier-of-entry for many and participation would be much, much lower. That’s probably all there is to that story.
Once the DB version is complete, then I’d expect full documentation and maybe efforts for self-hosting an sync server or whatever. For now, I think it’s still unreasable to expect all that. People are going to have to do some of their own hacking if they really want their own sync server and they should expect a lot of breakage at that.
@bitbonk People ask for a self-hosted option every other day on here. The answer is there isn’t one right now for the DB version. You’re much better off right now using the MD version and doing some kind of file sync (pick your favorite cloud file service, use syncthing for peer-to-peer goodness, or something else).
I’m doing this right now. And while the MD version notoriously has efficiency issues with larger graphs (I think more in the 10s of 1,000s of pages but some have complained after only a couple thousand), it’s the best one to use for rolling your own syncing method.
I, along with pretty much everyone else here, want there to be a self-hosted DB sync server option. But that doesn’t exist yet. And the DB version is still incomplete so I don’t expect we’ll get that yet. So the advice is to stick with the older MD version until the newer DB version is ready.
Speaking as a professional software engineer and long-term FLOSS proponent, I don’t get this at all. A development process four years along should not have its basic architectural decisions documented only in last-minute, ephemeral, disorganized chat rooms.
If that was all in Matrix, I would not be much better pleased. And the fact that that was your immediate alternative says a lot as well.
If everything was clearly laid out in GitHub’s (proprietary, yet accessible) tickets and planning and wiki, I wouldn’t mind so much, but as far as I can tell it’s basically just the unfiltered firehose of commit messages with no attempt to organize major points. (Until we get to changelogs, which are well-written, but a bit late for things like this.)
But hey, let’s not forget about the fact that this very forum has all we need for good results, since it’s not as formal as some big ‘ol “Design Documents” or “Architectural Design Records” or something. It does allow for some longer-term editing and collaboration and summarizing, and it has good searchability and visibility even to people with no accounts at all.
But no, important information (disappointing, and a regression from past realities and expectations) gets fragmented out to a stream-of-consciousness platform designed for constant fan engagement. Presumably the selection effect this has on the audience is entirely unintended, but it’s real and significant.
Just to be clear again: Discord has a signup wall to read information and makes it very difficult to search for past canonical sources. It’s focused on real-time interactive chatting, so if you have a question, you ask it. That’s great for avid fans who want to hear about this month’s DLC and the current metagame. It’s not so great for people who spend most of their time just using a tool and want it to keep working well for them stably and predictably. If you can’t see the structural differences that come from designing different platforms for different purposes (and yes, sure, the exploitative arbitration terms and so forth, too), then you should stop defending decisions you don’t understand.
Sync has been one of the most difficult parts of this entire long development process, and the early plans to open-source the sync server code are still very tentative and may never be completed. (They may never even be started.) If not, do you expect users to just independently reinvent that on their own and make it work with the internal bespoke protocols?
Financial pressure always veers away from open-sourcing key lock-in points like a hosted sync server, never toward that. (Usually this begins in a benign way, and only slowly becomes really community-hostile.) So if we don’t have one available now, and a business plan to support that, the smart money says we never will. If the business plan says “in 6-8 quarters maybe we can figure out a way to spend half our development effort for a few months on reducing our income to give back to the community”, then the smart money says that will never happen.
5 Likes
I’m not on the development team and I hope I didn’t give the impression that I speak for them. I don’t. I was simply saying that I understand why it is the way it is.
As for the DB version being FOUR years in the making, are you sure? They kind of decided to focus on that end of 2024, IIRC, and 2025 was the first full year of active development for it, with some of that work spilling naturally int the MD version as well. But I didn’t research their timeline deeply. I’m just going off of memory here.
I personally don’t think that developing a sync protocol first would be a terrible move. A full protocol spec could be done, I think, even before implementation. The last 20 years are full of good examples of sync protocols, many of which are open. Having that API up-front would probably allow others to build sync servers appropriately, etc.
I think the real difficulty they face is in the fact that they haven’t really figured out the database itself. What is the best way to structure the data for optimal DB performance AND that makes sense to the application user flows? I think that’s tricky. And maybe it’s just impossible to commit to a syncing protocol before that is all figured out.
One interesting thing when it comes to sync and especially UX for collaborative work on text files during different use cases, is the term CRDT I stumbled upon while learning about the Unwelling project, a protototype for collaborative text editing experience.
[[Conflict-free Replicated Data Type (CRDT)]]
You find more here:
Conflict-free replicated data type
url:: Conflict-free replicated data type - Wikipedia
tags:: #[[Automerge]], #[[Collaboration]], #[[Collaboration Tools]], #[[Collaborative Editing]], #[[Conflict-free Replicated Data Type (CRDT)]], #[[CRDT]], #[[Diff]], #[[DRDT]], #[[Editor]], #[[Framework]], #[[Ink and Switch]], #[[JavaScript]], #[[Non-fictional Writing]], #[[React]], #[[real-time collaboration with version control for writers]], #[[Suggested by Scott Jenson]], #[[Text Editor]], #[[TypeScript]], #[[Upwelling]], #[[Writing]]
abstractNote
In distributed computing, a conflict-free replicated data type (CRDT) is a data structure that is replicated across multiple computers in a network, with the following features:
The application can update any replica independently, concurrently and without coordinating with other replicas.
An algorithm (itself part of the data type) automatically resolves any inconsistencies that might occur.
Although replicas may have different state at any particular point in time, they are guaranteed to eventually converge.
The CRDT concept was formally defined in 2011 by Marc Shapiro, Nuno Preguiça, Carlos Baquero and Marek Zawirski. Development was initially motivated by collaborative text editing and mobile computing. CRDTs have also been used in online chat systems, online gambling, and in the SoundCloud audio distribution platform. The NoSQL distributed databases Redis, Riak and Cosmos DB have CRDT data types.
inkandswitch/upwelling-code
url:: GitHub - inkandswitch/upwelling-code: The Upwelling research prototype.
tags:: #[[Automerge]], #[[Collaboration]], #[[Collaboration Tools]], #[[Collaborative Editing]], #[[Conflict-free Replicated Data Type (CRDT)]], #[[CRDT]], #[[Diff]], #[[DRDT]], #[[Editor]], #[[Framework]], #[[Ink and Switch]], #[[JavaScript]], #[[Non-fictional Writing]], #[[React]], #[[real-time collaboration with version control for writers]], #[[Suggested by Scott Jenson]], #[[Text Editor]], #[[TypeScript]], #[[Upwelling]], #[[Writing]]
Upwelling: Combining real-time collaboration with version control for writers.
url:: Upwelling: Combining real-time collaboration with version control for writers.
tags:: #[[Automerge]], #[[Collaboration]], #[[Collaboration Tools]], #[[Collaborative Editing]], #[[Conflict-free Replicated Data Type (CRDT)]], #[[CRDT]], #[[Diff]], #[[DRDT]], #[[Editor]], #[[Framework]], #[[Ink and Switch]], #[[JavaScript]], #[[Non-fictional Writing]], #[[React]], #[[real-time collaboration with version control for writers]], #[[Suggested by Scott Jenson]], #[[Text Editor]], #[[TypeScript]], #[[Upwelling]], #[[Writing]]
-
abstractNote: Collaborative writing tools don’t work well for writers or editors. With Upwelling, we demonstrate a design that gives writers privacy while still offering editors transparency into how a document is changing.
Introducing Automerge 2.0
url:: Introducing Automerge 2.0
tags:: #[[Automerge]], #[[Collaboration]], #[[Collaboration Tools]], #[[Collaborative Editing]], #[[Conflict-free Replicated Data Type (CRDT)]], #[[CRDT]], #[[Diff]], #[[DRDT]], #[[Editor]], #[[Framework]], #[[Ink and Switch]], #[[JavaScript]], #[[Non-fictional Writing]], #[[React]], #[[real-time collaboration with version control for writers]], #[[Suggested by Scott Jenson]], #[[Text Editor]], #[[TypeScript]], #[[Upwelling]], #[[Writing]]
automerge/automerge
url:: GitHub - automerge/automerge: A JSON-like data structure (a CRDT) that can be modified concurrently by different users, and merged again automatically.
tags:: #[[Automerge]], #[[Collaboration]], #[[Collaboration Tools]], #[[Collaborative Editing]], #[[Conflict-free Replicated Data Type (CRDT)]], #[[CRDT]], #[[Diff]], #[[DRDT]], #[[Editor]], #[[Framework]], #[[Ink and Switch]], #[[JavaScript]], #[[Non-fictional Writing]], #[[React]], #[[real-time collaboration with version control for writers]], #[[Suggested by Scott Jenson]], #[[Text Editor]], #[[TypeScript]], #[[Upwelling]], #[[Writing]]
1 Like