Syncing Logseq DB

Questions & Help
,
1

Hello! I’ve been following the updates on the DB version here: Logseq DB - Changelog - #9 by danzu and I’m wondering if anybody has figured out a way to sync the files from a PC browser to a mobile browser.

From what I understand there are currently no desktop or mobile apps of the DB version and I was interested in trying things out.

I’ve been using Syncthing to sync the apps and it works pretty well to be honest.

4 Likes
2

Awaiting a better solution too, my current workflow when changing to another device is to export the latest DB graph to a synced folder (using Syncthing) and then import it on the other device.

When changing back I simply repeat the export/import. It works just fine if you don’t change over too many times a day…

2 Likes
3

I guess once there are apps we’ll be able to use Syncthing again easily.

Do you know if the sqlite database is a single file or multiple…?

4

I wouldn’t be so sure about that. How does SyncThing handle SQL files? Because looking at services like iCloud and Dropbox, every SQL file I actively use gets corrupted at some point because of those services. I can’t count on one hand how often my Calibre library got corrupted because of storing the SQL file in Dropbox…

Logseq DB will work with Logseq Sync and later also offer real-time collaboration. However, if you want to use tools like Git or SyncThing, I recommend you stick to Markdown/Org mode and not use DB mode.

The whole graph is stored in a single SQL file, with assets (images, PDFs, etc.) stored directly on disk like it’s now.

Correct, it can take one to a few months before the local DB apps are ready for testing. But see my explanation above (in this message); you’re better off sticking to Logseq MD if you want to use SyncThing as it’s likely not suited to sync SQL files. I could be wrong (I don’t use SyncThing myself), but please do some research (or have backups) before trying Logseq DB in combination with SyncThing.

2 Likes
5

Putting in another (admittedly Mac-centric) plug for the methods DevonThink uses for sync.
Bonjour over locally connected devices and Dropbox and iCloud for remote. They run their own sync from within the app on Dropbox, not just a database on a shared folder. I’m a user not a coder but it’s invisible to me when I use it.

There’s iCloud too but it’s very hit and miss and no one’s ever figured out the relationship of hit to miss, it seems to live behind an Apple wall of obfuscation.

I’ve run DT with iPhone as master of Bonjour syncs only to a laptop and desktop that rarely lived in the same geographic area. It’s all local syncs, no public server. Rock solid.

Sad to hear the GIT sync won’t work on the new DB version. I’ve learned a lot about GIT just by forcing myself to do manual LogSeq commits and pulls across four LogSeq graphs and watching what was happening before eventually leaving it up to the LogSeq apps. I’d be willing to give that up for what looks like some nice new features.

1 Like
6

I appreciate your reply very much! I have not tested syncing database files with Syncthing. From a quick search it doesn’t seem to be a very good idea/well supported.

Given all I have gained from using Logseq I am very much inclined to pay for Logseq Sync tho I usually prefer to keep control of my files myself.

I have a question about using Markdown mode: will that mode also get all the new “goodies” provided in the DB version (nodes, query builder, improved UI, table management)? So far I haven’t had any issues with Markdown mode other than a few bugs and improvements I’d love to see/ask for, but am waiting until after the DB version has shipped to bring them up because I understand that is the priority right now.

This also brings me to another important question. I seem to recall that there was mention of keeping both the MD and the DB version working. Are these sort of “separate” projects or do both stem from more or less the same code base? My intention in this question is to determine if, for example, some features will only be available in one version or another. I guess, given that the storage mode is different (MD files vs sqlite) some things will be different, but more or less to what extent. I also totally understand that this may be difficult to determine at this time.

7

Thank you :pray: Even with Sync, your data will always be saved locally (like it’s now). Of course, you have to trust our encryption as well.

The other modes will get some of the improvements, but many “goodies” like nodes, queries, and tables require block metadata that only the DB mode will support. Those features are the reason we’re building the DB mode, as they’re impossible to implement in the regular MD/Org modes (without heavily polluting the source files, and even then you won’t get the performance of a proper database we’re building now). You will also never get the performance of the DB mode in MD mode (unless you have just a few pages).

All modes share the same code base. The DB branch will be merged back into main—hopefully within H1 of this year. But as I mentioned above, many new features depend on block metadata, which will only be available in DB. Hence, certain new features will only work in DB mode, but we keep supporting the other modes (they’re in the same app/build, after all).

2 Likes
8

Thanks so much for taking the time to explain all of this. I’m really glad someone brought this up because I’ve been a bit timid about distracting the overall conversation with secondary points.

Could I just clarify; it sounds like there won’t be any real offline sync option as they’re currently is with the new DB version.

My use case is that I run sync thing on all my devices and obviously I’m using the MD version.
For me that’s quite important because internet signal where I live is mediocre at best, and at times intermittent.
One of the benefits of logseq for me was that so long as both devices were connected to the same WiFi my internet speed and reliability was irrelevant.
I’ve even found that running a hotspot off of my mobile despite having no cell service is sufficient for Syncthing to synchronize between my laptop and my phone.

I’m guessing though from what you said the DB version won’t play well with that at all. Obviously Syncthing can’t synchronize an SQL database so does that mean local sync as I currently use it is basically off the table unless I find a way to do it myself? (Which I’m not sure is possible)

P.S. I’m not saying it has to be charged for little old me I just want to know where I stand at the moment.

Edit: Forgot to say thank you :kissing_heart:.

1 Like
9

Thank you for your kind words @Thorah :pray:

I’ll be straight to the point: in your situation I’d just stick with Logseq Markdown.

All other modes (MD and Org) of Logseq have gotten many bug fixes, which will be available once we merge Logseq DB into the main GitHub branch. And while it will lack some features that require a proper database, it will get the features that don’t depend on them (like the improved flashcards feature/algorithm and down the line a better mobile app and improved whiteboards)

If you want to fully own your syncing between devices, especially for peer-to-peer, you should just stick to plain text files. Who knows what will later be possible in terms of syncing Logseq DB, but for now Logseq Sync for DB will mean that your notes are encrypted and then sent to our servers (as it’s currently with Logseq Sync for Markdown and Org files).

I don’t think there are plans yet to support P2P sync, but that is a great idea actually. I’m not sure how technically feasible it would be for us to build, but I will definitely discuss your feedback/situation with the developers.

1 Like
10

Isn’t Logseq Sync a paid service? Does this mean we will have to subscribe to sync the DB version of Logseq? Doesn’t this kind of creep into the terratory of making Logseq tiered software?

Surely there can be something for non-subscribers who still want to use Logseq DB? Maybe a homelab sync server or the like in the future?

11

Logseq Sync will indeed be part of a subscription (currently it requires a donation, but we haven’t set a fixed price). But as I explained above, we will keep supporting and developing the plaintext versions/modes of Logseq, which work with any regular file sync solution.

We’re merely running into technical limitations using plain text as permanent storage (see this announcement for more background), so we’re adding a mode that uses a proper database as permanent storage. This will significantly improve performance of the local app, as well improve the performance of Logseq Sync and will enable real-time collaboration (for those who need it).

The trade-off of using a database file for storage is indeed that you can’t use a class of sync services created for syncing regular text and multimedia files (not binary files).

I’m not sure what you mean by “tiered software” in this context. Can you please explain?

You will have to find a tool/service that lets you sync binary files or is specifically made with .sqlite files in mind.

I don’t know how much you know about software architecture, but there are always trade-offs. In this case, the trade-off of Logseq DB is that we have to use a database/binary file to enable a certain set of features and provide a better syncing experience to those using Logseq Sync.

I want to emphasize that we’re not purposefully creating a lock-in. We’ve made the decision (after talking to hundreds, possibly thousands of users) to offer a version/mode of Logseq (along the existing, plain text, ones) for the people who need more performance, more reliable syncing, real-time collaboration—and are willing to make Logseq a sustainable project by paying for services like that.

If you don’t want to contribute to Logseq in a financial way (or you just don’t want to make use of cloud services, which is a fair point), you will have two choices:

  1. Use Logseq DB and move around backups manually, or just use your DB graph on a single device.
  2. Keep using the plain text modes of Logseq and use any off-the-shelf syncing solution (we always recommend Git and SyncThing over any other solution).

Hopefully over time there will also be third-party solutions that offer syncing of .sqlite files like SyncThing does. But this is tough technical challenge, because of which I think there aren’t any yet. I’ve seen some early signs that there are people working on such solutions, so feel free to reach out to me if you know more about this topic (not just you, but anyone).

I don’t want to speak for @tienson, but IIRC I’ve heard him mention that everything is/will be open source. So if you want to run your own sync service for yourself, you will be able to.

But most of our users aren’t that technical and just want a reliable, performant solution. And they want to see Logseq around for years to come, so they’re okay with paying for a service that makes their life easier.

If that’s not you (because you can’t afford it, don’t want to spend money, or simply don’t want to use hosted cloud services), that’s fine. Just realize that everything in software has a trade-off, and these are the trade-offs for Logseq DB.

I hope this makes sense. Let me know if you anything is unclear or you have other questions.

1 Like
12
  • Hello, many thanks to the developers for logseq.
  • The advantages of logseq-DB over plaintext files, is very likely to make me switch to it. My understanding is that, its value is not only speed, but also the fact that it will enforce template-structures and types which in turn will facilitate queries.
  • You would have certainly thought through about syncing way much more than I what I want to say. So i say sorry in advance if what I say has been considered.
  • just penning this idea, if it could be made to work.
  • The first choice you mentioned above “1a : Use Logseq DB and move around backups manually.” . This would require backing up and restoring each time. It also requires the user to be careful about what is backed up and where it it being restored. Backing up and restoring could be tedious compared to what I describe below.
  • I was hopeful something like the below would be possible. Using logseq-db from android and a Home-PC and a Work-PC.
  • What would be sufficient to some users like myself who use it as a single user without need for multi-user collaboration would be to have like cooperative network-filesystem file-locks respecting logseqDB-clients and switch-overs.
  • Assumptions
    • 1 Only one user is using the logseqDB client on a device for a stretches of time, but after period of use on one device does move to another. So the same user, uses multiple devices sequentially
    • 2 One does not need concurrent-use / simultaneous/ real-time data syncing.
    • 3 It is sufficient if one can serially switch between devices, by taking the responsibility to unlock logseqdb client from one device and acquire lock from another, when one moves. Home ↔ on the Go ↔ Work.
    • 4 A user is self-cooperative and isn’t going to maliciously cause the locking system to break
    • 5 The speed of unlock/release acquire/relock does not have to be quick as these are done at the pace of a human user and can take 20 sec if need be for raising surity. Not necessary to raise the software guarantees for guarding against races/hazards. If user intends to move from device-1 to device-2 let’s not worry about device-3.
  • So a user on a client, when wanting to switch from using a logseqDB client on one device to another device, could trigger a yield or be timed-out after 10 minutes of inactivity. This would release and unlock the db from one logseq-client.
  • If a logseqDB client on a device determines that it cannot acquire a lock because another client on another device is using it, then it should inform the user which device has the lock, so the user can ensure the lock is released from that device.
  • If a logseqDB client acquired a lock and then lost network connection, It is fair to expect that a user must be directed to find the device that has the lock, reconnect to the internet and unlock it.
  • If a user has to break the lock on an unavailable and internet-offline device, user owns the corruption/data-loss consequences.
  • A logeqdb client should release the file-lock, once it has confirmed that all changes it made to the sqlite file have been synced to the network storage.
  • A user must not manually try to tamper with the lock files on the network drive, and this lock/unlock is done by the logseqDB client.
13

i understand the technical concerns about db corruption but this still could be considered a lock-in. by making it inconvenient and having to move files manually, export/import.
so… a soft lock-in?

IF you truly want to make it possible, i believe it is possible

  1. copy to local storage: before making changes, copy the database file from the synced folder to a local, non-synced directory
  2. work on local copy: make changes to this local copy
  3. copy back: once changes are complete (auto-save or a button for force copy), verify database integrity immediately after writing files, copy the updated file back to the synced folder

collaboration would still not be possible but people who want this are mostly single users

i understand why you would not want this, it would be one less reason to sell Logseq Sync

while not a hard lock-in, this could be safely considered a soft lock-in or business-driven lock-in (“you must use our paid service to make it more convenient”)

15

Selling subscriptions is not the reason to also offer Logseq DB. I recommend you read the original announcement, as it explains the reason for adding a mode (next to Markdown and Org).

I agree that it creates a type of (unintentional) lock-in. But as with most things in software, it’s a trade-off some people are willing to make. Those who don’t want to make the trade-off of storing their notes in a SQLite file, should just stick to using Logseq with Markdown or Org files.

The tools you link to look interesting for syncing SQLite files. I hope the community will experiment with them and document how to properly use these tools.

17

Apps like Bear and Drafts use SQL databases and sync seamlessly over iCloud, but they support CloudKit, and unfortunately I’ve been told that the DB version of Logseq won’t.

The latest versions of macOS and iOS have a Keep Downloaded feature that I found makes syncing classic .md Logseq over iCloud noticeably better but still glitchy sometimes. Idk whether Keep Downloaded will help with the DB version or not.

18

Having done some digging I’ve found this which might be of help.

I guess though that it would need a license by you guys to use but I might be wrong so I thought I’d throw it out there.

Licensing Page

19
  • In my previous post, I had written up “lock-file” mechanism to prevent simultaneous use.
  • Pinto, in the post after mine, mentioned a copy a copy-back strategy.
  • In this post, combining the two ideas, I want to pen a “read-only mode” that while dbfile is file-locked, the following is possible
    • copy dbfile, open logseq graph in read-only mode, copy-back after use
  • The read-only mode should allow for
    • (1) browsing pages/blocks/graphs in the logseq graph without beng able to edit anything
    • (2) doing queries and getting answers.
    • (3) also, when in the mode presents the info as to which device has the dblock
  • Needless to say, it would be a convinience to be able to consult and refer to ones notes even when the lock is held on another device.
20
  • correction in above:
    • the copy-back after use, would not be required for read-only mode.