Why the database version and how it's going?

[mentaloid]

@zizhuo

I want to know if I can find a specific text and replacement in DB version

In SQL-supporting databases it is possible to use some external client to execute a statement looking like this:

UPDATE blocks SET content = REPLACE(content, 'specific text', 'new text');

[tienson]

Find and replace will be built-in.

[tienson]

Afaik the DB version is much more stable than the current version because we’ve been focusing on avoiding any issue that can result in data invalidation, we’ll also try our best to reduce any issues that can lead to data loss.

[tienson]

Thanks for the suggestion!

This doesn’t work for the db version because the graph data is stored as key-value pairs (id->serialized node in a tree) in a table, the safe way to update contents might likes this:

;; Get all blocks
(def blocks-with-content
  (d/q
    '[:find ?b ?content
      :where [?b :block/content ?content]]
    db))

;; Replace "From" with "To"
(def tx-data
  (map (fn [block-id content]
         (let [new-content (clojure.string/replace content "From" "To")]
           [:db/add block-id :block/content new-content]))
    blocks-with-content))
(d/tranact! conn tx-data)

[mentaloid]

I think that @zizhuo’s question was twofold:

• Whether the database version will support replacing.
  • Which got a positive answer.
    • No good reason for not supporting it.
  • Actually Logseq could implement replacing even in its current version.
    • What editors like VScode do is relatively simple, though not always safe.
• Whether in the database version will still be possible to directly edit the storage.
  • This ability is an important feature to many users.
    • Especially those coming from text-files alternatives.
    • Their worry is about getting too much dependent on Logseq’s implementation choices.
  • If Logseq’s new database is SQL-compatible, there should be a way to edit it directly, even if:
    • it takes more effort to manipulate the serialized data
    • it is unsafe in potentially breaking Logseq’s assumptions
  • The point is that, as long as Logseq remains open, the possibilities are plenty.

[FlorianF]

with the current implementation, if I modify a template or a custom command I can’t have all templates in the markdown-based “db” updated accordingly (if even for an additional property I inserted in a template block). With VSCode it’s quite easy to do regex and to refactor any amount of complex find and replace stuff. I wish -but don’t hold my breath- that Logseq will have an easy way for users to do regex outside of advanced queries (like right in the Search - Ctrl+k - command or in some sort of visual query builder geared towards non devs…

[liuhancheng.cn]

@tienson

We’ll also extend invitations to a select group of users and companies to test our real-time collaboration feature once it’s ready for feedback.

Curious about who will be qualified or how you select “group of users”? Do the users have chances to apply for particapating?

[tienson]

Thanks for the question, we’ll start with our active contributors and sponsors.

[liuhancheng.cn]

Thanks for your quick reply!
I have no idea that whether i can be catorized as “actve contributors” (2 PRs, 1 merged and 1 WIP) or not, But as a heavy user of logseq, I really want to participate in early testing and am willing to explore and fix problems encountered during testing.

[tienson]

Thank you for your contributions!
Can you send me a direct message with your email address? I’ll add you to the testers when it’s ready.

[SpiderMatt]

I’m also very excited for the DB version as the improved stability will be very welcome. But as referenced in another comment, I’m a bit worried about lock-in. Will the new database be open source and allow for SQL queries? What’s the reason Logseq couldn’t go with a solution like SQLite?

Also, regarding sync, will that be one of the paywalled features that “rely on [Logseq] servers”? I’m wondering if there will be options for third-party syncing, possibly similar to how Joplin implements it (offering people the ability to host their own notes or pay to use Joplin Cloud). Or will people looking for free syncing have to stick with the markdown version?

[Jordan_Garrison]

@tienson

Are you going to deprecate Markdown files support?

• No, we’ll continue to support both file-based and database-based graphs, with a long-term goal of achieving seamless two-way sync between the database and markdown files. This will allow you to leverage the benefits of the database version while still being able to use other tools.

As a follow on to this, will this also be true for org file support?

[leod]

+1 for the same question on orgmode

[Zyrohex]

Great job! This is exciting stuff. Can’t wait to get my hands on and play with it.

[Federico_Frosini]

@tienson and the entire team.
Thanks for the hard work in the new DB version and thanks also for engaging with the community with this post.
Highly appreciated

[tienson]

All great questions, thanks!

I’m a bit worried about lock-in. Will the new database be open source and allow for SQL queries? What’s the reason Logseq couldn’t go with a solution like SQLite?

Our forked Datascript is at GitHub - logseq/datascript: Immutable database and Datalog query engine for Clojure, ClojureScript and JS, We added persistent storage support and other features, currently, the underlying data will be stored to SQLite through GitHub - sqlite/sqlite-wasm: SQLite Wasm conveniently wrapped as an ES Module..

Also, regarding sync, will that be one of the paywalled features that “rely on [Logseq] servers”? I’m wondering if there will be options for third-party syncing, possibly similar to how Joplin implements it (offering people the ability to host their own notes or pay to use Joplin Cloud). Or will people looking for free syncing have to stick with the markdown version?

The whole db can be exported as a SQLite file, which means you can still use third-party tools to sync. We have no plan to design a sync protocol so that the community can bring in other implementations because merging conflicts from other clients is very complex, but this could be changed in the future when RTC is stable and we have more resources to design a protocol.

[tienson]

@Jordan_Garrison

will this also be true for org file support?

@leod TBH we’ll not support org files in the beginning because we don’t have the resources to support both Markdown and Org Mode, our goal is to make the db version stable first.
I know it hurts because Logseq starts with Org mode and you guys come to Logseq because of me sharing the video/screenshots with Emacs
We’ll provide built-in support to convert Org files to Markdown files, and look for other ways to support Org, maybe through plugins.

[tienson]

Going to sleep soon, I’ll catch up tomorrow!
Thank you everyone!

[alex0]

This is very exciting. I hope that the multi-window mode will be more reliable with the DB version.