Advanced queries: rant

Feedback
1

Sorry, I have to rant a little. I really love Logseq, and I have no mentality of entitlement towards a free open source project, but when I’m passionate about a project, I also get frustrated more easily when I have a bad experience.

— rant on —

I looked at a lot of logseq datalog/advanced query videos and posts, I don’t want to offend anyone, but the quality is often subpar. It’s assumed that people already know the database schema, query rules, etc. Maybe they don’t know themselves sometimes. Or if they know, they just link to this which is - to put it mildly - not very beginner friendly.

Even the official docs, sorry, but not good enough. What is :block/name? What is page-ref? Where does it come from? Why did it break with version x? What else is possible?

Maybe I come around to understanding queries myself well enough to explain them some day, but what’s currently out there is not at all sufficient to have an easy start. I think queries are super important to get the most out of Logseq and I hope there will either be a complete overhaul of the query system or really good documentation that enables newcomers to generate fancy queries for their Logseq graph.

— rant off —

If someone of you is really good at advanced queries: Please do a tutorial where you go through the queries line by line. Show where each element comes from. Where you can see it in the database. How the database looks like in the background with a live example.

2

I have found https://qwxlea.github.io/#/page/datalog%2Fintro%20to%20datalog quite useful in understanding data log queries.

4 Likes
3

I agree with your rant. Good news is I do think Logseq team is aware of this. First it is amazing we can do such advanced queries. Basically, we can talk to the database at its lowest level.

I am a programmer but have no desire to spend hours and hours learning the query language just for Logseq.

What would be helpful is a query builder that exposes advanced query capabilities.

7 Likes
4

This is excellent! Tnx for pointing it out.

5

I too found the advanced queries difficult so built an online tool and a plugin to simplify building queries

Checkout Logseq Query Builder - an online tool to build queries from simple commands. It generates the advanced query code for you.

I also built a logseq plugin which is available at GitHub - adxsoft/logseq-query-builder-plugin: A logseq plugin that generates advanced queries from simple commands . Note it has to be manually installed at the moment. I have submitted it for inclusion in the marketplace.

Documentation is at GitHub - adxsoft/docs-logseq-query-builder-plugin

Video demo is at Logseq Query Builder Plugin Demo - YouTube

6 Likes
6

As a professional software developer, I did have some fun learning Datatomic and advanced queries and find them quite powerful. However I do admit it is not easy, and I can’t imagine how hard it would be for a non-technical person.

For me personally, the hardest part was figuring out how to debug the query when it wasn’t doing what I expected it to. I will share below some tips that helped me with that:

  • Use KLIPSE to try out snippets of clojure code on the browser.
  • double check on the database schema, DSL rules code and clojure core functions docs
  • Add :view :pprint to view the query results as text
  • Turn on developer mode (Settings -> Advanced -> Developer mode) so that I can right click on a bullet and (Dev) Show block data. That way I can see what specific blocks look like in the db, which helps writing the query.
  • When doing a custom :view, one may use [:div [:pre.code (pprint variable)]] or (log variable) to see what is going on
  • Check on this forum and on Discord if something similar to what you want to do has already been asked about. If still stuck, then ask a question yourself.
3 Likes
7

This is a great tool and also for learning! Tnx for building it.

8

Great tips! thank you

1 Like
9

I would be interested why Clojure was chosen as a query language — is it how Logseq queries work internally, and that is exposed to the user?

Because to me, a visual query builder is a great, but i’d rather have something simple and text-based like the SQL-ish Obsidian Dataview: I would argue it is readable (and therefore: hackable) by most people.

1 Like
10

Yes, to my understanding the syntax comes from Datomic, that is an implementation in Clojure of Datalog query language, but Logseq uses a reimplementation that runs in the browser called DataScript.

Of course Advanced Queries are an additional feature for advanced users, there are already Simple Queries and the devs are interested in shipping a graphical query builder in the future, you can find a thread in this forum.

It’s a really cool experiment to let the user interact with the DB with the same query language the application uses internally.

You can do virtually everything with Advanced Queries if you have the skills and some time. You can even use Hiccup syntax to put results in HTML code. Basically you are remodeling the application.

Blurring the line between users and developers is a great idea to exponentially grow an application as a platform.

At the moment you can hack Logseq as a user by installing plugins, using templates, macros or in general combining the many features that are quite general and don’t impose a specific workflow.

You can hack it as a Web developer with custom CSS/JS, Hiccup/HTML or developing themes and plugins.

You can hack it as a Clojure developer of course but also by manipulating the Markdown/Org files programmatically or interacting with Logseq through its HTTP API (reading/writing JSON with whatever language).

Other applications claim to be “the everything OS” and stuff like that but the only one that could say such a thing is Logseq.

How cool is this?

I wouldn’t say SQL is simple, I’d say it’s better to have a graphical query builder, Simple Queries that already ships their own very simple language and Advanced Queries. In the future we could even have Machine Learning turning natural language into queries.

I tried Dataview plugin for Obsidian and personally I didn’t like it. I didn’t know Datalog at all before discovering Logseq and I am very happy to know that better alternatives to SQL exist and this one seems particularly useful with a graph database.

3 Likes
11

Thanks for you thoughts!

I wouldn’t say SQL is simple, I’d say it’s better to have a graphical query builder

Probably yes, all depends on how good the query builder UX is :slight_smile:
e.g. it would have to be a query editor (not just creating a query but also able to edit existing ones).

And having all those different input methods (simple queries, advanced queries and query builder) with different (visual or non-visual) languagues is confusing to me — should i “learn” simple queries, or when do i need advanced queries? Does the query builder do everything the advanced queries do?

12

Check my concept:

To my understanding, the Query Builder will be a graphical way to edit an Advanced Query.

Everything it’s impossible since an Advanced Query is based to a very powerful and expressive query language, it can take advantage of Clojure functions, transform results and style them with Hiccup/HTML. But I think it is supposed to be much more powerful than Simple Queries.

For now learn Simple Queries and ask for help with Advanced Queries, someone more proficient could write one for you and you can learn how to tweak it. Properly learning Advanced Queries is comparable to learning a programming language.

I don’t think a graphical Query Builder and a ML-powered natural language converter are too far into the future, hopefully some months. I tried to ask ChatGPT for Simple Queries and it worked very well.

13

Hi. See https://mschmidtkorth.github.io/logseq-msk-docs/#/page/queries%2Fadvanced%20queries%2Ftutorial for a couple of examples in that style. I’ve split up additional links into beginner vs advanced users to help beginner users find relevant tutorials - https://docs.logseq.com/#/page/advanced%20queries/block/additional%20links

3 Likes
14

As a non-coder my reply to this is…whaaaaaaat?

I seriously have no clue what a single one of those bullet points means and wouldn’rt even know where to begin :frowning:

Not a dig at you.

2 Likes
15

Advanced Queries will always be developer-oriented, instead the team is interested in providing a visual query builder for all the other users.

16

Nope, definitely it is not simple, but it can be. And it is for sure more natural language like then the advanced (or even simple) queries in Logseq. That is a very big plus for SQL and I think the biggest plus for SQL having done a lot of SQL work.

This is exactly the reason where I think Logseq is slowly becoming too technical for the ‘normal’ user. The thought that developers want to code everywhere in there live is (at least I think) is wrong, with some exceptions excluded off-course for people who self-write code to automate their coffee machine.

It is important that the language to query the data in Logseq should be logical, natural, explanatory with no need for non-natural language characters. I see at least in the forum a lot of extremely technical ways of doing things mentioned and also a focus on the most complex things that should be possble. Just look at the combine pages and blocks discussion. I think very few people read it all the way through. I hope the development teams keeps the simplicity (after all it is only a note-taking app right) in mind. Thank you.

1 Like
17

I mean it’s nothing the average user would use.

This is really a personal preference, I find SQL one of the worst thing ever in computing in general, while Datalog (or at least Datomic as implemented in Logseq) seems way more natural for me. For sure it’s way more powerful than SQL that is for relational databases, while here we have a graph database, that is way more flexible.

Maybe it’s not clear: Logseq is not going to introduce a whole new query language as powerful as Datalog. Being able to use Datalog is a feature like it’s a feature to use HTML or Markdown.

Logseq already provides a so called Domain-Specific Language (DSL), that is what we call Simple Queries.

Simple Queries are more or less macros for Datalog queries that cover very simple use cases.

Advanced queries will always be possible because it’s very powerful to be able to write as user the same queries Logseq uses internally, transform the results using Clojure functions and style the results as HTML via Hiccup syntax. Basically Advanced queries let you remodel Logseq.

Of course this is not what Logseq is supposed to be for average users. For them there will be other solutions, but it will takes more time. Simple Queries are already there as a patch and probably there will be a visual query builder and maybe even a Machine Learning-powered function that turn natural language into Datalog queries.

Logseq is not becoming too technical, Logseq started with many developer-oriented features and it aims to be usable also for the average users. In the meanwhile the advanced features are there because they were easier to implement and unlock a lot of power for skilled users. Those can also help other users by teaching tricks or even provide Advanced Queries on-demand, as it often happen on #queries channel on Discord.

18

These tools really are super. A big help.

I do have a couple of requests though. It does not seem possible to sort with your tools. Being able to sort by date added/modified would help me a huge amount.
Secondly, selecting only pages written within a range of dates would also help me a great deal.

If these could be added, I think I would suddenly be able to do everything I need to with your tools rather than just a fraction like now.

19
  1. Sorting

Tried adding sortby command but the result-transform function does not seem to work consistently from my testing. Theres no documentation about how result-transform works so reluctantly I left sorting out. I will include it once I can determine a generic code pattern from trawling through enough examples.

PS You can sort in table view by clicking columns

For created date sorting you can add these lines manually

ASCENDING ORDER
:result-transform (fn [result]
(sort-by (fn [h]
(get-in h [:block/created-at]) result)))

DESCENDING ORDER
:result-transform (fn [result]
(reverse(sort-by (fn [h]
(get-in h [:block/created-at])) result)))
:breadcrumb-show? false

  1. Pages between dates

Good idea … I’ll add it in the next release

1 Like
20

Sorting by :block/created-at in many cases is not meaningful because that date comes from the file’s metadata which comes with 2 problems:

  • Not all pages have an actually file created. The file only gets created when you actually open the page and write something on it. If you only created a page-ref, then there is no file automatically created. That means that for pages without files, the created date will be reset every time you re-open Logseq.
  • If you use Git, none of the files’ metadata get included in the repo, since git only version file contents, not metadata.

Because of that, what is more meaningful in my opinion is the journal date of when something was first mentioned.

1 Like