Proposal: Changing How Namespaces Function in Logseq

Feedback Feature Requests
19

I would also like to see subfolders, for the following technical reasons:

  • to avoid long pathnames as brought up by the OP. There are ways to enable long pathnames on different OSs, but some programs do not support them and might fail in unexpected ways later on, files might get moved to an even deeper hierarchy etc.
  • to avoid too many files in one folder.
    • NTFS limit very high, but substantial slowdowns happen even with a few thousand files
    • FAT32 (USB drives) has a limit of 65k files. A number that can be reached easily
  • to share different subfolders, as suggested by @fredguth and @Vincer
  • to interface with other software through automatic generation of .md files in a watched folder

On the other hand, I don’t think subfolders hierarchies are a good way to make information accessible, because one easily runs into many problems, as discussed in an article.

I just wrote a feature request for hierarchical tags, which have many advantages over subfolders, including the ability for one item to be part of multiple hierarchies and no limit on the path length or nesting limit.

From that point of view, I think the subfolder system should be optimized so that the storage issues (no long pathnames and not too many files in one folder) are addressed first, and it should be kept simple enough to make interfacing with external programs easy. If folders serve as an extra tool for organization - that is great, but I wouldn’t sacrifice speed and stability.

Calibre let’s one pick different export formats to convert the internal folder structure, such as {author_sort}/{title}/{title} - {authors}. Logseq could also maintain a set of hardlinks to present a different view for external programs.

6 Likes
20

In addition to all these good suggestions, Aryan’s overview on the usage of namespaces was hugely helpful to me.

21

Folder structure is needed to create a content manager in Logseq

The folder.md should list also inside files (at least first child)

22

I discovered that I can move .md files in subfolders in /pages or other subfolders of the graph. Logseq use them like they are in the same folder.

So the folder structure in the file system adds another hierarchy that is independent from the others and Logseq is not aware of it.

For me this is another degree of freedom, I like it.

But I would like if Logseq let me choose in which folder a new page is created when I click “New Page”.

And I would also like a full-featured file manager instead of “All Pages” section, where I could move, rename etc the files.

3 Likes
23

See also Support subdirs for namespace hierarchy - I tried to simplify and clarify this feature request down to just “support subdirs as namespaces”.

24

Yes. Our vision is to separate namespace structure from the limited tree structure of file system.
Namespace is for concept disambiguation. Creating sub-dir by default would imply that every page should have a namespace (sub-dir), which violates our design.
Besides, the namespace file naming is going to be changed in Feat: new file name escaping rules by cnrpman · Pull Request #6134 · logseq/logseq · GitHub

@Gary_O title:: property is providing a “manual” way to manipulate the namespace structure

4 Likes
26

Our vision is to separate namespace structure from the limited tree structure of file system.

I’m utterly glad to hear that! :heart:

27

I think that if we could have flexible hierarchies like Dendron that support Dendron-style tree-refactoring, that would be UH-MA-ZING!!!

1 Like
28

I hope at some point we will have a way to browse “virtual file systems” by specify the criteria for creating the hierarchy, for example:

{{hierarchy <property-A> <property-B> ... }}

so that

{{hierarchy year author}}

would be rendered as

- [[2022]]
  - [[Alice]]
    - blocks with year:: 2022 and author:: Alice as properties
    - ...
  - [[Bob]]
    - ...
- [[2021]]
  - [[Alice]]
    - ...
  - [[Bob]]
    - ...
6 Likes
29

@Junyi_Du There are all kinds of idiosyncrasies of different OS/Filesystem combinations. Many programs have problems handling all path variations and might crash, fail in obscure ways, silently ignore files, or just slow down tremendously due to O(n²) runtime issues.

Unfortunately I am no expert on this, but I would strongly suggest to check with one and find a well-defined path spec that is guaranteed to work everywhere.

Any flat storage folder will very quickly lead to problems with too many files. On the Zotero forum, people are discussing libraries with more than 100k items.

One of the strength of Logseq is integration with other programs, and integrity of its storage should be the first priority. This includes making sure that Logseq doesn’t create folders with too many items.

30

Directory structure can still be adapted manually. But the hierarchy is ignored in the page title.
Binding namespace hierarchy to the file structure is not a good way to resolve the File System performance issue you mentioned. Instead, it make decoupling File System implementation with the abstract page structure impossible.

2 Likes
31

I agree with you, I wasn’t suggesting to bind namespace hierarchy to the file structure. This is the right decision.

My concern is that some strange filepath issues might lead to eventual inconsistencies (e.g. I had issues in Python, because of a difference how the standard library interprets file paths differently than Windows).

The second worry is that folders might have too many files, e.g. if someone reasonably imports a Zotero library with 10k items and 10k tags we suddenly have 20k files in a single folder. Theoretically this is only an issue with FAT32 (memory sticks), but many programs struggle with folders that have more than a few thousand items. Networked folders could also be an issue.

2 Likes
32

  1. Dealing with filepath on multiple OS is inevitable for plain txt storage, even without the current namespace design. There are tons of reserved characters / FS API special handling across different platforms.

  2. I can understand the concern. Some shard design could be adopted when it hits the performance ceiling since the FS is decoupled :slight_smile:

4 Likes
33

Great! You already thought about the issue. Just wanted to make sure it doesn’t get overlooked. Thanks!

1 Like
35

I’m not sure if this has moved to triage by developers as it’s own feature or part of a different one, perhaps the tag approach for polyhierarchy mentioned in @gax’s feature request, link removed from quote as I’m new

Is there a more active topic where I can find more updated information/discussion on how others are using namespaces in their current state to meet flexible polyhierarchy needs by @alex0’s “virtual filetree” and how an eventual feature may appear?

I know there’s some continued discussion on meeting some of the needs of generic structure via queries on this discussion

36

Welcome. Namespaces in their current state cannot provide flexible polyhierarchies (and I doubt they will ever do). Consider following Generate explicit hierarchy out of properties

1 Like
37

@Throwback6873 what Mentaloid said and check also Favourites tree and Hierarchy jump plugins. They let you extend the namespace hierarchy using a property, enabling polyhierarchies.

38

Would you mind elaborating as to what’s too limited about a file tree structure?

More to why I’m asking:
I get that filenames including emoji’s or other non-ascii characters (allthough utf-8 being quite widely supported these days) can be problematic, but that doesn’t seem to outweigh the advantage of being able to have the knowledge base structured on the fs level. I fail to understand I even have to explain the advantages, I guess I’m not young anymore. I’ve switched from Obsidian to logseq because it’s open source, but I’m really considering to go back since this whole namespace thing hinders the consumption of the content as you’re seeing the same information over and over again. Eg. in a technical documentation I’m describing how various behaviors have their effects on code and enforce clean separation of concerns, seeing the word Behavior highlighted inside a single page over 20 times because it’s the namespace parent of eg. Timestampable, just gives me headaches after a while. I want to use the logseq namespacing stragegy but it just ignores my carefully organized and git tracked collection of content.

Meanwhile I’ve seen how extensible logseq really is as there is already a git repo out there that addresses this and other tweaks very nicely, but if the logseq team insists on this feature being an external project instead of taking advantage of the fs without very good reasons - I’ve seen none so far - makes me kind of doubt if I should keep using it.

I like to read markdown / copy paste it with formatting into eg. an email, from for instance Gitlab or other web-based platforms providing markdown parsing (including mermaid) but with the current namespace implementation this is a pain as I have to manually update all [[]] links to traditional style markdown link syntax . The fact that Obsidian does this out of the box, automatically recognizing nested files whilst maintaining a title-only rendering and usage, blew my mind and got me really motivated to continue writing documentation about complex topics. Seeing the graph grow seemingly out of nothing is a great motivational factor for me.

Call me old or stupid, I don’t care, but as a markdown lover, working with logseq namespaces has become a serious downer for me: it gets me out of my writing flow as it’s just distracting and objectively harder to read over the repeated highlighted parent namespaces, doesn’t reflect any structure in any other markdown viewer, refuses to place new documents into folders that already exist on the fs, creates these weird looking filenames with double __, just to name a few… This has become a real vendor lock-in and I oppose that wholeheartedly.

This very topic already contains great arguments as to why this decision of the team is hard to defend, so I’m really curious to how your vision came to fruition. I hope I can be enlightened instead of becoming more and more disappointed as hierarchy means a lot to me and carefully crafted directory structures are a thing, just look at how your average software project is structured…

I’m obviously not talking about repositories with 10k files etc as I’m mostly using this for personal or private topics. If I ever have to deal with that kind of large amounts of data I sure hope I have a better tool than an “always deprecated” electron app anyways. I strongly believe the focus should remain on Markdown, the content and the act of writing. Reinventing a well established concept like hierarchy really brings more pain than gain at this point…

Look, if there were technical reasons preventing this from already being a built-in feature of logseq, I’d be more than happy to wait, but if the team has already decided that this will never be a thing, you’ve lost a once enthousiastic user. I was about to launch logseq as the next documentation tool for a large governmental institution, but as long as filesystem based hierarchy is simply ingored and discared by logseq, I won’t as I’ll just be grilled by all the old-skool technicians I have to deal with.

I really hope the team can change their vision on this topic and I will definitely reconsider as soon as I see even a roadmap mentioning this :crossed_fingers:

I have to admit: I kind of overlooked this topic:

still reading on :wink:

39

Welcome.

  • Logseq doesn’t try to be an open-source Obsidian.
    • Their big similarities don’t remove their big differences.
  • I don’t expect Logseq to prioritize old-school users’ preferences.
    • Hierarchy is a well-established concept.
      • But also well-established are by now:
        • its serious limitations (e.g. in modeling)
        • some better concepts (e.g. the graph datastructure)
  • Typical file systems are:
    • hierarchical in their nature
    • very limited in what they can model
      • It is impossible to model knowledge in fs.
        • no matter how carefully you try
        • If fs happens to model well your data, then your data is not knowledge but something simpler.
  • Software projects have:
    • some fs structure (out of necessity)
    • some API structure (this is the relevant one)
      • The best APIs are almost flat (max 2 levels under the root).
  • Logseq’s namespaces are:
    • primarily for disambiguation
      • i.e. rarely needed, but great when they are
    • hierarchical in their nature
    • yet the wrong tool for modeling anything else than namespaces
      • Abusing namespaces can cause depression.
  • Hierarchies in Logseq are modeled:
    • not through namespaces
    • but through a combination of:
      • nested blocks
        • i.e. Logseq’s outliner
      • properties
        • even better in the coming database version
  • If you are:
    • already locked-in to a specific approach, then Logseq isn’t for you
    • open to something different, we can see how Logseq can fit your specific needs
      • but in a separate topic
4 Likes
40

Hi @mentaloid,

thank you for sharing this condensed summary of what the thought process of the decision to ignore fs hierarchy consists of. Are you a member of the team?

Since I am counting on neatly organized markdown files in a sensible and readable way - even when reading in less - I guess logseq will never be the right tool for me then… I really gave it a good try, but when I noticed I had a complete mess of a folder after applying namespaces to my documents (eg. all my Behavior documentation now moved to /Application/Behaviors/Application__Behaviors__Timestampable.md I just reverted everything with git and removed logseq from my system completely as this is just a waste of time and effort at this point. I need multiple levels of namespacing as I’m documenting various complex topics and I just can’t / won’t keep all these files in a flat folder structure or with a naming scheme like that.

  1. I get that Logseq doesn’t try to be an open-source alternative to Obsidian, but it kind of is…
  2. I guess I’m too old now to be considered a target user then? I’m not using fs to model anything, just to give some sensible organisation between vastly different topics related to the same project.
  3. I don’t know what you consider a typical fs, but claiming that its impossible to model knowledge in it is complete and utter nonsense for anyone with eyes and a brain afaict. I’m using and collaborating on my “not knowledge” on a daily basis to explain to others (and to remember myself) the complicated topics related to software design and both Obsidian and logseq (albeit for a very short time) have helped me a lot to navigate around and wow the croud, but that apparently doesn’t matter since it’s just my data, something too simple to be worthy of the label knowledge or a tool like logseq…
  4. You glance over a very important key aspect (“out of necessity”) and then go on to tell me what’s the “best” API? I guess you haven’t really worked with moderately large API’s like eg. the one from Gitlab, Stripe or most social media providers…
  5. I don’t know what “knowledge” you have that rarely needs namespacing or disambiguation, but that’s just no true for me, especially when dealing with both legacy and bleeding edge in a large organization. “Abusing namespaces” doesn’t even make sense: there’s no such things as too much namespacing in my experience. It’s the fs that actually reduces the nesting of namespacing in the first place! How you turn the tables here is kind of impressive tbh.
  6. The most frustrating part of using logseq is not even the namespacing, but how every section of a well organized structure is shown as a flat list. For instance, I like to write disambiguation sections as part of the introduction in many markdown files, and seeing how logseq randomly lists them without context is just very confusing given the inherent hierarchy already given by the fs.
  7. I’m not locked-in to a particular approach, markdown and a fs (duh) are the only lock-in I’m happily embracing. It’s logseq’s behavior that is forcing me to use logseq and logseq only. Nice spin again, but sadly for you it’s the polar opposite: it’s logseq that’s forcing a vendor lock-in, not the other way around.

Please allow me to rant a bit now (:poop: ahead!): you seem to be excited about a coming database version? So not only a slow and deprecated electron app, but now with SQL support? That’s even funny at this point, glad I removed logseq before I ever have to witness that or got used to working around the aforementioned issues by becoming dependent on third-party plugins that might break with every update. “Schoenmaker, blijf bij uw leest” is a saying that goes around here… By now I really hope you are NOT a part of the team and I don’t even care anymore to look it up myself. I tried to love logseq, but “good riddens” at this point, “FREEDOM again :saluting_face:” “no more permittedInsecurePackages :computer: :smiling_face_with_three_hearts:”