Proposal: Changing How Namespaces Function in Logseq

[Vincer]

Just adding that besides overcoming the limit and making it easier to share only parts of a graph, it could open up logseq to automatically regonize new files inside folders properly in the hierarchy

That would enhance the crosscompatibility and usecases of using logseq alongside other apps, from markdown based to pretty much anything really.
For example i could right now hardlink some md file from elsewhere into a logseq subfolder and - if/when hierarquies work through folders- it would already be ‘where it should’ inside logseq

[Vincer]

I think that, if we had to choose only 1 way folders would outweight namespaces

BUT i think logseq could implement both and give us the option

Proposed way for having both methods:

• When users make titles using DOT, namespaces(filename) only
• when typing / logseq would insert the file in a matching folder or create a new one
• Then logseq could also match namespaced pages/hierarchy to folders, and treat both ways equally inside logseq itself (ie both files folderA.file1.md and file2.md inside a folder called ‘folderA’ would appear as children of a single folderA inside Logseq)

Plus logseq could create and delete folders on the fly not creating a file for their page until something is written on it

Some other cases to keep in mind:

• when a new page is created as page/new-page when ‘page’ already exists, logseq would need to create a folder for that page and move it inside of it

[mat_rhein]

In the end, this structure could even be used for navigation as an alternative to the graph.

[alex0]

Maybe don’t make a particular naming scheme mandatory but just use folder structure to organize namespaces:

Food (folder)
    Pizza.md
    Fruit (subfolder)
        Apple.md
        Banana.md

Same folder reference:

We are in Banana.md,
let's reference Apple:
[[Apple]]

Child folder reference:

We are in Pizza.md,
let's reference Apple:
[[Fruit/Apple]]

Parent folder reference:

We are in Apple.md,
let's reference Pizza.md:
[[../Pizza]]

[stef]

A subfolder hierarchy would be extremely appreciated rater than the parent%20child%20subchild.md file naming we currently have.

I don’t see a point of having a single folder with everything in it.

[gax]

I would also like to see subfolders, for the following technical reasons:

• to avoid long pathnames as brought up by the OP. There are ways to enable long pathnames on different OSs, but some programs do not support them and might fail in unexpected ways later on, files might get moved to an even deeper hierarchy etc.
• to avoid too many files in one folder.
  • NTFS limit very high, but substantial slowdowns happen even with a few thousand files
  • FAT32 (USB drives) has a limit of 65k files. A number that can be reached easily
• to share different subfolders, as suggested by @fredguth and @Vincer
• to interface with other software through automatic generation of .md files in a watched folder

On the other hand, I don’t think subfolders hierarchies are a good way to make information accessible, because one easily runs into many problems, as discussed in an article.

I just wrote a feature request for hierarchical tags, which have many advantages over subfolders, including the ability for one item to be part of multiple hierarchies and no limit on the path length or nesting limit.

From that point of view, I think the subfolder system should be optimized so that the storage issues (no long pathnames and not too many files in one folder) are addressed first, and it should be kept simple enough to make interfacing with external programs easy. If folders serve as an extra tool for organization - that is great, but I wouldn’t sacrifice speed and stability.

Calibre let’s one pick different export formats to convert the internal folder structure, such as {author_sort}/{title}/{title} - {authors}. Logseq could also maintain a set of hardlinks to present a different view for external programs.

[mat_rhein]

In addition to all these good suggestions, Aryan’s overview on the usage of namespaces was hugely helpful to me.

[tes]

Folder structure is needed to create a content manager in Logseq

The folder.md should list also inside files (at least first child)

[alex0]

I discovered that I can move .md files in subfolders in /pages or other subfolders of the graph. Logseq use them like they are in the same folder.

So the folder structure in the file system adds another hierarchy that is independent from the others and Logseq is not aware of it.

For me this is another degree of freedom, I like it.

But I would like if Logseq let me choose in which folder a new page is created when I click “New Page”.

And I would also like a full-featured file manager instead of “All Pages” section, where I could move, rename etc the files.

[Gary_O]

See also Support subdirs for namespace hierarchy - I tried to simplify and clarify this feature request down to just “support subdirs as namespaces”.

[Junyi_Du]

Yes. Our vision is to separate namespace structure from the limited tree structure of file system.
Namespace is for concept disambiguation. Creating sub-dir by default would imply that every page should have a namespace (sub-dir), which violates our design.
Besides, the namespace file naming is going to be changed in Feat: new file name escaping rules by cnrpman · Pull Request #6134 · logseq/logseq · GitHub

@Gary_O title:: property is providing a “manual” way to manipulate the namespace structure

[brsma]

Our vision is to separate namespace structure from the limited tree structure of file system.

I’m utterly glad to hear that!

[boisjere]

I think that if we could have flexible hierarchies like Dendron that support Dendron-style tree-refactoring, that would be UH-MA-ZING!!!

[alex0]

@Junyi_Du

Yes. Our vision is to separate namespace structure from the limited tree structure of file system.

I hope at some point we will have a way to browse “virtual file systems” by specify the criteria for creating the hierarchy, for example:

{{hierarchy <property-A> <property-B> ... }}

so that

{{hierarchy year author}}

would be rendered as

- [[2022]]
  - [[Alice]]
    - blocks with year:: 2022 and author:: Alice as properties
    - ...
  - [[Bob]]
    - ...
- [[2021]]
  - [[Alice]]
    - ...
  - [[Bob]]
    - ...

[gax]

@Junyi_Du There are all kinds of idiosyncrasies of different OS/Filesystem combinations. Many programs have problems handling all path variations and might crash, fail in obscure ways, silently ignore files, or just slow down tremendously due to O(n²) runtime issues.

Unfortunately I am no expert on this, but I would strongly suggest to check with one and find a well-defined path spec that is guaranteed to work everywhere.

Any flat storage folder will very quickly lead to problems with too many files. On the Zotero forum, people are discussing libraries with more than 100k items.

One of the strength of Logseq is integration with other programs, and integrity of its storage should be the first priority. This includes making sure that Logseq doesn’t create folders with too many items.

[Junyi_Du]

Directory structure can still be adapted manually. But the hierarchy is ignored in the page title.
Binding namespace hierarchy to the file structure is not a good way to resolve the File System performance issue you mentioned. Instead, it make decoupling File System implementation with the abstract page structure impossible.

[gax]

I agree with you, I wasn’t suggesting to bind namespace hierarchy to the file structure. This is the right decision.

My concern is that some strange filepath issues might lead to eventual inconsistencies (e.g. I had issues in Python, because of a difference how the standard library interprets file paths differently than Windows).

The second worry is that folders might have too many files, e.g. if someone reasonably imports a Zotero library with 10k items and 10k tags we suddenly have 20k files in a single folder. Theoretically this is only an issue with FAT32 (memory sticks), but many programs struggle with folders that have more than a few thousand items. Networked folders could also be an issue.

[Junyi_Du]

1. Dealing with filepath on multiple OS is inevitable for plain txt storage, even without the current namespace design. There are tons of reserved characters / FS API special handling across different platforms.

2. I can understand the concern. Some shard design could be adopted when it hits the performance ceiling since the FS is decoupled

[gax]

Great! You already thought about the issue. Just wanted to make sure it doesn’t get overlooked. Thanks!

[Throwback6873]

I’m not sure if this has moved to triage by developers as it’s own feature or part of a different one, perhaps the tag approach for polyhierarchy mentioned in @gax’s feature request, link removed from quote as I’m new

@gax

I just wrote a feature request for hierarchical tags, which have many advantages over subfolders

Is there a more active topic where I can find more updated information/discussion on how others are using namespaces in their current state to meet flexible polyhierarchy needs by @alex0’s “virtual filetree” and how an eventual feature may appear?

I know there’s some continued discussion on meeting some of the needs of generic structure via queries on this discussion