Hi all, is anyone using org-ql with a big number of files, probably in combination with org-roam, instead of say deft? How well does it scale to thousands of files? I know that org-agenda can get pretty sluggish around a thousand files, but it's my understanding that org-ql is faster, although maybe not that faster. Any experience? Thanks!

NOTE: this is not for TODOs but for general search, I've seen workarounds that exclude files not containing tasks nor projects from the search, but that's not what I'm interested in, I want to search every file and ideally using more structured queries like those of org-ql (vs deft).

EDIT: I've been experimenting a bit with this:

• I cloned this big nonsense ZK which is designed to stress-test note-taking apps.
• I converted all notes from md to org using pandoc: for f in *; do pandoc --to org "$f" > "${f%%.md}.org"; done. Not really necessary for what it follows, except for the file extension part, but anyway.
• Now, searching for some text with ripgrep is very fast, as expected, for example with 56/10000 matches:

```

time rg -c "to queer noises" *.org | wc -l 56

real 0m0.317s user 0m0.195s sys 0m0.316s ```

• Now with org-ql the same search setting org-directory to be the ZK folder, requiring no grouping and just random order, and doing a (regexp "to queer noises") freezes, at least I've been waiting for minutes now for it to end. Repeating the search in a subset of that ZK with just a few hundred notes, it ends after some seconds, but it's still borderline usable.

So I guess the answer is a sound NO, org-ql can't be reasonably used in a setup like this one. u/github-alphapapa do you have any objection to my methodology here? Thanks!

EDIT: deft has also been terribly sluggish with this same ZK, both in generating the deft buffer and in searching through it, so the best option seems to be something like consult-ripgrep.