Nim 2.0 thoughts

Nope. Its solution is "we made things uglier therefore you can mangle them in a way you like".

Strong convention is much better than pleasure of everyone.

> we made things uglier

That's unfounded, please try it at least once in the playground play.nim-lang.org/. To me it's very cool that wrapped C libs with identifiers like ATOMIC_RELEASE can be written as AtomicRelease without any trouble.

This has some Zsh-isms but should get you going, assuming it is even reproducible across machines.

    $ cd cligen-github-root/examples
    # default system nim.cfg
    $ nim c -d:danger --gc:arc -d:useMalloc --passC:-flto --passL:-flto grl

    # best to fetch this while you can as they get updated regularly
    # oh yeah, just unpacked; not configured/anything.
    $ cd /dev/shm/linux-5.12.5
    # Zsh-ism to eliminate file tree traversal variation
    $ fs=(**.[ch])
    # On an Intel i7-6700K
    $ repeat 5 utime ~1/grl -sburntsushi $fs
    0.143836198              0.23              0.21             305.9%
    0.137177744              0.22              0.22             320.8%
    0.136714417              0.22              0.22             321.8%
    0.149867097              0.23              0.22             300.3%
    0.139042282              0.22              0.22             316.5%

    # make sure there is no ripgreprc file.
    $ repeat 5 utime rg burntsushi $fs
    0.204383740              0.33              0.26             288.7%
    0.200495993              0.34              0.25             294.3%
    0.205767854              0.37              0.22             286.7%
    0.200485116              0.33              0.25             289.3%
    0.201692769              0.34              0.24             287.6%

    $ rg --version
    ripgrep 12.1.1
    -SIMD -AVX (compiled)
    +SIMD +AVX (runtime)

    200.5/136.7 =~ 1.47x

OS for the above is Linux 5.12 running on bare metal. When I do a gcc PGO build I can squeeze about another 7 ms off that 136.7 time, but this is probably not very informative. Honestly, even 1.5x is kinda small, too.

Anyway, were I to study this in more detail, as it sounds like you may be wanting to do, my recommendation would be to just factor out the substring/regex search (rg's vs glibc memmem, etc.) and study a more "pure IO" case. E.g., just switch to memchr(non-existent char) or such (e.g. Daniel Lemire's Faster 64-bit universal hashing using carry-less multiplications, Journal of Cryptographic Engineering is very fast or maybe even a SIMD-summation) to force all the IO but do other things SIMD-efficiently/at the highest GB/sec easily achievable. Then try that same simplified program with ripgrep's file size heuristics with threads vs the grl N-kid processes + always mmap. Just fewer moving parts/better isolation of behaviors.

This also might lead to more general/re-applicable knowledge for the best way to do read-only multi-core file IO, not only tuning ripgrep or only Rust or only any PL. If it all holds up, and that same "test setup" could be applied to differing OS contexts like OSX/Win/etc. and then maybe it might be worthwhile re-jiggering ripgrep IO. (It is not a foregone conclusion the answer will be the same across OSes or CPUs, or even that my hunches/hypotheses are correct. I am trying to help here without making strong, general claims to be cross-examined upon, as is the Internet's way...)

Sorry. In my kernel build script I patch _STK_LIM in include/uapi/linux/resource.h and MAX_ARG_PAGES in include/linux/binfmts.h to boost my command line lengths. There was a short time when Rob Pike had some patch that fixed this with a fun comment like "dynamic memory allocation is a done deal, guys". It could be that .h files are smaller than .c files (or just fewer/less total data) making it harder to reproduce. Another element to add to my suggested research program above besides eliminating string search algos would be to standardize file sizes to all 100 KiB or something. Generate them with random data or some such. Play around with file sizes in your experiments, etc. Besides that, your CPU has twice the cores as mine and we may have pretty different memory bandwidths as mine has very low latency and high BW RAM. Reproduction is actually not generally very easy which is why I said 1.5x was not so big. I often find it hard to reproduce ratios `grl` is really just a demo program for the 8-bit clean message passing variant in `procpool` (similar to Python's multiprocessing module). `procpool` uses parent-kid pipes for its communication. The pselect6's come from using select on those N pipes. And, yeah, a lot of selects are expected since the filename passing-answer receiving happens a lot with a lot of files. { Yes, yes...I know past 1024 fd's select will be a problem, but TSMC only just hit that 1nm mark. So, we probably still have a few years before I personally can access 1024 core-thread machines.. ;-) ;-) }

mmaps and threads was discussed already in [3] linked in my first post of this thread with a brief summary. To recap briefly - since threads share all memory and since mmap alters those page tables, it is plausible the kernel just locks the whole process out of simplicity, blocking execution of other threads. It is also possible OS-? can use devious tricks/semantics to avoid that locking, but I am unaware of an exhaustive survey or proof of impossibility. Because in `procpool`/`grl` the kids which are what do the per-file mmaps, are their own processes there is no suspension of the other kids when they mmap. So, they can realize the faster IO (from SIMD register use, done by Linux for context switch optimization or "performance in the large/system-wide"). This is (perhaps) why your last test has better performance without any parallelism. I had meant to test this last time we discussed mmap & threads, but didn't at the time. `grl` is just a very preliminary test along those lines.

This is all at least partly theoretical. Someone ought to study it and write a nice blog article about it. I do not have that kind of time right now (or a blog). Or, for all I know, nice academic paper(s) already study this somewhere. I have not looked. Sadly, as old as this thread is, the general HN pool will probably not see any of this to crowd source such wisdom with direct pointers for you. I almost missed your question in my old threads checking, and I have kind of used up my time budget for this right now. You can email if you want. I get better notifies that way.

You probably won't see this, but in fact I've really enjoyed playing with Lobster!

Thanks for telling me about it, I hadn't taken it seriously before.