Ask HN: What rabbit hole(s) did you dive into recently?

I recently fell into the Linux from Scratch rabbithole. The extra unexpected fun came while attempting to build LFS from within NixOS (my daily driver, which I only had a basic proficiency in). I quickly realized the challenges of following the LFS project's guidance, due in large part to how the Nix store is implemented. So I fell down another rabbithole of learning derivations and the Nix language - which took me a good way though building the LFS toolchain within Nix's intended sandbox. However, achieving FHS-compliance became another issue as I attempted to build essential LFS system tools within a chroot-like environment, but sought to do so in roughly the same declarative/reproducible manner as Nix's ideal. After a few lost hairs, I discovered the wonders of Nix's FHS build environment/bubblewrap implementation. A few hoop-jumps later, handling permissions and migrating to the final build partition, the project was complete with a mostly declarative, mostly reproducible, functional, minimal, bootable LFS build.

What sticks with me most through this experience is the brilliance of the open-source community (and a special satisfaction with now being able to say "I use nix btw")

My two rabbit holes from the last ~year:

JellyFin. Amazing. A little clunky here and there, so some automation is needed. Then the weird bugs - gotta get a debug instance going to see what is going on. Once you start thinking about it as a "personal Netflix" you start building out a larger collection which needs organizing. Then your friends want an account. Then you realize your satellite tuner has a streaming function so you start reading through the plugin code for other streaming boxes...

Next one was the Japanese PS2 DVR combo unit. Sold as junk (which mostly they are - none can read discs anymore basically) but very interesting in that they are beautiful and cool and weird. The English OSD translations will often break your unit in subtle ways - so I joined the discord and identified that this was actually a configuration management problem - you have to only ever use a translation which was done to a set of OSD files which match your firmware revision. So I started writing a framework to auto-translate the XML file strings. Then the guys on the community mentioned the images with strings, as well as detecting when your translated string is too long for the UI element... I am still working on the framework. No dealbreakers yet! I will probably buy a few more to generate all my own OSD translations. My wife is going to kill me - I have 2 PSX DVR's now, and probably will buy more on our trip to Japan soon...

PDF files, and why the heck they are so slow to read. Hours upon hours of perf(1) and fiddling with ugly things in C. My main takeaway is everyone in the world is doing things HORRIBLY wrong and there's no way to stop them.

(Digression: did you know libpng, the one everyone uses, is not supposed to be an optimized production library—rather it's a reference implementation? It's almost completely unoptimized, no really, take a look, anywhere in the codebase. Critical hot loops are 15 year old C that doesn't autovectorize. I easily got a 200% speedup with a 30-line patch, on something I cared about (their decoding of 1-bit bilevel to RGBA). I'm using that modified libpng right now. I know of nowhere to submit this patch. Why the heck is everyone using libpng?)

The worst offender (so far) is the JBIG2 format (several major libraries, including jbig2dec), a very popular format that gets EXTREMELY high compression ratios on bilevel images of types typical to scanned pdfs. But: it's also a format that's pretty slow to decompress—not something you want in a UI loop, like a PDF reader is! And, there's no way around that—if you look at the hot loop, which is arithmetic coding, it's a mess of highly branchy code that's purely serial and cannot be thread- nor SIMD- parallelized. (Standardized in 2000, so it wasn't an obvious downside then). I want to try to deep-dive into this one (as best as my limited skill allows), but I think it's unlikely there's any low-hanging optimization fruit, like there's so much of in libpng. It's all wrong that everyone's using this slow, non-optimizable compression format in PDF's today, but, no one really cares. Everyone's doing things wrong and there is no way to stop them.

Another observation: lots of people create PDF's at print-quality pixel density that's useless for screens, and greatly increases rendering latency. Does JBIG2 support interlacing or progressive decoding, to sidestep this challenge? Of course it doesn't.

Everyone's doing PDF things wrong and there is no way under the blue sky to make them stop.

This is really cool. I work in the plant industry, and most people don't know this but huge numbers of the plants bought in the U.S. are grown from tissue culture. I would guess it could be in the ballpark of half of all small and medium-sized plants (< 10" diameter), because it brings some benefits when propagating at scale for certain genera. For others, it's too difficult to grow a stable plant from tissue culture, so they need to be propagated vegetatively.

Unrelated to work: the black hole rabbit hole.

It started with an article about the hypothesis that planet nine may be a primordial black hole with 3-6 Earth masses.

What’s a primordial black hole? It’s one that formed in the first seconds after the Big Bang. We don’t know for sure they exist but many theories and simulations predict them.

They’re an excellent dark matter candidate. Could it be that simple? Could at least a lot of the missing mass be tied up in little baseball sized embers from the birth of the universe that rarely interact with anything so we don’t see them? They’d be small, would rarely interact, and unless they are sucking in mass (causing a hot accretion disk) would be dark.

Then I got onto Hawking radiation and whether micro black holes could exist. Along the way I read about loop quantum gravity (LQG) which looks to me like a decent stab at unifying QM and GR that’s much less baroque and more testable than string theory.

That then led to the LQG “bounce” hypothesis for black holes. See LQG does away with true infinite mass singularities. Instead a black hole would be matter packed to its theoretical maximum density (which is still insane). From there it would quickly “bounce” and become a white hole.

So wait… how do black holes persist then? Time dilation! From a the hole’s frame of reference it collapses and then instantly bounces and goes kaboom. From our frame of reference though all that gravity slows it to such a crawl that the black hole phase at or near max density looks like it’s stable. The bounce takes billions to even trillions of years!

Last but not least I learned about the black hole starship idea. It’s a set of ideas about how far future intelligences could use black holes as mass energy converters to reach relativistic velocities. Might be somewhat easier (for crazy sci-fi values of “easy”) than handling antimatter. This also gives SETI yet another wild extreme technosignature to look for.

… and back to the beginning I found a post about how if planet nine were a PBH we could use it to yeet probes to the stars at meaningful fractions of c… at least if we could make them able to survive insane g forces. Unlike the black hole starship this would be feasible today. It’d just be a gravity assist off a ludicrous gravity well.

Here I thought black holes were dull. Turns out they’re the most extreme objects in the universe and a whole lot of the most amazing physics intersects around them. If there is any way we could tap into the phenomenon we could potentially access sci-fi levels of energy too.