Live data from Hacker News

Restartable Sequences

justine.lol

61–70 of 82 posts

Re: Restartable Sequences

#61
post #45
post #2

Maybe I'm just getting old but the "if you don't spend $20,000 on a workstation you're going to be left behind like a dinosaur" at the top of this article is a huge turn off to reading any further. And I say that as someone who owns a workstation with more cores than the author's.

I wouldn't read it too literally. I read it as jokingly rationalizing an obviously overkill self-indulgent purchase.

I gotta think people are working pretty hard to not see it's a joke.

Re: Restartable Sequences

#62
post #36

Earlier quoted context omitted.

There's quite a bit of a spectrum between "trying to make a living doing open source" and "asking for people to pay for a house in one of the most expensive cities in the country - plus a private jet . It's also quite grating to see it written like we should be grateful that we are even allowed to donate to her. And if she's even half the genius she's claiming to be, why aren't the big tech companies in a bidding war…

> From what I've read of her in the past she seems to be a pretty damn good developer. But in the open source world those are a dime a dozen Not exactly. Very few people in recent decades have achieved anything comparable to αcτµαlly pδrταblε εxεcµταblε and Cosmopolitan libc - they're in the category of "that should not even be possible". Of course, Tunney's work doesn't touch Fabrice Bellard in terms of sheer breadt…

The concept of a polyglot[0] has been around for ages: the Wikipedia page mentions an 8-language one going around on Usenet in the 1990s. If it works for source code, and for scripting languages, and for things like PDF+ZIP, why wouldn't there be a pretty good chance the various executable formats are flexible enough to allow for it too?

A libc which is flexible enough to select specific code paths depending on runtime conditions? Every libc already does this to make use of the latest hardware features[1], using the same approach for platform-specific code isn't a huge stretch - you're basically doing a lightweight WINE.

Her work is definitely impressive, but it isn't magic. You'll see similar stuff if you look into the demoscene, or the IOCCC, or a decent chunk of the talks at CCC. And it's not like APE and Cosmo libc are seeing massive adoption: the people who want portability but can't even compile for multiple platforms are probably happier with something like Java due to the better ecosystem support.

[0]: https://en.wikipedia.org/wiki/Polyglot_(computing)

[1]: https://www.phoronix.com/news/Glibc-More-AVX-512-October-202...

Re: Restartable Sequences

#63
post #53

Earlier quoted context omitted.

Justine is writing her own libc and her own malloc so I'm not surprised she wants to use rseq from scratch.

That's fine, but I think an article claiming to give an introduction to a technology should at least mention that an essential library exists, and that writing assembly is no longer usually required.

> an article claiming to give an introduction to a technology

Is it claiming that?

Re: Restartable Sequences

#64
post #2

Maybe I'm just getting old but the "if you don't spend $20,000 on a workstation you're going to be left behind like a dinosaur" at the top of this article is a huge turn off to reading any further. And I say that as someone who owns a workstation with more cores than the author's.

Yep. The consolation is I guess that it might be better if developers with such modest means as “splurged on” two 20KUSD~ work stations (and left in the “poor house” for a few months, oh my) are competitive instead of absolutely all of us becoming compute renters.

But then they go on to take a Gemini job[1] so I dunno, more consolidation than consolation perhaps.

[1] TFA says “job offer” but another comment[2] says that they work there.

[2] https://news.ycombinator.com/item?id=48348919

Re: Restartable Sequences

#65

I'm surprised there was no reference to the librseq library, maintained by the rseq implementer: https://github.com/compudj/librseq This has helpers for common use cases like counters and linked lists. You shouldn't need to write assembly at all to use rseq in most applications.

I'm took a brief look and left confused. The list implementation seems completely bog standard with no special code for synchronization whatsoever. I don't see any counter and the rseq syscall seems unused except for feature detection. I don't think that's a viable replacement for any low level code.

The low level interface is documented at https://github.com/compudj/librseq/blob/master/include/rseq/..., the list is just an internal implementation detail.

The syscall these days is invoked by libc not the program; libc provides access to some symbols that let the program execute rseqs as well.

Re: Restartable Sequences

#66
post #4

If you had no idea what a restorable sequence is the takeaway is about halfway down the OP: “This is why Linux now provides rseq() which is a much more enlightened solution. With restartable sequences, you actually can get rid of both the mutex and atomics, while the OS continues to fully abstract scheduling. The way it works is you advise the kernel whenever your program enters a critical section of code that you do…

That’s clever — am I right to think it’s the intermediate solution between locks and full STM, implemented at the kernel level, and with zero abstraction cost?

I am not very well versed here but I think due to the requirement for assembly and single-instruction commit, practical uses of rseq is generally very simple. It is nowhere near the usefulness of locks.

Re: Restartable Sequences

#67

If you had no idea what a restorable sequence is the takeaway is about halfway down the OP: “This is why Linux now provides rseq() which is a much more enlightened solution. With restartable sequences, you actually can get rid of both the mutex and atomics, while the OS continues to fully abstract scheduling. The way it works is you advise the kernel whenever your program enters a critical section of code that you do…

I wonder what the speedup would be if runtimes like golang or iava were to adopt rseq on linux.

Re: Restartable Sequences

#68

If you had no idea what a restorable sequence is the takeaway is about halfway down the OP: “This is why Linux now provides rseq() which is a much more enlightened solution. With restartable sequences, you actually can get rid of both the mutex and atomics, while the OS continues to fully abstract scheduling. The way it works is you advise the kernel whenever your program enters a critical section of code that you do…

I don't get it, how does this work with multiple processes running at 100% CPU time? Atomics are necessary for the CPU level.

Re: Restartable Sequences

#70
post #62

Earlier quoted context omitted.

> From what I've read of her in the past she seems to be a pretty damn good developer. But in the open source world those are a dime a dozen Not exactly. Very few people in recent decades have achieved anything comparable to αcτµαlly pδrταblε εxεcµταblε and Cosmopolitan libc - they're in the category of "that should not even be possible". Of course, Tunney's work doesn't touch Fabrice Bellard in terms of sheer breadt…

The concept of a polyglot[0] has been around for ages : the Wikipedia page mentions an 8-language one going around on Usenet in the 1990s. If it works for source code, and for scripting languages, and for things like PDF+ZIP, why wouldn't there be a pretty good chance the various executable formats are flexible enough to allow for it too? A libc which is flexible enough to select specific code paths depending on runt…

Yes, polyglot binaries are a straightforward extension of the notion of polyglot source code to binary executable files. But APE/Cosmopolitan takes this to insane levels:

- a fully cross-platform libc with cross-platform support built into the output binary, integrated into clang and gcc toolchains

- not just cross-platform, but also cross-architecture: output binaries work unmodified on both AMD64 and ARM64 architectures

- sheer number of executable format compatibilities: it's a polyglot across macOS, linux, windows, 3 BSDs, and a bootable BIOS target

- polyglot not just for executable outputs, but also for PKZIP. You can open and modify an executable using common zip file editors. And post-hoc zipped-in data can be accessed/modified by code in the executable using interfaces in cosmopolitan libc

- Zip polyglot can be used to create portable embeddable code-packaged interpreters where the code being interpreted is loaded/executed from zipped-in data. One example: RedBean is a very high-performance APE-packaged web server that serves data in its zip archive and hosts web service code from Lua scripts in its zip archive. Built-in library enables Lua code to store/retrieve data from zipped-in SQLite databases.

The comparison to the demoscene and IOCCC is apt, but this is at another level because it's all production-ready. The Llamafile format is an APE binary compiled with Cosmopolitan libc with model weights zipped into it.

Post reply on HN