Just wanted to say I love this discussion. Have been pondering the need for a 128-bit OS for decades but several of the issues raised were completely novel to me. Fantastic to have so many people so much smarter than I am hash it out informally here. Feels like a master class.
The road to Zettalinux
141–150 of 199 posts
Re: The road to Zettalinux
#142Earlier quoted context omitted.
> it's difficult to imagine what kind of new physics would let someone fit that many bytes into a machine that's practical to control with a single Linux kernel instance. I nominally agree with most of your post. But I should note that modern systems seem to be moving towards a "one pointer space" for the entire cluster. For example, 8 GPUs + 2 CPUs would share the same virtual memory space (GPU#1 may take one slice,…
Distributed Shared Memory is a thing, but I'm not sure how widely it is used. I found that it gives you all the coordination problems of threads in symmetric multiprocessing but at a larger scale and with much slower synchronisation. https://en.wikipedia.org/wiki/Distributed_shared_memory
At least at this latest SIGMOD, it felt like everyone and their dog was researching databases in an RDMA environment… so I’d imagine this stuff hasn’t peaked in popularity.
Re: The road to Zettalinux
#143The section about 128-bit pointers being necessary for expanded memory sizes is unconvincing -- 64 bits provides 16 EiB (16 x 1024 x 1024 x 1024 x 1 GiB), which is the sort of address space you might need for byte-level addressing of a warehouse full of high-density HDDs. Memory sizes don't grow like they used to, and it's difficult to imagine what kind of new physics would let someone fit that many bytes into a mach…
And there's another disadvantage to 128-bit pointers - memory size and alignment.
It would follow that each struct field would become 16 byte-aligned, and pointers would bloat up as well, leading to even more memory consumption, especially in languages that favor pointer-heavy structures.
This was a major counterargument against 64-bit x86, where the transition came out as a net zero in terms of performance, due to the hit of larger pointer sizes counterbalanced by ISA improvements such as more addressable registers.
Many people in high-performance circles advocate using 32-bit array indices opposed to pointers, to counteract the cache pollution effects.
Re: The road to Zettalinux
#144Earlier quoted context omitted.
The article does talk about just making the pointer type used in syscalls 256 bit wide to "give room for any surprising future needs". The size of large networked disk arrays will grow beyond 64 bit addresses, but I don't think we will exceed 2^128 bits of storage of any size, for any practical application. Then again, there's probably people who thought the same about 32 bit addresses when we moved from 16bit to 32b…
>Then again, there's probably people who thought the same about 32 bit addresses when we moved from 16bit to 32bit addresses. There's a fun "quote" about 384k being all anyone would ever need, so clearly everyone just needs to settle down and figure out how to refactor their code.
The IBM PC's 20-bit addressing was 16 times the size of 16-bit addresses. From 20-bit to 32-bit, 4096 times larger. 32 to 64 is 4,294,967,296 times larger (!). The scale alone makes using all this space unlikely on a PC.
Re: The road to Zettalinux
#145Earlier quoted context omitted.
Leaving cluster coherent address space behind - like you say - is doable. But you lose what the parent was saying: > If everyone has the same address space, then you can share pointers / graphs between nodes and the underlying routing/ethernet software will be passing the data automatically between all systems. Its actually quite convenient.
Sounds like a disaster in terms of potential bugs.
Just because you can refer to the identity of a thing anywhere in the cluster doesn't mean it can't also be memory-safe, capability-based, and just an RPC.
Re: The road to Zettalinux
#146Quoted post unavailable.
As long as the posting of subscriber links in places like this is occasional, I believe it serves as good marketing for LWN - indeed, every now and then, I even do it myself. We just hope that people realize that we run nine feature articles every week, all of which are instantly accessible to LWN subscribers.
-- Jonathan Corbet, LWN founder & and grumpy editor in chief
https://news.ycombinator.com/item?id=1966033>
Multiple other approvals: https://hn.algolia.com/?dateRange=all&page=0&prefix=false&qu...>
Jon's own submissions: https://news.ycombinator.com/submitted?id=corbet>
And if we look for SubscriberLink submissions with significant (>20 comments) discussion ... they're showing up every few weeks, largely as Jon had requested.
https://hn.algolia.com/?dateRange=all&page=0&prefix=true&que...>
That said, those who are able to comfortably subscribe and find this information useful: please do support the site through subscriptions.
Re: The road to Zettalinux
#147Earlier quoted context omitted.
The original paper is a masterpiece to read as well, if you haven't read it. "A Parallel Algorithm for the Efficient Solution of a General Class of Recurrence Equation", by Kogge and Stone. It proves the result for _all_ associative operations (technically, a class slightly larger than associative. Kogge and Stone called this a "semi-associative" operation).
Well, just got it. Thanks for the reference! A bit sad that 1974 papers are still behind a IEEE paywall... Edit: Just finished reading it. I have to say that the generalization of 3.2 got a bit over me, but otherwise it's pretty amazing that they could define such a generalization. Intuition for those type of problem is often to proceed one step at a time, N times. That it is provably doable in log2(N) is great, espe…
> Hopefully next time I design a latency-constrained system I remember to look at that article
Nah. Your next step is to read "Data parallel algorithms" by Hillis and Steele, which starts to show how these principles can be applied to code. (Much higher-level, easier to follow, paper. From ACM too, so its free since its older than 2000)
Then you realize that all you're doing is following the steps towards "Map-reduce" and modern parallel code and just use Map Reduce / NVidia cub::scan / etc. etc. and all the modern stuff that is built from these fundamental concepts.
Kogge and Stone's paper sits at the root of it all though.
Re: The road to Zettalinux
#148Earlier quoted context omitted.
Leaving cluster coherent address space behind - like you say - is doable. But you lose what the parent was saying: > If everyone has the same address space, then you can share pointers / graphs between nodes and the underlying routing/ethernet software will be passing the data automatically between all systems. Its actually quite convenient.
Let's say you have nodes that have 10 TiB of RAM in them. You then need 1.6M nodes (not CPUs, but actual boxes) to use up 64bits of address space. It seems like the motivation is to continue to enable Top500 machines to scale. This wouldn't be coming to a commercial cloud offering for a long time.
Why limit yourself to in-memory storage? I'd definitely assume we have all our storage content memory mapped onto our cluster too, in this world. People have been building exabyte (1M gigabytes) scale datacenters since well before 2010, and 16 exabytes, the current Linux limit according to the most upvoted post here, isn't that much more inconceivable.
Having more space available usually opens up more interesting possibilities. I'm going to rattle off some assorted options. If there's multiple paths to a given bit of data, we could use different addresses to refer to different paths. We could do something like ILA in IPv6, using some of the address as a location identifier: having enough bits for both the location and the identity parts of the address without being too constrained would be helpful. We could use the extra pointer bits for tagged memory or something like CHERI, which allow all kinds of access-control or permission or security capabilities. Perhaps we create something like id's MegaTexture, where we can procedurally generate data on the fly if given an address. There's five options for why you'd want more address space than addressable storage. And I think some folks are already going to be quite limited & have quite a lot of difficulty partitioning up their address space, if they only have for example 1.6m buckets of 1TB (one possible partitioning scheme).
The idea of being able to refer to everything anywhere that does or did exist across a very large space sure seems compelling & interesting to me!
Re: The road to Zettalinux
#149I wonder if with 128-bit wide pointers it would make sense to start using early-lisp-style tagged pointers.
Re: The road to Zettalinux
#150Earlier quoted context omitted.
Distributed Shared Memory is a thing, but I'm not sure how widely it is used. I found that it gives you all the coordination problems of threads in symmetric multiprocessing but at a larger scale and with much slower synchronisation. https://en.wikipedia.org/wiki/Distributed_shared_memory
https://en.wikipedia.org/wiki/Remote_direct_memory_access Again, I'm not a supercomputer programmer. But the whitepapers often discuss RDMA. From my imagination, it sounds like any other "mmap". You, the programmer, just remembers that the mmap'd region is slower (since it is read/write to a Disk, rather than to RAM). Otherwise, you treat it "like RAM" from a programming perspective entirely for convenience sake. As…
Distributed Memory Access is just another kind of Non-Uniform Memory Access, which is Yet Another Leaky Abstraction. Specifically, if you care about performance at all you now have to worry about where in RAM your data lives.
Caring about where in memory your data lives is different from dealing with cache or paging. Programmers have to plan ahead to keep frequently accessed data in fast RAM, and infrequently accessed data in "slow" RAM. You'll probably need special APIs to allocate and manage memory in the different pools, not unlike the Address Windowing Extensions API in Microsoft Windows.
And once you extend "memory" outside the chassis, you'll have to design your application with the expectation that any memory access could fail because a network failure means the memory is no longer accessible.
If you only plan to deploy in a data center then maybe you can ignore pointer faults, but that is still a risk, especially if you decide to deploy something like Chaos Monkey to test your fault tolerance.