Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

The road to Zettalinux

lwn.net

Post reply on HN
Print view View original

131–140 of 199 posts

Re: The road to Zettalinux

#131

Post by MikeHalcrow »

I recall sitting in a packed room with over a hundred devs at the 2004 Ottawa Linux Symposium while the topic of the number of filesystem bits was being discussed (link: https://www.linux.com/news/ottawa-linux-symposium-day-2/). I recall people throwing out questions as to why we weren't just jumping to 128 or 256 bits, and at one point someone blurted out something about 1024 bits. Someone then made a comment about the number of atoms in the universe, everyone chuckled, and the discussion moved on. I sensed the feeling in the room was that any talk of 128 bits or more was simply ridiculous. Mind you this was for storage.

Fast-forward 18 years, and it's fascinating to me to see people now seriously floating the proposal to support 256-bit pointers.

Re: The road to Zettalinux

#132

Post by bdn_ »

PaulHoule wrote: post #2

On one hand The IBM System/38 used 128 bit pointers in the 1970s, despite having a 48 bit physical address bus. These were used to manage persistent objects on disk or network with unique ids a lot like uuids. On the other hand, filling out a 64 bit address space looks tough. I struggled to find something of the same magnitude of 2^64 and I got ‘number of iron atoms in an iron filing’, From a nanotechnological point…

I thought up a few ways to visualize 2^64 unique items:

- You could give every ant on Earth ~920 unique IDs without any collisions

- You could give unique IDs for every brain neuron for all ~215 million people in Brazil

- The ocean contains about 20 × (2^64) gallons of water (3.5267 × 10^20 gallons total)

- There are between 100-400 billion stars in the Milky Way, so you could assign each star between 46,000,000–184,000,000 unique IDs each

- You could assign ~2.5 unique IDs to each grain of sand on Earth

- If every cell of your body contained a city with 500,000 people each, every "citizen" of your body could have a unique ID without any collisions

Calculating these figures is actually a lot of fun!

Re: The road to Zettalinux

#133

Post by lostmsu »

lostmsu wrote: post #47
jmillikin wrote: post #18

The section about 128-bit pointers being necessary for expanded memory sizes is unconvincing -- 64 bits provides 16 EiB (16 x 1024 x 1024 x 1024 x 1 GiB), which is the sort of address space you might need for byte-level addressing of a warehouse full of high-density HDDs. Memory sizes don't grow like they used to, and it's difficult to imagine what kind of new physics would let someone fit that many bytes into a mach…

So 64 bit address is only 1024 16TB HDDs? That number may go down quickly. There is a 100TB SSD already.

Ah, parent already corrected their mistake. The comment I was responding to was saying 16*1024*1024*1GB.

Re: The road to Zettalinux

#134

Post by Eduard »

bdn_ wrote: post #132
PaulHoule wrote: post #2

On one hand The IBM System/38 used 128 bit pointers in the 1970s, despite having a 48 bit physical address bus. These were used to manage persistent objects on disk or network with unique ids a lot like uuids. On the other hand, filling out a 64 bit address space looks tough. I struggled to find something of the same magnitude of 2^64 and I got ‘number of iron atoms in an iron filing’, From a nanotechnological point…

I thought up a few ways to visualize 2^64 unique items: - You could give every ant on Earth ~920 unique IDs without any collisions - You could give unique IDs for every brain neuron for all ~215 million people in Brazil - The ocean contains about 20 × (2^64) gallons of water (3.5267 × 10^20 gallons total) - There are between 100-400 billion stars in the Milky Way, so you could assign each star between 46,000,000–184,…

There are only ~368 grains of sand per ant?

Re: The road to Zettalinux

#135

Post by alain94040 »

masklinn wrote: post #21
LinkLink wrote: post #6

For reference 2^64 = ~10^19.266 I don't think this is unreasonable at all, its unlikely that computers will largely stay the same in the coming years. I believe we'll see many changes to how things like mass addressing of data and computing resources is done. Right now our limitations in these regards are addressed by distributed computing and databases, but in a hyper-connected world there may come a time when such…

I don’t know it seems excessive to me. I could see the cold storage maybe, with spanning storage pools (by my reckoning there were 10TB drives in 2016 and the largest now are 20, so 16 years from now should be 320 if it keeps doubling, which is 5 orders of magnitude below still). > Right now our limitations in these regards are addressed by distributed computing and databases, but in a hyper-connected world there may…

Thanks to Moore's law, you can assume that DRAM capacity will double every 1-3 years. Every time it doubles, you need one more bit. So if we use 48 bits today, we have 16 bits left to grow, which gives us at least 16 years of margin, and maybe 48 years. (and it could be even longer if you believe that Moore's law is going to keep slowing down).

Re: The road to Zettalinux

#136

Post by ghoward »

It would be sad if we, as an industry, do not take this opportunity to create a better OS.

First, we should decide whether to have a microkernel or a monolithic kernel.

I think the answer is obvious: microkernel. This is much safer, and seL4 has shown that performance need not suffer too much.

Next, we should start by acknowledging the chicken-and-egg problem, especially with drivers. We will need drivers.

So let's reuse Linux drivers by implementing a library for them to run in userspace. This should be difficult, but not impossible, and the rewards would be massive, basically deleting the chicken-and-egg problem for drivers.

To solve the userspace chicken-and-egg problem (having applications that run on the OS), implement a POSIX API on top of the OS. Yes, this will mean that some bad legacy like `fork()` will exist, but it will solve that chicken-and-egg problem.

From there, it's a simple matter of deciding what the best design is.

I believe it would be three things:

1. Acknowledging hardware as in [1].

2. A copy-on-write filesystem with a transactional API (maybe a modified ZFS or BtrFS).

3. A uniform event API like Windows' handles and Wait() functions or Plan 9's file descriptors.

For number 3, note that not everything has to be a file, but receiving events like signals and events from child processes should be waitable, like in Windows or Linux's signalfd and pidfd.

For number 2, this would make programming so much easier on everybody, including kernel and filesystem devs. And I may be wrong, but it seems like it would not be hard to implement. When doing copy-on-write, just copy as usual, and update the root B-tree node; the transaction commits when the root B-tree node is flushed to disk, and the flush succeeds.

(Of course, this would also require disks that don't lie, but that's another problem.)

[1]: https://www.usenix.org/conference/osdi21/presentation/fri-ke...

Re: The road to Zettalinux

#137

Post by torginus »

On a bit tangential note, RAM price for a given cost used to increase exponentially until the 2010s or so.

Since then, it only roughly halved. What happened?

https://jcmit.net/memoryprice.htm

I know it's not process geometry, since we went from 45nm->5nm in the time, a roughly 81x decrease.

Is is realistic to assume scaling will resume?

Re: The road to Zettalinux

#138

Post by hnuser123456 »

torginus wrote: post #137

On a bit tangential note, RAM price for a given cost used to increase exponentially until the 2010s or so. Since then, it only roughly halved. What happened? https://jcmit.net/memoryprice.htm I know it's not process geometry, since we went from 45nm->5nm in the time, a roughly 81x decrease. Is is realistic to assume scaling will resume?

We decided to slow down giving programmers excuses to make chat applications as heavy as web browsers

Re: The road to Zettalinux

#139

Post by turtletontine »

PaulHoule wrote: post #128
lloeki wrote: post #99

Earlier quoted context omitted.

> filling out a 64 bit address space looks tough. I struggled to find something of the same magnitude of 2^64 and I got ‘number of iron atoms in an iron filing’ Reminded me of Jeff Bonwick's answer to the following question about his 'boiling the oceans' quip related to ZFS being a "128 bit filesystem": > 64 bits would have been plenty ... but then you can't talk out of your ass about boiling oceans then, can you? Sa…

One absolute limit of computation is that it takes (1/2) kT of energy to delete one bit of information where k is the Boltzmann constant and T is the temperature. Let T = 300° K (room temperature) I multiplied that by 2¹²⁸, and got 1.41×10¹⁸ J of energy. 1 ton of TNT is 4.2×10¹² J, so that is a 335 kiloton explosion worth of energy just to boot. That's not impossible, that much heat is extracted from a nuclear reacto…

In fairness, the need for 128-bit addressable systems will be when 64 address bits is not enough. That will be long before people are using 2^128 bytes on one system. So doing the calculation with 2^65 bytes would be a more even handed estimate of the machine that would require this

Re: The road to Zettalinux

#140

Post by vlovich123 »

rektide wrote: post #50
cmrdporcupine wrote: post #44

Earlier quoted context omitted.

It seems to me that such a memory space could be physically mapped quite large while still presenting 64-bit virtual memory addresses to the local node? How likely is it that any given node would be mapping out more than 2^64 bytes worth of virtual pages? The VM system could quite simply track the physical addresses as a pair of `u64_t`s or whatever, and present those pages as 64-bit pointers. It seems in particular…

Leaving cluster coherent address space behind - like you say - is doable. But you lose what the parent was saying: > If everyone has the same address space, then you can share pointers / graphs between nodes and the underlying routing/ethernet software will be passing the data automatically between all systems. Its actually quite convenient.

Let's say you have nodes that have 10 TiB of RAM in them. You then need 1.6M nodes (not CPUs, but actual boxes) to use up 64bits of address space. It seems like the motivation is to continue to enable Top500 machines to scale. This wouldn't be coming to a commercial cloud offering for a long time.
Post reply on HN