Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

I/O Is Faster Than CPU – Let’s Partition Resources and Eliminate OS Abstractions [pdf]

penberg.org

Post reply on HN
Print view View original

161–170 of 291 posts

Re: I/O Is Faster Than CPU – Let’s Partition Resources and Eliminate OS Abstractions [pdf]

#161

Post by dbcurtis »

Animats wrote: post #118

Mainframe designers had this problem under control by 1970. Mainframes had, and have, "channels". A channel is part of the processor architecture. It takes commands, sends them to a peripheral, and manages the data transfer in both directions. Channels have some privileged functions through which the OS tells them where the data is supposed to go in memory. The architecture of channels is well defined, and peripheral…

That is an IBM- and Univac-centric view of I/O. Control Data Corp had the "peripheral processor", or PP. PP's only ran OS code, usually called "driver overlays".

It was actually very elegant. There were 10 copies of PP state (20 in the 7600) and only 1 actual execution unit (2 in 7600). Hardware multi-threading in 1959! So there were 10 PP's executing PP overlay code (drivers) at 1/10 the instruction rate of the main CPU. Each PP had 4K of 12-bit words, which served for both PP code and I/O buffer space. The main memory was 60 bits wide (12*5) and the addresses were 18 bits, so the PP's had 18 bit accumulators for computing addresses.

Since PP's ran only trusted code, they were allowed to scribble anywhere in main memory that they wanted to. At the end of the I/O operation, the PP computed an address for the main CPU and that directly became the interrupt vector address. This meant that the CPU never had to deal with low level interrupts, only the much less frequent I/O operation completion interrupt at the end of a long operation.

(In a past life, I did system software at CDC, and CPU logic design at Sperry-Univac and Amdahl.)

Re: I/O Is Faster Than CPU – Let’s Partition Resources and Eliminate OS Abstractions [pdf]

#162

Post by Spivak »

scroot wrote: post #149
silversconfused wrote: post #128

Earlier quoted context omitted.

Ok, but what does the level below that look like?

I'm not sure there has to be a lower level at all, aside from asm and machine language

Okay so you have a device driver that exposes your underlying hard disk as a single contiguous segment of blocks of a given size. You have objects which are blobs of heterogeneously sized arbitrary data with an addressing scheme.

The software that marries these two things is basically a filesystem driver (in that you can implement filesystem semantics on top of it -- hell Ceph does it right now).

Nothing about a modern Linux/BSD system really stands in the way of doing this.

Re: I/O Is Faster Than CPU – Let’s Partition Resources and Eliminate OS Abstractions [pdf]

#163

Post by dooglius »

Animats wrote: post #118

Mainframe designers had this problem under control by 1970. Mainframes had, and have, "channels". A channel is part of the processor architecture. It takes commands, sends them to a peripheral, and manages the data transfer in both directions. Channels have some privileged functions through which the OS tells them where the data is supposed to go in memory. The architecture of channels is well defined, and peripheral…

This sounds pretty similar to what goes on now, except that the channel processor sits on the other side of the (e.g.) PCIe connector.

Re: I/O Is Faster Than CPU – Let’s Partition Resources and Eliminate OS Abstractions [pdf]

#164

Post by riazrizvi »

jandrewrogers wrote: post #12

At least in database kernels, we noticeably reached this threshold around five years ago with typical server hardware. This is an interesting computer science problem in that virtually all of our database literature is based on the presumption that I/O is much slower than CPU. If you cleanroom a database kernel design based on the assumption that I/O performance is not the bottleneck, you end up with an architecture…

I can see how the memory hierarchy will have layers removed, but it still exists because registers are where operations on data predominantly take place. So you still need to identify small working sets of variables that you extract return values from. I can see how a lot of caching logic will be obviated but not the principle of memory hierarchy, which says ‘load small batches of problem set, solve, store results and repeat’.

Re: I/O Is Faster Than CPU – Let’s Partition Resources and Eliminate OS Abstractions [pdf]

#165

Post by saltcured »

thaumaturgy wrote: post #82

The paper reads like it's suggesting moving the burden of complexity in dealing with varying hardware interfaces from the kernel to userland so that userland can take direct advantage of higher performance hardware when it's available. I could see that for some very small niches, but in general I think it would be a terrible development for the industry. Hardware vendors don't like to share. They don't share code, th…

There is a constant dance at the fringe of high performance systems. It leads to a recurring pattern of "revolutionizing" with some kind of bypass or coprocessor architecture, then eventually reverting to traditional structures as the new performance realities reach commodity levels.

Part of it is the economics at the fringe can pursue speed at any cost. And part is the heady appeal of doing things differently for researchers and advanced practitioners. But, in the long view, I think you are right that it is a bad idea. If you care about maintenance and sustainability, you usually find that these bypass solutions get abandoned as soon as the more conventional approaches can approximate their speed on newer commodity hardware. So there is huge churn in these specialist devices with specialist APIs and tooling.

There is a recurring theme in high performance networking where crazy things are tried and all sorts of fancy protocol offloading written, then eventually deprecated because it is seen as a support burden and a source of bugs. Because each of these specialized stacks has a smaller user base, they are have less economy of scale to invest in maintenance and stabilization.

Re: I/O Is Faster Than CPU – Let’s Partition Resources and Eliminate OS Abstractions [pdf]

#166

Post by the-rc »

bluetomcat wrote: post #70
thaumaturgy wrote: post #52

Earlier quoted context omitted.

I would like to see an updated file system architecture that's closer to a database, with tagging and all that. And I could see that extending to processes too. But what are you imagining as alternatives for i/o and terminals?

What if we surpass file systems completely? Imagine every process having its own "non-volatile" memory area, where all persistent state is kept between restarts, with a clean OS interface to share that state to other processes? Terminals and process hierarchy are a complete no-go in a fresh design. Every entity in the system can be identified through unique IDs which can be handed down to other processes based on dif…

That's the design of the System/38, AS/400, iSeries and System i. The unique IDs are 128bit pointers or capabilities.

Re: I/O Is Faster Than CPU – Let’s Partition Resources and Eliminate OS Abstractions [pdf]

#167

Post by baybal2 »

yingw787 wrote: post #13

There was a great blog post I read a while back about constructing a caching layer across network by Dan Luu: https://danluu.com/infinite-disk/ I asked a friend who works in a quant firm and he was like yes it’s true, and it is pretty insane. I think there’s research Microsoft and Google are doing for RDMA over 100G Ethernet for intra data center communication as well. Pretty neat.

I worked on something similar like that 3 years ago as a sub-sub-subcontractor for a company making DCs for Alibaba. It took them almost 2.5 years since me signing off on work to roll it out in a limited commercial trial in their alicloud hosting.

The original idea was to let purpose made hardware be distributed across DC rather than every server having to carry it: video codecs on FPGAs, hardware wire speed crypto/compression, databases and k/v stores exposed over RDMA, and remote block storage on SSDs.

I was invited to the opening ceremony for the DC. When company's bosses were showed sfx infused 3D graphs allegedly representing their AI things running on it, I was unable to restrain myself from ruining the atmosphere by asking how it is running when all servers in the DC were shut down :D

Re: I/O Is Faster Than CPU – Let’s Partition Resources and Eliminate OS Abstractions [pdf]

#168

Post by PeterCorless »

jandrewrogers wrote: post #159
gambler wrote: post #151

Earlier quoted context omitted.

>In-memory databases are not economically scalable. SAP HANA is widely used for very large data sets.

How are you defining "very large"? I've used HANA and even SAP doesn't make claim of real scalability -- the practical limitations are in their own documentation. And in real scalability testing, it struggles long before you reach those limits. For sensor data, I wouldn't use it for more than a few terabytes, and even then it has a few sharp edges when it comes to performance if you are not careful. No one is going t…

This checks out. A single 12 TiB RAM u-12tb1.metal on AWS EC2 goes for $30.539/hr. For a month for one (24 * 30), that's a bill of $21,988.08 USD.

A single i3.16xlarge, with 15.2TB NVMe SSD goes for $4.992/hr, or $3,594.24 USD a month. 16.33% of the price.

I'm not going to argue whether an in-memory database (measured in ns) wouldn't give you performance improvements over fetching data from an SSD (measured in ms). But not everyone needs that speed or can afford it for that price.

Sources: https://aws.amazon.com/blogs/aws/now-available-amazon-ec2-hi...

https://aws.amazon.com/ec2/instance-types/i3/

https://aws.amazon.com/ec2/pricing/on-demand/

Re: I/O Is Faster Than CPU – Let’s Partition Resources and Eliminate OS Abstractions [pdf]

#169

Post by Animats »

kazinator wrote: post #153
Animats wrote: post #118

Mainframe designers had this problem under control by 1970. Mainframes had, and have, "channels". A channel is part of the processor architecture. It takes commands, sends them to a peripheral, and manages the data transfer in both directions. Channels have some privileged functions through which the OS tells them where the data is supposed to go in memory. The architecture of channels is well defined, and peripheral…

> Channels have some privileged functions through which the OS tells them where the data is supposed to go in memory. So the channel controller system has DMA; just not the peripheral. > Minicomputers of the 1970s had low transistor counts and slow CPUs. So peripherals were usually put directly on the memory bus, with full access to memory. That's "bus mastering" DMA. There is such a thing as "third party" DMA, which…

Those are more like add-on features a driver can use if present. They don't push peripheral interfaces into a standard channel-like format. Nor are they close to one that can be exposed to application programs.

Re: I/O Is Faster Than CPU – Let’s Partition Resources and Eliminate OS Abstractions [pdf]

#170

Post by gambler »

jandrewrogers wrote: post #159
gambler wrote: post #151

Earlier quoted context omitted.

>In-memory databases are not economically scalable. SAP HANA is widely used for very large data sets.

How are you defining "very large"? I've used HANA and even SAP doesn't make claim of real scalability -- the practical limitations are in their own documentation. And in real scalability testing, it struggles long before you reach those limits. For sensor data, I wouldn't use it for more than a few terabytes, and even then it has a few sharp edges when it comes to performance if you are not careful. No one is going t…

I missed the part about "petabytes a day". I'm curious, which companies stores petabytes of data a day in a single database?
Post reply on HN