I/O Is Faster Than CPU – Let’s Partition Resources and Eliminate OS Abstractions [pdf] • Hacker News Board

Re: I/O Is Faster Than CPU – Let’s Partition Resources and Eliminate OS Abstractions [pdf]

#181

Post by crazygringo » Fri, May 03, 2019, 7:41 PM UTC

bunderbunder wrote: post #84

mike_ivanov wrote: post #40

Earlier quoted context omitted.

No, it is not. Normalization is an artefact of Relational Algebra (a set theory being applied to tabular data). It has nothing to do with concrete implementations and their concerns (such as IO). http://wiki.c2.com/?RelationalModel . Moreover, ALL existing relational databases do denormalize data behind the scenes (think about caches and indexes) for the sake of IO optimization, which means that normalization does no…

It really is. Originally , schema normalization was just about logical consistency. But that was before query optimizers were invented, and long before they grew to be so very optimized for working with normalized schemata. Nowadays, even someone who cares 100% about performance and 0% about data integrity (And is somehow still using a relational database, yeah. So what? It's MY hypothetical.) still has good reason t…

> Nowadays, even someone who cares 100% about performance and 0% about data integrity (And is somehow still using a relational database, yeah. So what? It's MY hypothetical.) still has good reason to default to something like 3NF

That's just... wrong.

Best practices with database design are to start with a totally normalized schema because it reduces scope for errors/inconsistencies.

And then to intentionally denormalize where required to increase performance, in order to reduce lookups from joins or computing aggregate functions -- at the cost of having to manually ensure duplicated data remains consistent. Query optimizers can only do so much.

(If it weren't to increase performance, I can't really think of any reason why you ever would denormalize on a single database server, unless you have some extremely complicated calculations that SQL isn't capable of expressing?)

Re: I/O Is Faster Than CPU – Let’s Partition Resources and Eliminate OS Abstractions [pdf]

#182

Post by notduncansmith » Fri, May 03, 2019, 7:51 PM UTC

bluetomcat wrote: post #42

We need entirely new OS abstractions to replace the dated notions of hierarchical file systems built around the metaphor of file cabinets, I/O as streams of bytes, terminals, process hierarchy. Essentially, say goodbye to the Unix model after 50 years. It would open up an entirely new world of software experimentation and craftsmanship.

Why are these insufficient? What abstractions should we building on?

Re: I/O Is Faster Than CPU – Let’s Partition Resources and Eliminate OS Abstractions [pdf]

#183

Post by MrTonyD » Fri, May 03, 2019, 8:07 PM UTC

Well, I spent a number of years writing drivers for PC systems. Some years the I/O chips were faster than CPU, and some years the CPU was faster than I/O chips. DMA was usually slower, just because release cycles for CPUs tended to be faster than release cycles for I/O controllers. Eventually, most driver writers decided that it was usually better to use CPU, even if the I/O controller was faster. That way, when the CPU got upgraded, you would automatically get a speed boost. While programming an I/O controller was both more arcane and more more likely to require a complete reimplementation in a couple of years (as well as customer complaints and market share losses.)

I'm not saying that things are the same today - but it kind of sounds to me like they are. Back in the days, people were always claiming that we should switch to the newest and fastest I/O controller since CPUs were more general purpose and would therefore always be slower. It just didn't work out that way in practice.

Re: I/O Is Faster Than CPU – Let’s Partition Resources and Eliminate OS Abstractions [pdf]

#184

Post by bunderbunder » Fri, May 03, 2019, 8:11 PM UTC

crazygringo wrote: post #181

bunderbunder wrote: post #84

Earlier quoted context omitted.

It really is. Originally , schema normalization was just about logical consistency. But that was before query optimizers were invented, and long before they grew to be so very optimized for working with normalized schemata. Nowadays, even someone who cares 100% about performance and 0% about data integrity (And is somehow still using a relational database, yeah. So what? It's MY hypothetical.) still has good reason t…

> Nowadays, even someone who cares 100% about performance and 0% about data integrity (And is somehow still using a relational database, yeah. So what? It's MY hypothetical.) still has good reason to default to something like 3NF That's just... wrong. Best practices with database design are to start with a totally normalized schema because it reduces scope for errors/inconsistencies. And then to intentionally denorma…

> And then to intentionally denormalize where required to increase performance

Hopefully under profiler guidance, like for any other optimization. For my part, the frequency that I find that trying a denormalization helps performance has been on a downward trend for a while now. Mostly down to just straight-up duplicating data at this point. Databases really are getting very good at what they do.

Re: I/O Is Faster Than CPU – Let’s Partition Resources and Eliminate OS Abstractions [pdf]

#185

Post by jstimpfle » Fri, May 03, 2019, 8:14 PM UTC

pjmlp wrote: post #66

thaumaturgy wrote: post #56

Earlier quoted context omitted.

Yeah, Unix was a terrible design. Composable architecture combined with standardized i/o and IPC models. I can't imagine anything worse, especially compared to today's proliferation of data formats and inscrutable APIs. It's a good thing nobody uses it anymore.

Had Bell Labs been allowed to sell it in first place and that would have been exactly its outcome.

missing the sarcasm?

Re: I/O Is Faster Than CPU – Let’s Partition Resources and Eliminate OS Abstractions [pdf]

#186

Post by temac » Fri, May 03, 2019, 8:15 PM UTC

Animats wrote: post #118

Mainframe designers had this problem under control by 1970. Mainframes had, and have, "channels". A channel is part of the processor architecture. It takes commands, sends them to a peripheral, and manages the data transfer in both directions. Channels have some privileged functions through which the OS tells them where the data is supposed to go in memory. The architecture of channels is well defined, and peripheral…

The PS/2 was not successful not because of technicalities but because IBM did not want it to be cloned. Third parties decided what the PC should be (mostly continue to be, there was less differences between cloners and the original PC than with the PS/2), and IBM followed. In insight, the PS/2 could not win in this context, because IBM did not matter anymore very early on for the history of the PC, regardless of peripheral vendors. This market was never driven by peripherals anyway, especially not internal extension boards.

You can use VFIO, yes it needs an IOMMU but that basically what you ask for: hardware support - and yeah, I would like IOMMU to be generalized, also because it is dangerous to have some external ports without one, nowadays (even without thinking about efficient access from userspace). And because of that you can not delegate the core of the thing to "peripheral" anymore, basically you need IOMMU and once you have it, well what else do you need?

On the storage level, you can mmap on Optane on DDR4 without intermediate software copy I think. But that's merely an implementation detail anyway. When you think about the complexities of modern controllers and the diversity of options to implement them - and at the same time the most important latencies; well the only thing that matter is the end result, and obviously it can be better to not burn some CPU cycles in some cases (for some very specific workloads), but it is not trivial what to put in HW or SW to achieve that: RAM is already crazy slow, and the faster SSD is still even slower: so you will need to be smart in SW anyway to not be bitten by latencies. So in a sense; no! modern CPU are certainly not slow!

I also think you misrepresent the common modern interfaces availables for (even interoperable) SSDs; they are certainly not "emulat[ing] rotating disks".

I'd like that you give the following some thought to finish: 1/ HW backward compat is not a prime requirement anymore (because a/ smartphone dev model b/ even PC model has shifted enough so that it actually does not support complete backward compat anymore, mostly due to SoC integration). 2/ There is "sufficient" competition, where sufficient can be defined as effective to drive innovations because they can bring market advantages, and there are a non trivial number of competitors on the market (I'll mix a little but this is still relevant to what we are talking about: Intel, AMD, Arm, nVidia, Apple, Samsung, etc.) 3/ The designers working on those chips and systems are not stupid. What we have now is of course affected by technical history, but what we have now is already immensely different from what we had during the PS/2 era. Some of the historical impacts are possibly confined in less than a square millimeter on chips, and not only your PCI was not your ISA but your modern PCIe is also already quite different from your PCI... So the impact of peripheral vendors in the PS/2 era today for all of that: absolutely null, without a doubt.

Re: I/O Is Faster Than CPU – Let’s Partition Resources and Eliminate OS Abstractions [pdf]

#187

Post by dkersten » Fri, May 03, 2019, 8:17 PM UTC

Animats wrote: post #169

kazinator wrote: post #153

Earlier quoted context omitted.

> Channels have some privileged functions through which the OS tells them where the data is supposed to go in memory. So the channel controller system has DMA; just not the peripheral. > Minicomputers of the 1970s had low transistor counts and slow CPUs. So peripherals were usually put directly on the memory bus, with full access to memory. That's "bus mastering" DMA. There is such a thing as "third party" DMA, which…

Those are more like add-on features a driver can use if present. They don't push peripheral interfaces into a standard channel-like format. Nor are they close to one that can be exposed to application programs.

So would it be correct to say that functionally, it’s similar to DMA, but the API is different because “channels” expose a consistent interface while DMA doesn’t?

Re: I/O Is Faster Than CPU – Let’s Partition Resources and Eliminate OS Abstractions [pdf]

#188

Post by jandrewrogers » Fri, May 03, 2019, 8:27 PM UTC

owaislone wrote: post #22

jandrewrogers wrote: post #12

At least in database kernels, we noticeably reached this threshold around five years ago with typical server hardware. This is an interesting computer science problem in that virtually all of our database literature is based on the presumption that I/O is much slower than CPU. If you cleanroom a database kernel design based on the assumption that I/O performance is not the bottleneck, you end up with an architecture…

Are you aware of any efforts by existing databaes or entirely new databases built with this realization in mind?

Yes, there are efforts in this area. I started a first-principles database kernel design 4-5 years ago which has a couple different licensees and implementations, and should be showing up in a broad platform relatively soon. In open source, ScyllaDB is a credible implementation that goes half-way to addressing this (as an intentional reimplementation of Cassandra, it inherits some of the old style architectural features of Cassandra for better and worse). I am aware of a few other efforts at tech companies but in my opinion these are a mixed bag because they are borrowing too heavily from existing classic architectures for the sake of expediency to fully realize the benefits.

The archetypical differentiating features, relative to classic designs, of these future-looking (for lack of a better term) kernel designs is the prodigious exploitation of user space I/O, dearth of multithreaded synchronization, and the paucity of classic tree-like data structures (not necessarily hash tables, just not trees). The industry will get here eventually, some emerging tech sectors absolutely require it for their data models.

Re: I/O Is Faster Than CPU – Let’s Partition Resources and Eliminate OS Abstractions [pdf]

#189

Post by loeg » Fri, May 03, 2019, 8:29 PM UTC

the8472 wrote: post #137

loeg wrote: post #115

Earlier quoted context omitted.

Not really. These are all incremental performance improvements on POSIX but don't address the author's concerns / desires in the paper. All of them continue to require the kernel to mediate IO between userspace and the hardware. For some reason the author is fixated on direct user access to partitioned hardware queues. Netflix's CDN operating system is based on FreeBSD, and they did add a kind of kTLS implementation,…

The means may be different, but the ends they aim for seem to be similar. The ends here are improved latency, throughput, parallelism, non-blocking APIs and security. The above-mentioned improvements aim to address the first four without completely bypassing the kernel, instead changing APIs so they step out of the way most of the time, limiting them to coordination tasks and then either offloading to the hardware or…

> It's basically the difference between a green field design and tacking all those innovations onto the glueball that is linux. The latter may be ugly and complex, but it has the advantage of being backwards-compatible.

I'm definitely not trying to argue in favor of the paper's opinion. :-)

I just believe the author of the article would disagree that the glueball your earlier remarks describe addresses the author's concerns. They specifically say that Linux cannot be fixed to their satisfaction, or at least argue that idea.

Look for the section on page 5 headed, "Why not use kernel-bypass techniques on Linux?"

Re: I/O Is Faster Than CPU – Let’s Partition Resources and Eliminate OS Abstractions [pdf]

#190

Post by temac » Fri, May 03, 2019, 8:32 PM UTC

masklinn wrote: post #35

shereadsthenews wrote: post #33

Earlier quoted context omitted.

I don't know ... we used to interleave our hard disk formats because a hard disk could stream data faster than an i386 CPU could ingest it, and there was plenty of database research done prior to 1990.

I assume the IO you're talking about is (/was) sequential. And DBs are specifically engineered to sequentialise their IO (e.g. clustered indexes). But even if SSDs do blow HDDs out the water on sequential IO, it's on random IO that the difference is most stark, the cost difference between sequential and random is much lower on SSD than on HDDs, and both random IO throughput and random IOPS shoot through the roof rela…

> low-mid tens microseconds

From a modern CPU point of view, that's an eternity.