Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

I/O Is Faster Than CPU – Let’s Partition Resources and Eliminate OS Abstractions [pdf]

penberg.org

Post reply on HN
Print view View original

151–160 of 291 posts

Re: I/O Is Faster Than CPU – Let’s Partition Resources and Eliminate OS Abstractions [pdf]

#151

Post by gambler »

jandrewrogers wrote: post #125
sytelus wrote: post #48

Earlier quoted context omitted.

There are plenty of in-memory database architectures that takes full advantage of fast random access offered by DRAMs. I think paper focuses on OS design which is still tied with assumption of slow disks.

In-memory databases are not economically scalable. To a first approximation, database volume is the integral of database throughput, so databases that require extreme I/O throughput also tend to require extremely high volumes of online data. Many typical sensor data models, which drive much of the need for high-throughput I/O, grow by petabytes per day so the multiple orders of magnitude difference in media cost make…

>In-memory databases are not economically scalable.

SAP HANA is widely used for very large data sets.

Re: I/O Is Faster Than CPU – Let’s Partition Resources and Eliminate OS Abstractions [pdf]

#152

Post by acdha »

virgilp wrote: post #140
acdha wrote: post #116

Earlier quoted context omitted.

I never saw it portrayed that way: it was always higher-level programmer ease of use pitches along the lines of “isn’t this easier than doing a JOIN?” pitch shortly before the speaker had a valuable learning experience about the value of data integrity and consistency.

You really don't remember the "web scale" memes about Mongo? It was always primarily about "performance".

I don't remember seeing that predicated as a low-level I/O savings — it was always things about being easy to cluster or avoiding the performance overheads of ACID. The pitch was usually “it's easier and you don't need a skilled DBA”.

Re: I/O Is Faster Than CPU – Let’s Partition Resources and Eliminate OS Abstractions [pdf]

#153

Post by kazinator »

Animats wrote: post #118

Mainframe designers had this problem under control by 1970. Mainframes had, and have, "channels". A channel is part of the processor architecture. It takes commands, sends them to a peripheral, and manages the data transfer in both directions. Channels have some privileged functions through which the OS tells them where the data is supposed to go in memory. The architecture of channels is well defined, and peripheral…

> Channels have some privileged functions through which the OS tells them where the data is supposed to go in memory.

So the channel controller system has DMA; just not the peripheral.

> Minicomputers of the 1970s had low transistor counts and slow CPUs. So peripherals were usually put directly on the memory bus, with full access to memory.

That's "bus mastering" DMA. There is such a thing as "third party" DMA, which uses generic DMA controller, which is programmed to push data back and forth between memory and relatively dumb peripherals.

That approximates the channel concept.

PC's have had DMA controllers since 1980 something. The IBM PC had it in the form of the Intel 8237 chip. This is documented as having four "channels", wouldn't you know it.

Re: I/O Is Faster Than CPU – Let’s Partition Resources and Eliminate OS Abstractions [pdf]

#154

Post by convolvatron »

inetknght wrote: post #114

Look to GPUs for better solutions: make more discrete -- less functional overall but higher performance -- cores, and move them closer to the data; then let the CPU just handle coordination of the discrete processors. Think of SIMD but on a massive scale. Think of blocks of RAM with math processors. Or the same in your NVMe/NIC/etc.

'systems code' like protocol implementations and device drivers tend to be very control-flow centric. on a simd machine in the worst case this means narrowing the 'vector length' to effectively 1. since these are throughput machines, thats often pretty bad.

I do agree with you that lots of little processors is a good way forward here with a careful eye towards reducing sharing of state, but maybe its useful in this case for them to have their own instruction streams.

Re: I/O Is Faster Than CPU – Let’s Partition Resources and Eliminate OS Abstractions [pdf]

#155

Post by the_af »

thaumaturgy wrote: post #144
the_af wrote: post #131

Earlier quoted context omitted.

Wasn't this kind of filesystem an abandoned plan for one of the (now old) versions of Windows? Like Windows 2000 or XP? I don't remember why they ditched this plan. Maybe it wasn't such a good idea after all, who knows.

There have been a few toy filesystems which have explored this. BeOS back in the late 90s came the closest to broad adoption, even though it didn't fully embrace the paradigm, just aspects of it. I think there are a few reasons it hasn't happened: 1. Filesystems are hard. Every new filesystem architecture ends up requiring a large pool of talented developers. 2. Nobody wants to break backwards compatibility. Current…

I agree and didn't mean to imply database filesystems are necessarily bad. Just wondering.

I forgot about BeOS! Do you remember which version of Windows was going to do it, or at least had preliminary plans for it? I distinctly remember reading about it. Was it an early version of Windows XP or what? I can't remember...

Re: I/O Is Faster Than CPU – Let’s Partition Resources and Eliminate OS Abstractions [pdf]

#157

Post by thaumaturgy »

the_af wrote: post #155
thaumaturgy wrote: post #144

Earlier quoted context omitted.

There have been a few toy filesystems which have explored this. BeOS back in the late 90s came the closest to broad adoption, even though it didn't fully embrace the paradigm, just aspects of it. I think there are a few reasons it hasn't happened: 1. Filesystems are hard. Every new filesystem architecture ends up requiring a large pool of talented developers. 2. Nobody wants to break backwards compatibility. Current…

I agree and didn't mean to imply database filesystems are necessarily bad. Just wondering. I forgot about BeOS! Do you remember which version of Windows was going to do it, or at least had preliminary plans for it? I distinctly remember reading about it. Was it an early version of Windows XP or what? I can't remember...

Are you thinking of WinFS? I think that was sometime between XP and Vista.

Re: I/O Is Faster Than CPU – Let’s Partition Resources and Eliminate OS Abstractions [pdf]

#158

Post by the_af »

thaumaturgy wrote: post #157
the_af wrote: post #155

Earlier quoted context omitted.

I agree and didn't mean to imply database filesystems are necessarily bad. Just wondering. I forgot about BeOS! Do you remember which version of Windows was going to do it, or at least had preliminary plans for it? I distinctly remember reading about it. Was it an early version of Windows XP or what? I can't remember...

Are you thinking of WinFS? I think that was sometime between XP and Vista.

Yes! WinFS. Thank you!

Re: I/O Is Faster Than CPU – Let’s Partition Resources and Eliminate OS Abstractions [pdf]

#159

Post by jandrewrogers »

gambler wrote: post #151
jandrewrogers wrote: post #125

Earlier quoted context omitted.

In-memory databases are not economically scalable. To a first approximation, database volume is the integral of database throughput, so databases that require extreme I/O throughput also tend to require extremely high volumes of online data. Many typical sensor data models, which drive much of the need for high-throughput I/O, grow by petabytes per day so the multiple orders of magnitude difference in media cost make…

>In-memory databases are not economically scalable. SAP HANA is widely used for very large data sets.

How are you defining "very large"? I've used HANA and even SAP doesn't make claim of real scalability -- the practical limitations are in their own documentation. And in real scalability testing, it struggles long before you reach those limits. For sensor data, I wouldn't use it for more than a few terabytes, and even then it has a few sharp edges when it comes to performance if you are not careful. No one is going to be putting petabytes into it, which isn't that much operational data these days.

And to my point you quoted, SAP HANA is extremely expensive to operate compared to alternatives. The licensing costs alone will kill you, never mind the hardware requirements.

Re: I/O Is Faster Than CPU – Let’s Partition Resources and Eliminate OS Abstractions [pdf]

#160

Post by snazz »

the_af wrote: post #155
thaumaturgy wrote: post #144

Earlier quoted context omitted.

There have been a few toy filesystems which have explored this. BeOS back in the late 90s came the closest to broad adoption, even though it didn't fully embrace the paradigm, just aspects of it. I think there are a few reasons it hasn't happened: 1. Filesystems are hard. Every new filesystem architecture ends up requiring a large pool of talented developers. 2. Nobody wants to break backwards compatibility. Current…

I agree and didn't mean to imply database filesystems are necessarily bad. Just wondering. I forgot about BeOS! Do you remember which version of Windows was going to do it, or at least had preliminary plans for it? I distinctly remember reading about it. Was it an early version of Windows XP or what? I can't remember...

Longhorn? https://en.m.wikipedia.org/wiki/Development_of_Windows_Vista
Post reply on HN