Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

I/O Is Faster Than CPU – Let’s Partition Resources and Eliminate OS Abstractions [pdf]

penberg.org

Post reply on HN
Print view View original

121–130 of 291 posts

Re: I/O Is Faster Than CPU – Let’s Partition Resources and Eliminate OS Abstractions [pdf]

#121

Post by lame88 »

sdegutis wrote: post #77
dustingetz wrote: post #44

Earlier quoted context omitted.

http://www.datomic.com

Datomic is just a Prolog logical layer on top of traditional RDBMS (e.g. Postgres, H2, etc).

I'd agree it probably doesn't quite apply to this case, but there are similarities from what I've researched, and I think that description doesn't quite do it justice. First being that datomic is temporal with full on-demand history, which I think is a big deal; and has a serialized writer, n-reader architecture. As to how it may relate to this problem, I believe peer applications (in-process) for on-prem are able to exploit locality with the full indexes in-memory without round-trip queries - also the case for ions with datomic cloud. I'll let anyone more experienced clarify or correct me. The concept at least is really cool; I hope to give it a real test drive some time soon.

Re: I/O Is Faster Than CPU – Let’s Partition Resources and Eliminate OS Abstractions [pdf]

#122

Post by dorlaor »

masklinn wrote: post #35
shereadsthenews wrote: post #33

Earlier quoted context omitted.

I don't know ... we used to interleave our hard disk formats because a hard disk could stream data faster than an i386 CPU could ingest it, and there was plenty of database research done prior to 1990.

I assume the IO you're talking about is (/was) sequential. And DBs are specifically engineered to sequentialise their IO (e.g. clustered indexes). But even if SSDs do blow HDDs out the water on sequential IO, it's on random IO that the difference is most stark, the cost difference between sequential and random is much lower on SSD than on HDDs, and both random IO throughput and random IOPS shoot through the roof rela…

With LSM approach (log structured merge tree) we have at Scylla, we make all the writes batched/sequential and receive an awesome boost.

Some reads can be sequential too if you scan through the clustering keys.

Re: I/O Is Faster Than CPU – Let’s Partition Resources and Eliminate OS Abstractions [pdf]

#123

Post by dorlaor »

tracker1 wrote: post #47
bogomipz wrote: post #39

Earlier quoted context omitted.

I was curious about the parenthetical "early" in your comment. Is this optimization no longer the case then? Could you elaborate on recent developments regarding this? I've been out of the NoSQL loop for some time and am genuinely curious.

The problem is, it varies... you have what are effectively key/value stores to document databases, column stores and everything in between. You have systems built on other systems. RethinkDB and Cockroach have different approaches than Redis, Mongo, Cassandra or others. CockroachDB gives an SQL interface over the top of a distributed data store with better consistency and relations. Cassandra has no real relations ov…

We at Scylla tune the kernel, drives, stride, clock source automatically and benchmark the disk to figure out what's its maximum parallelism so it won't queue (we can saturate 8 NVMe) and we'll do the queuing within the DB's userspace so we can prioritize real time queries over background tasks:

https://www.scylladb.com/2018/04/19/scylla-i-o-scheduler-3/

Re: I/O Is Faster Than CPU – Let’s Partition Resources and Eliminate OS Abstractions [pdf]

#125

Post by jandrewrogers »

sytelus wrote: post #48
jandrewrogers wrote: post #12

At least in database kernels, we noticeably reached this threshold around five years ago with typical server hardware. This is an interesting computer science problem in that virtually all of our database literature is based on the presumption that I/O is much slower than CPU. If you cleanroom a database kernel design based on the assumption that I/O performance is not the bottleneck, you end up with an architecture…

There are plenty of in-memory database architectures that takes full advantage of fast random access offered by DRAMs. I think paper focuses on OS design which is still tied with assumption of slow disks.

In-memory databases are not economically scalable. To a first approximation, database volume is the integral of database throughput, so databases that require extreme I/O throughput also tend to require extremely high volumes of online data. Many typical sensor data models, which drive much of the need for high-throughput I/O, grow by petabytes per day so the multiple orders of magnitude difference in media cost make DRAM storage infeasible.

If the bottleneck is not storage bandwidth, and it isn't for many recent systems and modern software architectures, then there is no practical performance advantage to keeping your data purely in-memory. This has been demonstrable ever since PCIe HBAs with arrays of cheap SSDs became practical.

Re: I/O Is Faster Than CPU – Let’s Partition Resources and Eliminate OS Abstractions [pdf]

#126

Post by michaelmu »

Animats wrote: post #118

Mainframe designers had this problem under control by 1970. Mainframes had, and have, "channels". A channel is part of the processor architecture. It takes commands, sends them to a peripheral, and manages the data transfer in both directions. Channels have some privileged functions through which the OS tells them where the data is supposed to go in memory. The architecture of channels is well defined, and peripheral…

Thanks for this explanation. Super interesting!

Re: I/O Is Faster Than CPU – Let’s Partition Resources and Eliminate OS Abstractions [pdf]

#127

Post by dorlaor »

sytelus wrote: post #48
jandrewrogers wrote: post #12

At least in database kernels, we noticeably reached this threshold around five years ago with typical server hardware. This is an interesting computer science problem in that virtually all of our database literature is based on the presumption that I/O is much slower than CPU. If you cleanroom a database kernel design based on the assumption that I/O performance is not the bottleneck, you end up with an architecture…

There are plenty of in-memory database architectures that takes full advantage of fast random access offered by DRAMs. I think paper focuses on OS design which is still tied with assumption of slow disks.

The advantage of in-memory it's also its biggest disadvantage - it's costly. NVMe can serve almost as fast as ram and is 100x cheaper. Scylla offers single digit msec (usually 1msec) 99% latency and way more cost effective

Re: I/O Is Faster Than CPU – Let’s Partition Resources and Eliminate OS Abstractions [pdf]

#128

Post by silversconfused »

scroot wrote: post #79
Spivak wrote: post #69

Earlier quoted context omitted.

What would you replace it with? At the very core you need two things: the notion of a unit of data and some metadata to address it. To be as flexible as possible that unit of data would probably modeled as a sequence of individually addressable bytes but it could be more structured. Such a thing is dangerous because if your structure isn't sufficiently expressive 20 years down the road people will end up imposing the…

Object systems

Ok, but what does the level below that look like?

Re: I/O Is Faster Than CPU – Let’s Partition Resources and Eliminate OS Abstractions [pdf]

#129

Post by dorlaor »

johnm1019 wrote: post #89
thaumaturgy wrote: post #82

The paper reads like it's suggesting moving the burden of complexity in dealing with varying hardware interfaces from the kernel to userland so that userland can take direct advantage of higher performance hardware when it's available. I could see that for some very small niches, but in general I think it would be a terrible development for the industry. Hardware vendors don't like to share. They don't share code, th…

Disclaimer: IANA kernel developer. Is there any benefit to having those specialized developers create frameworks or libraries in Userland which other developers can leverage? This way they remain the interface to uncooperative hardware, but the code is in Userland so the bold folks can try their own approach.

Seastar.io is another one, it's an async engine that utilizes all of the cores in a modern system and is the heart of Scylla's DB. However, it's complex to use it correctly

Re: I/O Is Faster Than CPU – Let’s Partition Resources and Eliminate OS Abstractions [pdf]

#130

Post by thaumaturgy »

bluetomcat wrote: post #101
thaumaturgy wrote: post #88

Earlier quoted context omitted.

> Imagine every process having its own "non-volatile" memory area, where all persistent state is kept between restarts... I'll treat this seriously: as software development practices are just now beginning to mature, with more emphasis on test coverage being considered best practice, sure ... maybe. But there's still a lot of software out there for which "restart the application" (or even, "restart the stinking OS")…

> But there's still a lot of software out there for which "restart the application" (or even, "restart the stinking OS") is the only practical solution when it's behaving badly. Which, in principle, doesn't prevent a program from crashing or misbehaving when it encounters a corrupted file left over from its last run. In the imagined system, the application developer would be fully aware whether he is putting a data s…

> Which, in principle, doesn't prevent a program from crashing or misbehaving when it encounters a corrupted file left over from its last run.

Of course. But saving state to nvram also has this problem, plus the additional problem of saving broken program state. Like I say, I'm not totally pessimistic on this anymore: there are some promising trends that make me think we might get there in the not distant future. But we're not there yet.

Maybe supporting some way for a user to manually reset broken program state would be a good enough compromise -- as long as application developers didn't do something dumb, like store their license information in their program state. Autodesk immediately comes to mind there.

> Why would you need to communicate with any process?

Because some other process wants to communicate with you.

Composability and common interfaces and loose coupling is a huge advantage in Unix-like operating systems or any other software architecture that embraces those principles.

They mean that you can write software with a much longer lifespan, and usually for less effort. In your example, someone may come along wanting to write "Firewall App" long after you've abandoned your software. If your software is extensible and supports standardized IPC, "Firewall App" is possible. If it doesn't, then it gets thrown out and replaced with something else.

Post reply on HN