31–40 of 291 posts
Does this mean we go from virtual machines (i.e. VMWare) to kubernetes & containers in data centers? Something similar to RanchOS
Earlier quoted context omitted.
NoSQL is based on this realization. Rather than optimize for IO with normalization, denormalize and pipe data more or less directly from storage to clients, simplifying the compute and consistency model to make distributed data easier.
The point of normalization is logical consistency, not IO optimization.
At least in database kernels, we noticeably reached this threshold around five years ago with typical server hardware. This is an interesting computer science problem in that virtually all of our database literature is based on the presumption that I/O is much slower than CPU. If you cleanroom a database kernel design based on the assumption that I/O performance is not the bottleneck, you end up with an architecture…
At least in database kernels, we noticeably reached this threshold around five years ago with typical server hardware. This is an interesting computer science problem in that virtually all of our database literature is based on the presumption that I/O is much slower than CPU. If you cleanroom a database kernel design based on the assumption that I/O performance is not the bottleneck, you end up with an architecture…
What was the situation like before SSDs? Were the fastest hard drives at the time (15k rpm?) faster than the fastest CPUs?
https://serverfault.com/questions/190451/what-is-the-through...
https://www.anandtech.com/show/9185/intel-xeon-d-review-perf...
Now, that 45.8 GB/sec can't usually be fully utilized by the application, but it's a lot higher than the 200MB/sec of the server hard drive.
There's also the complication of RAID, max memory fetch rate of a single core, etc. And the fact that the database has to do some processing besides moving data around.
But it seems to me that server CPU bandwidth of the last generation is significantly higher than spinning disks.
At least in database kernels, we noticeably reached this threshold around five years ago with typical server hardware. This is an interesting computer science problem in that virtually all of our database literature is based on the presumption that I/O is much slower than CPU. If you cleanroom a database kernel design based on the assumption that I/O performance is not the bottleneck, you end up with an architecture…
I don't know ... we used to interleave our hard disk formats because a hard disk could stream data faster than an i386 CPU could ingest it, and there was plenty of database research done prior to 1990.
But even if SSDs do blow HDDs out the water on sequential IO, it's on random IO that the difference is most stark, the cost difference between sequential and random is much lower on SSD than on HDDs, and both random IO throughput and random IOPS shoot through the roof relative to spinning rust.
An SSD might be an order of magnitude better than an HDD on sequential IO, but will often be 2 or 3 on random IO. That means the loss of throughput from sequential IO to random IO (while still there due to commands overhead and the like) is much, much, much smaller on an SSD than on an HDD. And the story is similar on the latency front, an HDD might have a command latency in the low tens milliseconds, an SSD in the low-mid tens microseconds.
Very interesting shift that happened over the last 2 decades. We likely haven't designed OSes or CPUs to match this new reality.
Largely because it's not really a new reality. IBM faced the same issues on the 360's half a decade (edit: sorry, century!) ago -- you could stream data off of stacked platters in a drive into core much faster than a CPU could manage the copy. And the solution was to invent "I/O Channels", which were early DMA controllers. And the VM layer (when it was added) was cognizant of this stuff, so applications could be writ…
Guy made world record He drive car on two wheels See his video how he is driving http://bit.ly/2ZOYaWX
See the friendship between cat and squirrel That's so cute see how much they are taking care of each others http://bit.ly/2VdHs4E
Strictly speaking, the sentence "I/O is faster then CPU", aka "memory access is faster then computations" is nonsense, because it compares apples with bananas. One could probably say "transfering x data between CPU and SSD is faster then performing the computation f(x) on the CPU", where still f remains undefined.
> a 40 GbE NIC can receive a cache line sized packet every 5 ns, but the last level cache (LLC) access latency is already up to 15 ns, which means a single LLC access can already prevent the OS from keeping up with arriving packets
and
> NVMe SSDs perform I/O faster than the OS can accept new (asynchronous) I/O requests and notify their completion.
They also note e.g. that while nvme provides for 65k command queues OS generally have one IO queue per CPU.
Earlier quoted context omitted.
The point of normalization is logical consistency, not IO optimization.
Sure. But the point of (early) NoSQL was IO optimization, not logical consistency.
Earlier quoted context omitted.
The point of normalization is logical consistency, not IO optimization.
It's both.
Moreover, ALL existing relational databases do denormalize data behind the scenes (think about caches and indexes) for the sake of IO optimization, which means that normalization does not achieve the IO optimization goal by itself (otherwise, why bother?)