Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

What Does a Database for SSDs Look Like?

brooker.co.za

Post reply on HN
Print view View original

91–100 of 127 posts

Re: What Does a Database for SSDs Look Like?

#91

Post by sreekanth850 »

cmrdporcupine wrote: post #77
sreekanth850 wrote: post #24

Unpopular Opinion: Database were designed for 1980-90 mechanics, the only thing that never innovates is DB. It still use BTree/LSM tree that were optimized for spinning disc. Inefficiency is masked by hardware innovation and speed (Moores Law).

There's plenty of innovation in DB storage tech, but the hardware interface itself is still page-based. It turns out that btrees are still efficient for this work. At least until the hardware vendors deign to give us an interface to SSD that looks more like RAM. Reading over https://www.cs.cit.tum.de/dis/research/leanstore/ and associated papers and follow up work is recommended. In the meantime with RAM prices sky r…

Btrees are not optimal for SSD, and the only reason we still use them is legacy constraints of page-oriented storage and POSIX block interfaces.We pay a lot of unnecessary write amplification, metadata churn, and small random writes because we’re still force-fitting tree structures into a block device abstraction.

Re: What Does a Database for SSDs Look Like?

#92

Post by packetlost »

hawk_ wrote: post #89
Palomides wrote: post #85

Earlier quoted context omitted.

I just tested the mediocre enterprise nvme I have sitting on my desk (micron 7400 pro), it does over 30000 fsyncs per second (over a thunderbolt adapter to my laptop, even)

If you tested this on macos, be careful. The fsync on it lies.

fsync on most OSes lie to some degree

Re: What Does a Database for SSDs Look Like?

#93

Post by cmrdporcupine »

sreekanth850 wrote: post #91
cmrdporcupine wrote: post #77

Earlier quoted context omitted.

There's plenty of innovation in DB storage tech, but the hardware interface itself is still page-based. It turns out that btrees are still efficient for this work. At least until the hardware vendors deign to give us an interface to SSD that looks more like RAM. Reading over https://www.cs.cit.tum.de/dis/research/leanstore/ and associated papers and follow up work is recommended. In the meantime with RAM prices sky r…

Btrees are not optimal for SSD, and the only reason we still use them is legacy constraints of page-oriented storage and POSIX block interfaces.We pay a lot of unnecessary write amplification, metadata churn, and small random writes because we’re still force-fitting tree structures into a block device abstraction.

I don't think we're disagreeing. But the issue is at the boundary between software and hardware, which the hardware device manufacturers have dictated, not further up.

Re: What Does a Database for SSDs Look Like?

#94

Post by Palomides »

hawk_ wrote: post #89
Palomides wrote: post #85

Earlier quoted context omitted.

I just tested the mediocre enterprise nvme I have sitting on my desk (micron 7400 pro), it does over 30000 fsyncs per second (over a thunderbolt adapter to my laptop, even)

If you tested this on macos, be careful. The fsync on it lies.

nope, linux python script that writes a little data and calls os.fsync

Re: What Does a Database for SSDs Look Like?

#95

Post by rdtsc »

hawk_ wrote: post #88
rdtsc wrote: post #86

Earlier quoted context omitted.

> Skipping flushing the local disk seems rather silly to me It is. Coordinated failures shouldn't be a surprise these days. It's kind of sad to here that from an AWS engineer. Same data pattern fills the buffers and crashes multiple servers, while they were all "hoping" that others fsynced the data, but it turns out they all filled up and crashed. That's just one case there are others.

Durability always has an asterisk i.e. guaranteed up to N number of devices failing. Once that N is set, your durability is out the moment those N devices all fail together. Whether that N counts local disks or remote servers.

This is about not even trying durability before returning a result ("Commit-to-disk on a single system is [...] unnecessary") it's hoping that servers won't crash and restart together: some might fail but others will eventually commit. However that assumes a subset of random (uncoordinated) hardware failures, maybe a cosmic ray blasts the ssd controller. That's fine, but it fails to account for coordinated failure where, a particular workload leads to the same overflow scenario on all servers the same. They all acknowledge the writes to the client but then all crash and restart.

Re: What Does a Database for SSDs Look Like?

#96

Post by jandrewrogers »

sreekanth850 wrote: post #31
PunchyHamster wrote: post #21

> WALs, and related low-level logging details, are critical for database systems that care deeply about durability on a single system. But the modern database isn’t like that: it doesn’t depend on commit-to-disk on a single system for its durability story. Commit-to-disk on a single system is both unnecessary (because we can replicate across storage on multiple systems) and inadequate (because we don’t want to lose w…

The biggest lie we’ve been told is that databases require global consistency and a global clock. Traditional databases are still operating with Newtonian assumptions about absolute time, while the real world moves according to Einstein’s relativistic theory, where time is local and relative. You dont need global order, you dont need global clock.

You need a clock but you can have more than one. This is an important distinction.

Arbitrating differences in relative ordering across different observer clocks is what N-temporal databases are about. In databases we usually call the basic 2-temporal case “bitemporal”. The trivial 1-temporal case (which is a quasi-global clock) is what we call “time-series”.

The complexity is that N-temporality turns time into a true N-dimensional data type. These have different behavior than the N-dimensional spatial data types that everyone is familiar with, so you can’t use e.g. quadtrees as you would in the 2-spatial case and expect it to perform well.

There are no algorithms in literature for indexing N-temporal types at scale. It is a known open problem. That’s why we don’t do it in databases except at trivial scales where you can just brute-force the problem. (The theory problem is really interesting but once you start poking at it you quickly see why no one has made any progress on it. It hurts the brain just to think about it.)

Re: What Does a Database for SSDs Look Like?

#97

Post by saltcured »

Palomides wrote: post #85
ayende wrote: post #73

Earlier quoted context omitted.

Committing to NVMe drive properly is really costly. I'm talking using O_DIRECT | OSYNC or fsync here. Can be in the order of whole milliseconds, easily. And it is much worse if you are using cloud systems.

I just tested the mediocre enterprise nvme I have sitting on my desk (micron 7400 pro), it does over 30000 fsyncs per second (over a thunderbolt adapter to my laptop, even)

Another complexity here besides syncs per second is the size of the requests and duration of this test, since so many products will have faster cache/buffer layers which can be exhausted. The effect is similar whether this is a "non-volatile RAM" area on a traditional RAID controller, intermediate write zones in a complex SSD controller, or some logging/journaling layer on another volume storage abstraction like ZFS.

It is great as long as your actual workload fits, but misleading if a microbenchmark doesn't inform you of the knee in the curve where you exhaust the buffer and start observing the storage controller as it retires things from this buffer zone to the other long-term storage areas. There can also be far more variance in this state as it includes not just slower storage layers, but more bookkeeping or even garbage-collection functions.

Re: What Does a Database for SSDs Look Like?

#98

Post by koverstreet »

evanelias wrote: post #79
koverstreet wrote: post #61

Earlier quoted context omitted.

bcachefs's btree still beats the pants off of the entire rocksdb lineage :)

Rocksdb / myrocks is heavily used by Meta at extremely massive scale. For sake of comparison, what's the largest real-world production deployment of bcachefs?

We're talking about database performance here, not deployment numbers. And personally, I don't much care what Meta does, they're not pushing the envelope on reliability anywhere that I know of.

Re: What Does a Database for SSDs Look Like?

#99

Post by Sesse__ »

Tostino wrote: post #56
Sesse__ wrote: post #25

Earlier quoted context omitted.

Because it is in addition to your writes, not instead of them. That's what “ahead” points to.

Look up how "checkpointing" works in Postgres.

I know how checkpointing works in Postgres (which isn't very different from how it works in most other redo-log implementations). It still does not change that you need to actually update the heap at some point.

Postgres allows a group commit to try to combine multiple transactions to avoid the multiple fsyncs, but it adds delay and is off by default. And even so, it reduces fsyncs, not writes.

Re: What Does a Database for SSDs Look Like?

#100

Post by rafabulsing »

evanelias wrote: post #72
pmontra wrote: post #36

A tangent: > Companies are global, businesses are 24/7 Only a few companies are global, so only a few of them should optimize for those kind of workload. However maybe every startup in SV must aim to becoming global, so probably that's what most of them must optimize for, even the ones that eventually fail to get traction. 24/7 is different because even the customers of local companies, even B2B ones, mighty feel lik…

> Only a few companies are global, so only a few of them should optimize for those kind of workload A massive number of companies have global customers , regardless of where the company itself has employees. For example my b2b business is relatively tiny, yet my customer base spans four continents. Or six continents if you count free users!

(I believe) OP's point is about a company being global relative to amount of users, not just their geography. If you have single digit thousands of users or less, you still don't need those optimizations even if those users are located all around the world.
Post reply on HN