Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

IO Devices and Latency

planetscale.com

Post reply on HN
Print view View original

41–50 of 159 posts

Re: IO Devices and Latency

#41

Post by dzr0001 »

jstimpfle wrote: post #26
bob1029 wrote: post #18

I've been advocating for SQLite+NVMe for a while now. For me it is a new kind of pattern you can apply to get much further into trouble than usual. In some cases, you might actually make it out to the other side without needing to scale horizontally. Latency is king in all performance matters. Especially in those where items must be processed serially. Running SQLite on NVMe provides a latency advantage that no other…

I still measure 1-2ms of latency with an NVMe disk on my Desktop computer, doing fsync() on a file on a ext4 filesystem. Update: about 800us on a more modern system.

What drive is this and does it need a trim? Not all NVMe devices are created equal, especially in consumer drives. In a previous role I was responsible for qualifying drives. Any datacenter or enterprise class drive that had that sort of latency in direct IO write benchmarks after proper pre-conditioning would have failed our validation.

Re: IO Devices and Latency

#42

Post by rbranson »

the8472 wrote: post #36
ucarion wrote: post #31

Really, really great article. The visualization of random writes is very nicely done. On: > Another issue with network-attached storage in the cloud comes in the form of limiting IOPS. Many cloud providers that use this model, including AWS and Google Cloud, limit the amount of IO operations you can send over the wire. [...] > If instead you have your storage attached directly to your compute instance, there are no a…

For network-attached storage IOPS limits packets per second, not bandwidth, since IO operations can happen at different sizes (e.g. 4K vs. 16K blocks).

More specific details for EC2 instances can be seen in the docs here: https://docs.aws.amazon.com/ec2/latest/instancetypes/gp.html...

Re: IO Devices and Latency

#43

Post by ucarion »

rbranson wrote: post #39
ucarion wrote: post #31

Really, really great article. The visualization of random writes is very nicely done. On: > Another issue with network-attached storage in the cloud comes in the form of limiting IOPS. Many cloud providers that use this model, including AWS and Google Cloud, limit the amount of IO operations you can send over the wire. [...] > If instead you have your storage attached directly to your compute instance, there are no a…

> 1. The ratelimit on "IOPS" is precisely a ratelimit on a particular kind of network traffic, right? Namely traffic to/from an EBS volume? "IOPS" really means "EBS volume network traffic"? The EBS volume itself has a provisioned capacity of IOPS and throughput, and the EC2 instance it's attached to will have its own limits as well across all the EBS volumes attached to it. I would characterize it more like a differe…

That makes sense. The weirdness of https://docs.aws.amazon.com/ebs/latest/userguide/ebs-io-char... makes more sense now. Reminds me of DynamoDB capacity units.

Re: IO Devices and Latency

#44

Post by jstimpfle »

rbranson wrote: post #32
jstimpfle wrote: post #26

Earlier quoted context omitted.

I still measure 1-2ms of latency with an NVMe disk on my Desktop computer, doing fsync() on a file on a ext4 filesystem. Update: about 800us on a more modern system.

Not so sure that's true. This is single-threaded direct I/O doing a fio randwrite workload on a WD 850X Gen4 SSD: write: IOPS=18.8k, BW=73.5MiB/s (77.1MB/s)(4412MiB/60001msec); 0 zone resets slat (usec): min=2, max=335, avg= 3.42, stdev= 1.65 clat (nsec): min=932, max=24868k, avg=49188.32, stdev=65291.21 lat (usec): min=29, max=24880, avg=52.67, stdev=65.73 clat percentiles (usec): | 1.00th=[ 33], 5.00th=[ 34], 10.00…

I checked again with O_DIRECT and now I stand corrected. I didn't know that O_DIRECT could make such a huge difference. Thanks!

Re: IO Devices and Latency

#45

Post by madisp »

jstimpfle wrote: post #26
bob1029 wrote: post #18

I've been advocating for SQLite+NVMe for a while now. For me it is a new kind of pattern you can apply to get much further into trouble than usual. In some cases, you might actually make it out to the other side without needing to scale horizontally. Latency is king in all performance matters. Especially in those where items must be processed serially. Running SQLite on NVMe provides a latency advantage that no other…

I still measure 1-2ms of latency with an NVMe disk on my Desktop computer, doing fsync() on a file on a ext4 filesystem. Update: about 800us on a more modern system.

I'm not an expert, but I think an enterprise NVMe will have some sort of power loss protection so it can afford to fsync to ram/caches as they will be written down in a power loss. Consumer NVMe drives afaik lack this so fsync will force the file to be written.

Re: IO Devices and Latency

#46

Post by rcrowley »

jhgg wrote: post #19

Metal looks super cool, however at my last job when we tried using instance local SSD's on GCP, there were serious reliability issues (e.g. blocks on the device losing data). Has this situation changed? What machine types are you using? Our workaround was this: https://discord.com/blog/how-discord-supercharges-network-di...

Neat workaround! We only started working with GCP Local SSDs in 2024 and can report we haven't experienced read or write failures due to bad sectors in any of our testing.

That said, we're running a redundant system in which MySQL semi-sync replication ensures every write is durable to two machines, each in a different availability zone, before that write's acknowledged to the client. And our Kubernetes operator plus Vitess' vtorc process are working together to aggressively detect and replace failed or even suspicious replicas.

In GCP we find the best results on n2d-highmem machines. In AWS, though, we run on pretty much all the latest-generation types with instance storage.

Re: IO Devices and Latency

#47

Post by crazygringo »

bob1029 wrote: post #18

I've been advocating for SQLite+NVMe for a while now. For me it is a new kind of pattern you can apply to get much further into trouble than usual. In some cases, you might actually make it out to the other side without needing to scale horizontally. Latency is king in all performance matters. Especially in those where items must be processed serially. Running SQLite on NVMe provides a latency advantage that no other…

> I've been advocating for SQLite+NVMe for a while now.

Why SQLite instead of a traditional client-server database like Postgres? Maybe it's a smidge faster on a single host, but you're just making it harder for yourself the moment you have 2 webservers instead of 1, and both need to write to the database.

> Latency is king in all performance matters.

This seems misleading. First of all, your performance doesn't matter if you don't have consistency, which is what you now have to figure out the moment you have multiple webservers. And secondly, database latency is generally miniscule compared to internet round-trip latency, which itself is miniscule compared to the "latency" of waiting for all page assets to load like images and code libraries.

> Especially in those where items must be processed serially.

You should be avoiding serial database queries as much as possible in the first place. You should be using joins whenever possible instead of separate queries, and whenever not possible you should be issuing queries asynchronously at once as much as possible, so they execute in parallel.

Re: IO Devices and Latency

#48

Post by bddicken »

logsr wrote: post #33
bddicken wrote: post #3

Author of the blog here. I had a great time writing this. By far the most complex article I've ever put together, with literally thousands of lines of js to build out these interactive visuals. I hope everyone enjoys.

Amazing presentation. It really helps to understand the concepts. The only add is that it understates the impact of SSD parallelism. 8 Channel controllers are typical for high end devices and 4K random IOPS continue to scale with queue depth, but for an introduction the example is probably complex enough. It is great to see PlanetScale moving in this direction and sharing the knowledge.

Thank you for the info! Do you have any good references on this for those who want to learn more?

Re: IO Devices and Latency

#49

Post by __turbobrew__ »

I think something about distributed storage which is not appreciated in this article:

1. Some systems do not support replication out of the box. Sure your cassandra cluster and mysql can do master slave replication, but lots of systems cannot.

2. Your life becomes much harder with NVME storage in cloud as you need to respect maintenance intervals and cloud initiated drains. If you do not hook into those system and drain your data to a different node, the data goes poof. Separating storage from compute allows the cloud operator to drain and move around compute as needed and since the data is independent from the compute — and the cloud operator manages that data system and draining for that system as well — the operator can manage workload placements without the customer needing to be involved.

Re: IO Devices and Latency

#50

Post by [deleted] »

jstimpfle wrote: post #26
bob1029 wrote: post #18

I've been advocating for SQLite+NVMe for a while now. For me it is a new kind of pattern you can apply to get much further into trouble than usual. In some cases, you might actually make it out to the other side without needing to scale horizontally. Latency is king in all performance matters. Especially in those where items must be processed serially. Running SQLite on NVMe provides a latency advantage that no other…

I still measure 1-2ms of latency with an NVMe disk on my Desktop computer, doing fsync() on a file on a ext4 filesystem. Update: about 800us on a more modern system.

[deleted]
Post reply on HN