Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

IO Devices and Latency

planetscale.com

Post reply on HN
Print view View original

111–120 of 159 posts

Re: IO Devices and Latency

#111

Post by dzr0001 »

jstimpfle wrote: post #89
dzr0001 wrote: post #41

Earlier quoted context omitted.

What drive is this and does it need a trim? Not all NVMe devices are created equal, especially in consumer drives. In a previous role I was responsible for qualifying drives. Any datacenter or enterprise class drive that had that sort of latency in direct IO write benchmarks after proper pre-conditioning would have failed our validation.

My current one reads SAMSUNG MZVL21T0HCLR-00BH1 and is built into a quite new work laptop. I can't get below around 250us avg. On my older system I had a WD_BLACK SN850X but had it connected to an M.1 slot which may be limiting. This is where I measured 1-2ms latency. Is there any good place to get numbers of what is possible with enterprise hardware today? I've struggled for some time to find a good source.

Unfortunately, this data is harder to find than it should be. For instance, just looking at Kioxia, which I've found to be very performant, their datasheets for the CD series drives don't mention write latency at all. Blocks and Files[1] mentions that they claim [1] https://blocksandfiles.com/2023/08/07/kioxias-rocketship-dat...

Re: IO Devices and Latency

#112

Post by 392 »

__turbobrew__ wrote: post #49

I think something about distributed storage which is not appreciated in this article: 1. Some systems do not support replication out of the box. Sure your cassandra cluster and mysql can do master slave replication, but lots of systems cannot. 2. Your life becomes much harder with NVME storage in cloud as you need to respect maintenance intervals and cloud initiated drains. If you do not hook into those system and dr…

This is where s2.dev could in theory come to the rescue. Able to keep up with the streaming bandwidth, but durable.

Re: IO Devices and Latency

#113

Post by CSDude »

For years, I just didn't get why replicated databases always stick with EBS and deal with its latency. Like, replication is already there, why not be brave and just go with local disks? At my previous orgs, where we ran Elasticsearch for temporary logs/metrics storage, I proposed we do exactly that since we didn't even have major reliability requirements. But I couldn't convince them back then, we ended up with even worse AWS Elasticsearch.

I get that local disks are finite, yeah, but I think the core/memory/disk ratio would be good enough for most use cases, no? There are plenty of local disk instances with different ratios as well, so I think a good balance could be found. You could even use local hard disk ones with 20TB+ disks for implementing hot/cold storage.

Big kudos to the PlanetScale team, they're like, finally doing what makes sense. I mean, even AWS themselves don't run Elasticsearch on local disks! Imagine running ClickHouse, Cassandra, all of that on local disks.

Re: IO Devices and Latency

#114

Post by TylerE »

mixermachine wrote: post #37
jasonthorsness wrote: post #12

Earlier quoted context omitted.

The visuals are awesome; the bouncing-box is probably the best illustration of relative latency I've seen. Your "1 in a million" comment on durability is certainly too pessimistic once you consider the briefness of the downtime before a new server comes in and re-replicates everything, right? I would think if your recovery is 10 minutes for example, even if each of three servers is guaranteed to fail once in the mont…

1 in a million is the probability that all three servers die in one months, without swapping out the broken ones. So at some point in the month all the data is gone. If you replace the failed(or failing) node right away, the failure percentage goes down greatly. You would likely need the probability of a node going done in 30 minutes time space. Assuming the migration can be done in 30 min. (i hope this calculation i…

One thing I will suggest: you’re assuming failures are non-correlated and have an equally weighted chance per in it of time.

Neither is a good assumption from my experience. Failures being correlated to any degree greatly increases the chances of what the aviation world refers to as “the holes in the Swiss cheese lining up”.

Re: IO Devices and Latency

#115

Post by TylerE »

bob1029 wrote: post #92
sedatk wrote: post #72

Earlier quoted context omitted.

"...has orders of magnitude less latency than..." [citation needed]. Local network access shouldn't be much different than local IPC.

> Local network access In what production scenarios do MySQL, Postgres, DB2, Oracle, et. al., live on the same machine as the application that uses them? I am pretty sure most of these vendors would offer strict guidance to not do that.

Like 95% of websites that aren’t Amazon or google? Ton of sites that run in a single small vm. Postgres scales down quite nicely and will happily run in say, 512MB.

Re: IO Devices and Latency

#116

Post by logsr »

bddicken wrote: post #48
logsr wrote: post #33

Earlier quoted context omitted.

Amazing presentation. It really helps to understand the concepts. The only add is that it understates the impact of SSD parallelism. 8 Channel controllers are typical for high end devices and 4K random IOPS continue to scale with queue depth, but for an introduction the example is probably complex enough. It is great to see PlanetScale moving in this direction and sharing the knowledge.

Thank you for the info! Do you have any good references on this for those who want to learn more?

Just going off specs sheets from manufacturers and reviews (mostly consumer products, so enterprise should be the same or better).

There are only a few major NAND manufacturers: Samsung, Micron, Kioxia / Western Digital, SK Hynix, and their branded products are usually the best.

There are also several 3rd party controller developers: Phison, Marvell, Silicon Motion, which I think are the largest, and then a bunch of others.

I hadn't looked at this in a couple years, so 16 channel controllers are more common now, but only on high end enterprise devices.

4KB random read/write specs are definitely not trustable without testing. They are usually at max queue depth and, at least for consumer devices, based on writing to a buffer in SLC mode, so they will be a lot lower once the buffer is exhausted. Enterprise specs might be more realistic but there isnt as much public testing data available.

Re: IO Devices and Latency

#117

Post by _1tem »

bob1029 wrote: post #67
crazygringo wrote: post #47

Earlier quoted context omitted.

> I've been advocating for SQLite+NVMe for a while now. Why SQLite instead of a traditional client-server database like Postgres? Maybe it's a smidge faster on a single host, but you're just making it harder for yourself the moment you have 2 webservers instead of 1, and both need to write to the database. > Latency is king in all performance matters. This seems misleading. First of all, your performance doesn't matt…

The entire point is to avoid the network hop. Application SQLite NVMe has orders of magnitude less latency than Application Postgres Client Network Postgres Server NVMe > You should be avoiding serial database queries as much as possible in the first place. I don't get to decide this. The business does.

Postgres supports Unix sockets when running on the same machine. That’s what I use, for a significant latency improvement over the TCP stack even at 127.0.0.1.

Re: IO Devices and Latency

#118

Post by _1tem »

sedatk wrote: post #72
bob1029 wrote: post #67

Earlier quoted context omitted.

The entire point is to avoid the network hop. Application SQLite NVMe has orders of magnitude less latency than Application Postgres Client Network Postgres Server NVMe > You should be avoiding serial database queries as much as possible in the first place. I don't get to decide this. The business does.

"...has orders of magnitude less latency than..." [citation needed]. Local network access shouldn't be much different than local IPC.

I’ve tested this before and Postgres is measurably faster over Unix socket than over local network.

Re: IO Devices and Latency

#119

Post by _1tem »

chatmasta wrote: post #106
bob1029 wrote: post #68

Earlier quoted context omitted.

> If only one thread of writing is required, then SQLite works absolutely great. The whole point of getting your commands down to microsecond execution time is so that you can get away with just one thread of writing. Entire financial exchanges operate on this premise.

Entire financial exchanges are not running single threaded writes to their persistent data store. If they are, and you have a link, I’d love to be proven wrong.

https://use.expensify.com/blog/scaling-sqlite-to-4m-qps-on-a...

Re: IO Devices and Latency

#120

Post by kev009 »

jstimpfle wrote: post #26
bob1029 wrote: post #18

I've been advocating for SQLite+NVMe for a while now. For me it is a new kind of pattern you can apply to get much further into trouble than usual. In some cases, you might actually make it out to the other side without needing to scale horizontally. Latency is king in all performance matters. Especially in those where items must be processed serially. Running SQLite on NVMe provides a latency advantage that no other…

I still measure 1-2ms of latency with an NVMe disk on my Desktop computer, doing fsync() on a file on a ext4 filesystem. Update: about 800us on a more modern system.

NVMe is just a protocol. There are drives that are absolute shit and others that cost as much as luxury automobiles. In either case not quite DRAM latency because it is expansion bus attached.
Post reply on HN