Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

SSDs have become fast, except in the cloud

databasearchitects.blogspot.com

Post reply on HN
Print view View original

261–270 of 427 posts

Re: SSDs have become fast, except in the cloud

#261

Post by Ericson2314 »

Nextgrid wrote: post #252
Ericson2314 wrote: post #240

Earlier quoted context omitted.

Um... why the hell does the network care whether I am doing random or sequential access? Your left that part out of your argument.

Ah sorry, my bad. You are correct that you can fire off many random access operations in parallel and get good throughput that way. The problem is that this is not possible when the next IO request depends on the result of a previous one, like in a database where you must first read the index to know the location of the row data itself.

OK thanks yes that makes sense. Pipelining problems are real.

Re: SSDs have become fast, except in the cloud

#262

Post by nine_k »

kristopolous wrote: post #243
LeifCarrotson wrote: post #73

Earlier quoted context omitted.

I wonder how many people have built failed businesses that never had enough customer data to exceed the DDR4 in the average developer laptop, and never had so many simultaneous queries it couldn't be handled by a single core running SQLite, but built the software architecture on a distributed cloud system just in case it eventually scaled to hundreds of terabytes and billions of simultaneous queries.

You're not considered serious if you don't. Kinda stupid.

In the startup world, this is correct.

The success that VCs are after is when your customer base doubles every month. Better yet, every week. Having a reasonably scalable infra at the start ensures that a success won't kill you.

Of course, the chances of a runaway success like this are slim, so 99% or more startups overbuild, given their resulting customer base. But it's like 99% or more pilots who put on a parachute don't end up using it; the whole point is the small minority who do, and you never know.

For a stable, predictable, medium-scale business it may make total sense to have a few dedicated physical boxes and run their whole operation from them comfortably, for a fraction of cloud costs. But starting with it is more expensive than starting with a cloud, because you immediately need an SRE, or two.

Re: SSDs have become fast, except in the cloud

#263

Post by vitus »

scottlamb wrote: post #188
KaiserPro wrote: post #181

Earlier quoted context omitted.

You're right, but TCP doesn't like packets being dropped halfway through a stream. If you have a highly QoS'd link then you'll see latency spikes.

Again, I'm not talking about spikes (though better tail latency is always desirable) but poor latency in the 50%ile case. And for high-QoS applications, not batch stuff. The snap paper linked elsewhere in the thread shows 10 µs latencies; they've put in some optimization to achieve that, but I don't really understand why we don't expect close to that with standard kernel networking and TCP.

> The snap paper linked elsewhere in the thread shows 10 µs latencies; they've put in some optimization to achieve that, but I don't really understand why we don't expect close to that with standard kernel networking and TCP.

You can get similar results by looking at comparisons between DPDK and kernel networking. Most of the usual gap comes from not needing to context-switch for kernel interrupt handling, zero-copy abstractions, and busy polling (wherein you trade CPU for lower latency instead of sleeping between iterations if there's no work to be done).

https://talawah.io/blog/linux-kernel-vs-dpdk-http-performanc... goes into some amount of detail comparing request throughput of an unoptimized kernel networking stack, optimized kernel networking stack, and DPDK. I'm not aware of any benchmarks (public or private) comparing Snap vs DPDK vs Linux, so that's probably as close as you'll get.

Re: SSDs have become fast, except in the cloud

#264

Post by zten »

Aurornis wrote: post #249
c0l0 wrote: post #48

Seeing the really just puny "provisioned IOPS" numbers on hugely expensive cloud instances made me chuckle (first in disbelief, then in horror) when I joined a "cloud-first" enterprise shop in 2020 (having come from a company that hosted their own hardware at a colo). It's no wonder that many people nowadays, esp. those who are so young that they've never experienced anything but cloud instances, seem to have little…

> It's no wonder that many people nowadays, esp. those who are so young that they've never experienced anything but cloud instances, seem to have little idea of how much performance you can actually pack in just one or two RUs today. On the contrary, young people often show up having learned on their super fast Apple SSD or a top of the line gaming machine with NVMe SSD. Many know what hardware can do. There’s no nee…

> On the contrary, young people often show up having learned on their super fast Apple SSD or a top of the line gaming machine with NVMe SSD.

Yes, this is often a big surprise. You can test out some disk-heavy app locally on your laptop and observe decent performance, and then have your day completely ruined when you provision a slice of an NVMe SSD instance type (like, i4i.2xlarge) and discover you're only paying for SATA SSD performance.

Re: SSDs have become fast, except in the cloud

#265

Post by solardev »

lolc wrote: post #255
solardev wrote: post #223

Earlier quoted context omitted.

For what kind of workloads would a slower SSD be a significant bottleneck?

I tend some workloads that transform data grids of varying sizes. The grids are anon mmaps so that when mem runs out, they get paged out. This means processing stays mostly in-mem yet won't abort when mem runs tight. The processes that get hit by paging slow to a crawl though. Getting faster SSD means they're still crawling but crawling faster. Doubling SSD throughput would pretty much half the tail latency.

I see. Thanks for explaining!

Re: SSDs have become fast, except in the cloud

#266

Post by spintin »

I'm going with hybrid:

- 2011 X-25E 64GB (2W write and almost nothing read/idle) at 100.000 writes per bit for OS

- 2021 PM897 3.7TB (2.3 Watt (read) ¦ 3 Watt (write) ¦ 1.4 Watt (idle) down from the PM983 (8.7 Watt (read) ¦ 10.6 Watt (write) ¦ 4 Watt (idle)) for DB.

This way I can get the most robust solution, with largest DB at lowest power. They are both in a 8-core Atom Mini-ITX board at 25W TDP.

Re: SSDs have become fast, except in the cloud

#267

Post by nine_k »

goguy wrote: post #200
malfist wrote: post #115

Earlier quoted context omitted.

I totally hear you about that. I work for FAANG, and I'm working on a service that has to be capable of sending 1.6m text messages in less than 10 minutes. The amount of complexity the architecture has because of those constraints is insane. When I worked at my previous job, management kept asking for that scale of designs for less than 1/1000 of the throughput and I was constantly pushing back. There's real costs to…

That really doesn't require that much complexity. I used to send something like 250k a minute complete with delivery report processing from a single machine running a bunch of other services like 10 years ago.

Nice.

But average latency is not the whole picture; tail latency is. For good tail latency and handling of spikes, you have to have a sizable "untapped" reserve of performance.

Re: SSDs have become fast, except in the cloud

#268

Post by folmar »

tw04 wrote: post #97
pclmulqdq wrote: post #11

This was a huge technical problem I worked on at Google, and is sort of fundamental to a cloud. I believe this is actually a big deal that drives peoples' technology directions. SSDs in the cloud are attached over a network, and fundamentally have to be. The problem is that this network is so large and slow that it can't give you anywhere near the performance of a local SSD. This wasn't a problem for hard drives, whi…

>The problem is that this network is so large and slow that it can't give you anywhere near the performance of a local SSD. *As implemented in the public cloud providers. You can absolutely get better than local disk speeds from SAN devices and we've been doing it for decades. To do it on-prem with flash devices will require NVMe over FC or Ethernet and an appropriate storage array. Modern all-flash array performance…

A quick comparison of marketing slides lands you with 0.5 MOPS for FC-NVMe for Qlogic 2770 and 0.7 MOPS for PCIe Micron 9100 PRO, so the better speeds are not quite there, although spending quite a lot on server gear lands you near the performance of a workstation-grade drive from 2017.

Which is still not bad, when I was shopping around in 2018 no money could buy performance comparable to a locally-attached NVMe in a more professional/datacenter-ready form.

[1] https://media-www.micron.com/-/media/client/global/documents...

[2] https://www.marvell.com/content/dam/marvell/en/public-collat...

Re: SSDs have become fast, except in the cloud

#269

Post by nine_k »

Twirrim wrote: post #57

Disclaimer: I work for OCI, opinion my own etc. We offer faster NVMe drives in instances. Our E4 Dense shapes ship with SAMSUNG MZWLJ7T6HALA-00AU3, which supports Sequential Reads of 7000 MB/s, and Sequential Write 3800 MB/s. From a general perspective, I would say the _likely_ answer to why AWS doesn't have faster NVMes at the moment is likely to be lack of specific demand. That's a guess, but that's generally how t…

I very much expect AWS SSD / NVMe upgrades to be well thought-out ahead of time, and optimized for both upfront cost and for longevity / durability. Speed may be a third consideration.

Re: SSDs have become fast, except in the cloud

#270

Post by magicalhippo »

bravetraveler wrote: post #245
magicalhippo wrote: post #203

Earlier quoted context omitted.

In say VirtualBox you can create a file backed on the physical disk, and attach it to the VM so the VM sees it as a NVMe drive. In my experience this is also orders of magnitude slower that true direct access, ie PCIe pass-through, as all access has to pass through the VM storage driver and so could explain what is happening.

The storage driver may have more impact on VBox. You can get very impressive results with 'virtio' on KVM

Yeah I've yet to try that. I know I get a similar lack of performance with Bhyve (FreeBSD) using VirtIO, so it's not a given it's fast.

I have no idea how AWS run their VMs, was just saying a slow storage driver could give such results.

Post reply on HN