Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

SSDs have become fast, except in the cloud

databasearchitects.blogspot.com

Post reply on HN
Print view View original

391–400 of 427 posts

Re: SSDs have become fast, except in the cloud

#391

Post by rowanG077 »

the-rc wrote: post #368
rowanG077 wrote: post #339

Earlier quoted context omitted.

Why are you acting as if PCIe switches are some secret technology? It was extremely grating for me to read your comments.

Although it used them for years, the first mention by Google of PCIe switches was probably in the 2022 Aquila paper, which doesn't really talk about storage anyway...

I don't understand why you would expect Google to state that. They have been standard technology for almost 2 decades. You don't see google claiming they use jtag or using SPI flash or whatever. It's just not special.

Re: SSDs have become fast, except in the cloud

#392

Post by scottlamb »

vitus wrote: post #263
scottlamb wrote: post #188

Earlier quoted context omitted.

Again, I'm not talking about spikes (though better tail latency is always desirable) but poor latency in the 50%ile case. And for high-QoS applications, not batch stuff. The snap paper linked elsewhere in the thread shows 10 µs latencies; they've put in some optimization to achieve that, but I don't really understand why we don't expect close to that with standard kernel networking and TCP.

> The snap paper linked elsewhere in the thread shows 10 µs latencies; they've put in some optimization to achieve that, but I don't really understand why we don't expect close to that with standard kernel networking and TCP. You can get similar results by looking at comparisons between DPDK and kernel networking. Most of the usual gap comes from not needing to context-switch for kernel interrupt handling, zero-copy…

Thanks for the link. How does this compare to the analogous situation for SSD access? I know there are also userspace IO stacks for similar reasons, but it seems like SSD-via-kernel is way ahead of network-via-kernel in the sense that it adds less latency per operation over the best userspace stack.

Re: SSDs have become fast, except in the cloud

#393

Post by dekhn »

pclmulqdq wrote: post #294
dekhn wrote: post #218

Earlier quoted context omitted.

it sounds like you're trying to say "PCI switch" without saying "PCI switch" (I worked at Google for over a decade, including hardware division).

That is what I am trying to say without actually giving it out. PCIe switches are very much not transparent devices. Apparently AWS has not published anything about this, and doesn't have Nitro moderating access to "local" SSD, though - that I did get confused with EBS.

Like many other people in this thread, I think we disagree that a PCI switch means that an SSD "is connected over a network" to the host bus.

Now if you can show me two or more hosts connected to a box of SSDs through a PCI switch (and some sort of cool tech for coordinating between the hosts), that's interesting.

Re: SSDs have become fast, except in the cloud

#394

Post by nullindividual »

tjoff wrote: post #101
nullindividual wrote: post #10

Earlier quoted context omitted.

I don't like this answer. When I look at cloud, I get to think "finally! No more hardware to manage. No OS to manage". It's the best thing about the cloud, provided your workload is amenable to PaaS. It's great because I don't have to manage Windows or IIS. Microsoft does that part for me and significantly cheaper than it would be to employ me to do that work.

And now you have to manage the cloud instead. Which turns out to be more hassle and with no overlap with the actual problem you are trying to solve. So not only do you spend time on the wrong thing you don't even know how it works. And the providers goals are not aligned either as all they care about is locking you in. How is that better?

That may be your experience, but not mine.

Yes, the cloud is _different_ to manage and has some of the same fundamentals to overcome such as security and networking, but lacks some of the very large pain points of managing an OS, like updates, ancillary local services, local accounts, and so on.

I'm not sure why you would state that it doesn't solve the problem I'm invested in -- namely operating websites. It is the perfect cloud workload.

Re: SSDs have become fast, except in the cloud

#395

Post by Too »

wistlo wrote: post #364

At my job at a telco, I had a 13 billion record file to scan and index for duplicates and bad addresses. Consultants brought in to move our apps (some of which were Excel macros, others SAS scripts running on old desktop) to Azure. The Azure architects identified Postgres as the best tool. Consultants attempted to create a Postgres index in a small Azure instance but their tests would fail without completion (they we…

Ebsv5 does not have a local nvme. To get local drive you need to pick the version with a “d” in the name.

Re: SSDs have become fast, except in the cloud

#396

Post by Too »

silverquiet wrote: post #375
rcarmo wrote: post #372

Earlier quoted context omitted.

As someone who works with Azure daily, I am amazed not just at the initial consultant's conclusion (that is, alas, typical of folk who do not understand database engines), but also to your struggle with NVMe storage (I have some pretty large SQLite databases on my personal projects). You should not have needed an Ebsv5 (memory-optimised) instance. For that kind of thing, you should only have needed a D-series VM with…

> Anyway, many people fail to understand that Azure Storage works more like a SAN than a directly attached disk--when you attach a disk volume to the VM, you are actually attaching a _replica set_ of that storage that is at least three-way replicated and distributed across the datacenter to avoid data loss. You get RAID for free, if you will. I've said this a bit more sarcastically elsewhere in this thread, but basic…

The cloud is also being sold as “don’t worry about data loss”.

To actually deliver on that promise while maintaining abstraction of just “dump your data on C:/ as you are used to”, there are compromises in performance that need to be taken. This is one of the biggest pitfalls of the cloud if you care more about performance than resiliency. Finding disks that don’t have such guarantees is still possible, just be aware of it.

Re: SSDs have become fast, except in the cloud

#397

Post by [deleted] »

Nemo_bis wrote: post #363
namibj wrote: post #361

Earlier quoted context omitted.

Samsung 980 Pro SSDs, the early generation ones (they seem to later have replaced them with a different, likely worse architecture), have average latency 4k reads of 30~70~120 microseconds for single queue/70% of the IOPS the 120us gets you/maximum parallelism before latency goes through the roof. The metrics you mention have to be pagecache hits. Basically all MLC NAND is in the double digit microseconds for unconte…

They likely are cache hits, indeed (any suggestion what other metrics would be more comparable?). Still, at the end of the day I don't care whether a disk operation was made fast by kernel caching or by some other optimization at a lower level, I only care about the final result. With public cloud virtualization there are more layers where something may go wrong, and good luck finding answers from Amazon or Microsoft…

[deleted]

Re: SSDs have become fast, except in the cloud

#398

Post by rcarmo »

silverquiet wrote: post #390
rcarmo wrote: post #387

Earlier quoted context omitted.

Well, in short, people need to understand that cloud is not their computer. It is resource allocation with underlying assumptions around availability, redundancy and performance at a scale well beyond what they would experience in their own datacenter. And they absolutely must understand this to avoid mis-designing things. Failure to do so is just bad engineering, and a LOT of time is spent educating customers on the…

I would think that lifting and shifting a Hadoop setup into the cloud would be considered an anti-pattern anyway; typically you would be told to find a managed, cloud-native solution.

You would be surprised at what corporate thinking and procurement departments actually think is best.

Re: SSDs have become fast, except in the cloud

#399

Post by Ericson2314 »

Ericson2314 wrote: post #261
Nextgrid wrote: post #252

Earlier quoted context omitted.

Ah sorry, my bad. You are correct that you can fire off many random access operations in parallel and get good throughput that way. The problem is that this is not possible when the next IO request depends on the result of a previous one, like in a database where you must first read the index to know the location of the row data itself.

OK thanks yes that makes sense. Pipelining problems are real.

(The network indeed doesn't care, but bandwidth of dependent rather than independent accesses depends on latency)

Re: SSDs have become fast, except in the cloud

#400

Post by pdimitar »

silent_cal wrote: post #374
bcaxis wrote: post #371

Earlier quoted context omitted.

This matches my experience. Something about cloud systems makes them incredibly slow compared to real hardware. Not just the disk, the CPU is more limited too. I fire up vCPU or dedicated or bare metal in the cloud, doesn't matter, I simply cannot match the equivalent compute of real hardware and it's not even close.

Isn't that expected? I would assume cloud stuff is slower because it's essentially an emulation of the real thing.

Why should it be expected? I'm being sold compute with quoted GHz CPU speeds, RAM and types of SSDs.

I would vaguely expect it to not match my workstation, sure, but all throughout this thread (and others) people have cited outrageous disparities i.e. 5x less performance that you'd expect even if you managed your expectations to e.g. 2x less due to the cloud compute not being a bare metal machine.

In other words, and to illustrate this with a bad example: I'd be fine paying for an i7 CPU and ending up at i5 speeds... but I'm absolutely not fine with ending up at Celeron speeds.

Post reply on HN