Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

SSDs have become fast, except in the cloud

databasearchitects.blogspot.com

Post reply on HN
Print view View original

141–150 of 427 posts

Re: SSDs have become fast, except in the cloud

#141

Post by deadmutex »

mike_hearn wrote: post #134
ianburrell wrote: post #118

Earlier quoted context omitted.

The reason for having most instances use network storage is that it makes possible migrating instances to other hosts. If the host fails, the network storage can be pointed at the new host with a reboot. AWS sends out notices regularly when they are going to reboot or migrate instances. Their probably should be more local instance storage types for using with instances that can be recreated without loss. But it is si…

Sure, I understand that, but this user is claiming that on GCP even local SSDs aren't really local, which raises the question of why not. I suspect the answer is something to do with their manufacturing processes/rack designs. When I worked there (pre GCP) machines had only a tiny disk used for booting and they wanted to get rid of that. Storage was handled by "diskful" machines that had dedicated trays of HDDs conne…

The GCE claim is unverified. OP seems to be referring to PD-SSD and not LocalSSD

Re: SSDs have become fast, except in the cloud

#142

Post by icedchai »

LeifCarrotson wrote: post #73
malfist wrote: post #45

Earlier quoted context omitted.

I keep hearing that, but that's simply not true. SSDs are fast, but they're several orders of magnitude slower than RAM, which is orders of magnitude slower than CPU Cache. Samsung 990 Pro 2TB has a latency of 40 μs DDR4-2133 with a CAS 15 has a latency of 14 nano seconds. DDR4 latency is 0.035% of one of the fastest SSDs, or to put it another way, DDR4 is 2,857x faster than an SSD. L1 cache is typically accessible i…

I wonder how many people have built failed businesses that never had enough customer data to exceed the DDR4 in the average developer laptop, and never had so many simultaneous queries it couldn't be handled by a single core running SQLite, but built the software architecture on a distributed cloud system just in case it eventually scaled to hundreds of terabytes and billions of simultaneous queries.

Many. I regularly see systems built for "big data", built for scale using "serverless" and some proprietary cloud database (like DynamoDB), storing a few hundred megabytes total. 20 years ago we would've built this on PHP and MySQL and called it a day.

Re: SSDs have become fast, except in the cloud

#143

Post by pclmulqdq »

jsnell wrote: post #71
pclmulqdq wrote: post #42

Earlier quoted context omitted.

It's because the "local" SSDs are not actually physically attached and there's a network protocol in the way.

I think you're wrong about that. AWS calls this class of storage "instance storage" [0], and defines it as: > Many Amazon EC2 instances can also include storage from devices that are located inside the host computer, referred to as instance storage. There might be some wiggle room in "physically attached", but there's none in "storage devices located inside the host computer". It's not some kind of AWS-only thing eit…

That's the abstraction they want you to work with, yes. That doesn't mean it's what is actually happening - at least not in the same way that you're thinking.

As a hint for you, I said "a network", not "the network." You can also look at public presentations about how Nitro works.

Re: SSDs have become fast, except in the cloud

#144

Post by darzu »

avg_dev wrote: post #59
malfist wrote: post #45

Earlier quoted context omitted.

I keep hearing that, but that's simply not true. SSDs are fast, but they're several orders of magnitude slower than RAM, which is orders of magnitude slower than CPU Cache. Samsung 990 Pro 2TB has a latency of 40 μs DDR4-2133 with a CAS 15 has a latency of 14 nano seconds. DDR4 latency is 0.035% of one of the fastest SSDs, or to put it another way, DDR4 is 2,857x faster than an SSD. L1 cache is typically accessible i…

pretty cool comparisons. quite some differences there. tangent, I remember reading some post called something like "Latency numbers every programmer should know" and being slightly ashamed when I could not internalize it.

probably this one: https://gist.github.com/hellerbarde/2843375

Re: SSDs have become fast, except in the cloud

#145

Post by scottlamb »

kccqzy wrote: post #138
scottlamb wrote: post #117

Earlier quoted context omitted.

> The problem is that this network is so large and slow that it can't give you anywhere near the performance of a local SSD. This wasn't a problem for hard drives, which was the backing technology when a lot of these network attached storage systems were invented, because they are fundamentally slow compared to networks, but it is a problem for SSD. Certainly true that SSD bandwidth and latency improvements are hard…

That document is probably deliberately on the pessimistic side to encourage your code to be portable across all kinds of "data centers" (however that is defined). When I previously worked at Google, the standard RPC system definitely offered 50 microseconds of round trip latency at the median (I measured it myself in a real application), and their advanced user-space implementation called Snap could offer about 10 mi…

Interesting. I worked at Google until January 2021. I see 2019 dates on that PDF, but I wasn't aware of snap when I left. There was some alternate RPC approach (Pony Express, maybe? I get the names mixed up) that claimed 10 µs or so but was advertised as experimental (iirc had some bad failure modes at the time in practice) and was simply unavailable in many of the datacenters I needed to deploy in. Maybe they're two names for the same thing. [edit: oh, yes, starting to actually read the paper now, and: "Through Snap, we created a new communication stack called Pony Express that implements a custom reliable transport and communications API."]

Actual latency with standard Stubby-over-TCP and warmed channels...it's been a while, so I don't remember the number I observed, but I remember it wasn't that much better than 0.5 ms. It was still bad enough that I didn't want to add a tier that would have helped with isolation in a particularly high-reliability system.

Re: SSDs have become fast, except in the cloud

#146

Post by jeffbee »

zokier wrote: post #131
sprachspiel wrote: post #88

Earlier quoted context omitted.

Just tested a i4i.32xlarge: $ lsblk NAME MAJ:MIN RM SIZE RO TYPE MOUNTPOINTS loop0 7:0 0 24.9M 1 loop /snap/amazon-ssm-agent/7628 loop1 7:1 0 55.7M 1 loop /snap/core18/2812 loop2 7:2 0 63.5M 1 loop /snap/core20/2015 loop3 7:3 0 111.9M 1 loop /snap/lxd/24322 loop4 7:4 0 40.9M 1 loop /snap/snapd/20290 nvme0n1 259:0 0 8G 0 disk ├─nvme0n1p1 259:1 0 7.9G 0 part / ├─nvme0n1p14 259:2 0 4M 0 part └─nvme0n1p15 259:3 0 106M 0…

I wonder if there is some tuning that needs to be done here, it seems suprising that the advertised rate would be this much off otherwise.

I would start with the LBA format, which is likely to be suboptimal for compatibility.

Re: SSDs have become fast, except in the cloud

#147

Post by wtallis »

dangoodmanUT wrote: post #119
sprachspiel wrote: post #88

Earlier quoted context omitted.

Just tested a i4i.32xlarge: $ lsblk NAME MAJ:MIN RM SIZE RO TYPE MOUNTPOINTS loop0 7:0 0 24.9M 1 loop /snap/amazon-ssm-agent/7628 loop1 7:1 0 55.7M 1 loop /snap/core18/2812 loop2 7:2 0 63.5M 1 loop /snap/core20/2015 loop3 7:3 0 111.9M 1 loop /snap/lxd/24322 loop4 7:4 0 40.9M 1 loop /snap/snapd/20290 nvme0n1 259:0 0 8G 0 disk ├─nvme0n1p1 259:1 0 7.9G 0 part / ├─nvme0n1p14 259:2 0 4M 0 part └─nvme0n1p15 259:3 0 106M 0…

that's 16m blocks, not 4k

Last I checked, Linux splits up massive IO requests like that before sending them to the disk. But there's no benefit to splitting a sequential IO request all the way down to 4kB.

Re: SSDs have become fast, except in the cloud

#149

Post by Szpadel »

kccqzy wrote: post #138
scottlamb wrote: post #117

Earlier quoted context omitted.

> The problem is that this network is so large and slow that it can't give you anywhere near the performance of a local SSD. This wasn't a problem for hard drives, which was the backing technology when a lot of these network attached storage systems were invented, because they are fundamentally slow compared to networks, but it is a problem for SSD. Certainly true that SSD bandwidth and latency improvements are hard…

That document is probably deliberately on the pessimistic side to encourage your code to be portable across all kinds of "data centers" (however that is defined). When I previously worked at Google, the standard RPC system definitely offered 50 microseconds of round trip latency at the median (I measured it myself in a real application), and their advanced user-space implementation called Snap could offer about 10 mi…

with such speed and CXL gaining traction (think ram and GPUs over network) why network SSD is still issue? you could have like one storage server per rack that would serve storage only for that particular rack

you could easily have like 40GB/s with some over provisioning / bucketing

Re: SSDs have become fast, except in the cloud

#150

Post by Retric »

cduzz wrote: post #116
Retric wrote: post #21

Earlier quoted context omitted.

Redundancy, local storage is a single point of failure. You can use local SSD’s as slow RAM, but anything on it can go away at any moment.

I've seen SANs get nuked by operator error or by environmental issues (overheated DC == SAN shuts itself down). Distributed clusters of things can work just fine on ephemeral local storage (aka local storage ). A kafka cluster or an opensearch cluster will be fine using instance local storage, for instance. As with everything else.... "it depends"

Sure distributed clusters get back to network/workload limitations.
Post reply on HN