Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

SSDs have become fast, except in the cloud

databasearchitects.blogspot.com

Post reply on HN
Print view View original

71–80 of 427 posts

Re: SSDs have become fast, except in the cloud

#71

Post by jsnell »

pclmulqdq wrote: post #42
jsnell wrote: post #28

Earlier quoted context omitted.

According to the submitted article, the numbers are from AWS instance types where the SSD is "physically attached" to the host, not about SSD-backed NAS solutions. Also, the article isn't just about SSDs being no faster than a network. It's about SSDs being two orders of magnitude slower than datacenter networks.

It's because the "local" SSDs are not actually physically attached and there's a network protocol in the way.

I think you're wrong about that. AWS calls this class of storage "instance storage" [0], and defines it as:

> Many Amazon EC2 instances can also include storage from devices that are located inside the host computer, referred to as instance storage.

There might be some wiggle room in "physically attached", but there's none in "storage devices located inside the host computer". It's not some kind of AWS-only thing either. GCP has "local SSD disks"[1], which I'm going to claim are likewise local, not over the network block storage. (Though the language isn't as explicit as for AWS.)

[0] https://aws.amazon.com/ec2/instance-types/

[1] https://cloud.google.com/compute/docs/disks#localssds

Re: SSDs have become fast, except in the cloud

#72

Post by Dylan16807 »

pclmulqdq wrote: post #11

This was a huge technical problem I worked on at Google, and is sort of fundamental to a cloud. I believe this is actually a big deal that drives peoples' technology directions. SSDs in the cloud are attached over a network, and fundamentally have to be. The problem is that this network is so large and slow that it can't give you anywhere near the performance of a local SSD. This wasn't a problem for hard drives, whi…

Even assuming that "local" storage is a lie, hasn't the network gotten a lot faster? The author is only asking for a 5x increase at the end of the post.

Re: SSDs have become fast, except in the cloud

#73

Post by LeifCarrotson »

malfist wrote: post #45
siliconc0w wrote: post #13

Core count plus modern nvme actually make a great case for moving away from the cloud- before it was, "your data probably fits into memory". These are so fast that they're close enough to memory so it's "your data surely fits on disk". This reduces the complexity of a lot of workloads so you can just buy a beefy server and do pretty insane caching/calculation/serving with just a single box or two for redundancy.

I keep hearing that, but that's simply not true. SSDs are fast, but they're several orders of magnitude slower than RAM, which is orders of magnitude slower than CPU Cache. Samsung 990 Pro 2TB has a latency of 40 μs DDR4-2133 with a CAS 15 has a latency of 14 nano seconds. DDR4 latency is 0.035% of one of the fastest SSDs, or to put it another way, DDR4 is 2,857x faster than an SSD. L1 cache is typically accessible i…

I wonder how many people have built failed businesses that never had enough customer data to exceed the DDR4 in the average developer laptop, and never had so many simultaneous queries it couldn't be handled by a single core running SQLite, but built the software architecture on a distributed cloud system just in case it eventually scaled to hundreds of terabytes and billions of simultaneous queries.

Re: SSDs have become fast, except in the cloud

#74

Post by boulos »

pclmulqdq wrote: post #11

This was a huge technical problem I worked on at Google, and is sort of fundamental to a cloud. I believe this is actually a big deal that drives peoples' technology directions. SSDs in the cloud are attached over a network, and fundamentally have to be. The problem is that this network is so large and slow that it can't give you anywhere near the performance of a local SSD. This wasn't a problem for hard drives, whi…

I'm not sure which external or internal product you're talking about, but there are no networks involved for Local SSD on GCE: https://cloud.google.com/compute/docs/disks/local-ssd

Are you referring to PD-SSD? Internal storage usage?

Re: SSDs have become fast, except in the cloud

#75

Post by nik_0_0 »

sprachspiel wrote: post #44
zokier wrote: post #16

> Since then, several NVMe instance types, including i4i and im4gn, have been launched. Surprisingly, however, the performance has not increased; seven years after the i3 launch, we are still stuck with 2 GB/s per SSD. AWS marketing claims otherwise: Up to 800K random write IOPS Up to 1 million random read IOPS Up to 5600 MB/second of sequential writes Up to 8000 MB/second of sequential reads https://aws.amazon.com/b…

This is for 8 SSDs and a single modern PCIe 5.0 has better specs than this.

Is it? The line preceding the bullet list on that page seems to state otherwise:

“”

  Each storage volume can deliver the following performance (all measured using 4 KiB blocks):

  * Up to 8000 MB/second of sequential reads
“”

Re: SSDs have become fast, except in the cloud

#76

Post by Dylan16807 »

bugbuddy wrote: post #39
mkoubaa wrote: post #25

Earlier quoted context omitted.

Dumb question. Why does the network have to be slow? If the SSDs are two feet away from the motherboard and there's an optical connection to it, shouldn't it be fast? Are data centers putting SSDs super far away from motherboards?

> One theory is that EC2 intentionally caps the write speed at 1 GB/s to avoid frequent device failure, given the total number of writes per SSD is limited. This is the theory that I would bet on because it lines up with their bottom line.

But the sentence right after undermines it.

> However, this does not explain why the read bandwidth is stuck at 2 GB/s.

Faster read speeds would give them a more enticing product without wearing drives out.

Re: SSDs have become fast, except in the cloud

#77

Post by nostrademons »

pclmulqdq wrote: post #11

This was a huge technical problem I worked on at Google, and is sort of fundamental to a cloud. I believe this is actually a big deal that drives peoples' technology directions. SSDs in the cloud are attached over a network, and fundamentally have to be. The problem is that this network is so large and slow that it can't give you anywhere near the performance of a local SSD. This wasn't a problem for hard drives, whi…

Makes me wonder if we're on the crux of a shift back to client-based software. Historically changes in the relative cost of computing components have driven most of the shifts in the computing industry. Cheap teletypes & peripherals fueled the shift from batch-processing mainframes to timesharing minicomputers. Cheap CPUs & RAM fueled the shift from minicomputers to microcomputers. Cheap and fast networking fueled the shift from desktop software to the cloud. Will cheap SSDs & TPU/GPUs fuel a shift back toward thicker clients?

There are a bunch of supporting social trends toward this as well. Renewed emphasis on privacy. Big Tech canceling beloved products, bricking devices, and generally enshittifying everything - a lot of people want locally-controlled software that isn't going to get worse at the next update. Ever-rising prices which make people want to lock in a price for the device and not deal with increasing rents for computing power.

Re: SSDs have become fast, except in the cloud

#78

Post by silverquiet »

_Rabs_ wrote: post #15
pclmulqdq wrote: post #11

This was a huge technical problem I worked on at Google, and is sort of fundamental to a cloud. I believe this is actually a big deal that drives peoples' technology directions. SSDs in the cloud are attached over a network, and fundamentally have to be. The problem is that this network is so large and slow that it can't give you anywhere near the performance of a local SSD. This wasn't a problem for hard drives, whi…

So much of this. The amount of times I've seen someone complain about slow DB performance when they're trying to connect to it from a different VPC, and bottlenecking themselves to 100Mbits is stupidly high. Literally depending on where things are in a data center... If you're looking for closely coupled and on a 10G line on the same switch, going to the same server rack. I bet you performance will be so much more co…

> Literally depending on where things are in a data center

I thought cloud was supposed to abstract this away? That's a bit of a sarcastic question from a long-time cloud skeptic, but... wasn't it?

Re: SSDs have become fast, except in the cloud

#79

Post by crazygringo »

ssl-3 wrote: post #30
Filligree wrote: post #18

Earlier quoted context omitted.

They don't. Some cloud providers (i.e. Hetzner) let you rent VMs with locally attached NVMe, which is dramatically faster than network-attached even factoring in the VM tax. Of course then you have a single point of failure, in the PCIe fabric of the machine you're running on if not the NVMe itself. But if you have good backups, which you should, then the juice really isn't worth the squeeze for NAS storage.

A network adds more points of failure. It does not reduce them.

A network adds more points of failures but also reduces user-facing failures overall when properly architected.

If one CPU attached to storage dies, another can take over and reattach -- or vice-versa. If one network link dies, it can be rerouted around.

Re: SSDs have become fast, except in the cloud

#80

Post by 20after4 »

jsnell wrote: post #71
pclmulqdq wrote: post #42

Earlier quoted context omitted.

It's because the "local" SSDs are not actually physically attached and there's a network protocol in the way.

I think you're wrong about that. AWS calls this class of storage "instance storage" [0], and defines it as: > Many Amazon EC2 instances can also include storage from devices that are located inside the host computer, referred to as instance storage. There might be some wiggle room in "physically attached", but there's none in "storage devices located inside the host computer". It's not some kind of AWS-only thing eit…

If the SSD is installed in the host server, doesn't that still allow for it to be shared among many instances running on said host? I can imagine that a compute node has just a handful of SSDs and many hundreds of instances sharing the I/O bandwidth.
Post reply on HN