Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

SSDs have become fast, except in the cloud

databasearchitects.blogspot.com

Post reply on HN
Print view View original

91–100 of 427 posts

Re: SSDs have become fast, except in the cloud

#91

Post by discodave »

20after4 wrote: post #80
jsnell wrote: post #71

Earlier quoted context omitted.

I think you're wrong about that. AWS calls this class of storage "instance storage" [0], and defines it as: > Many Amazon EC2 instances can also include storage from devices that are located inside the host computer, referred to as instance storage. There might be some wiggle room in "physically attached", but there's none in "storage devices located inside the host computer". It's not some kind of AWS-only thing eit…

If the SSD is installed in the host server, doesn't that still allow for it to be shared among many instances running on said host? I can imagine that a compute node has just a handful of SSDs and many hundreds of instances sharing the I/O bandwidth.

If you have one of the metal instance types, then you get the whole host, e.g. i4i.metal:

https://aws.amazon.com/ec2/instance-types/i4i/

Re: SSDs have become fast, except in the cloud

#92

Post by baq »

Nextgrid wrote: post #51
ejb999 wrote: post #19

Earlier quoted context omitted.

How much faster would the network need to get, in order to meet (or at least approach) the speed of a local SSD? are we talking about needing to 2x or 3x the speed, or by factors of hundreds or thousands?

The problem isn't necessarily speed, it's random access latency. What makes SSDs fast and "magical" is their low random-access latency compared to a spinning disk. The sequential-access read speed is merely a bonus. Networked storage negates that significantly, absolutely killing performance for certain applications. You could have a 100Gbps network and it still won't match a direct-attached SSD in terms of latency (…

100Gbps direct shouldn't be too bad, but it might be difficult to get anyone to sell it to you for exclusive usage in a vm...

Re: SSDs have become fast, except in the cloud

#93

Post by doubled112 »

bombcar wrote: post #87
doubled112 wrote: post #83

Earlier quoted context omitted.

Reality always beats the abstraction. After all, it's just somebody else's computer in somebody else's data center.

Which can cause considerable "amusement" depending on the provider - one I won't name directly but is much more centered on actual renting racks than their (now) cloud offering - if you had a virtual machine older than a year or so, deleting and restoring it would get you on a newer "host" and you'd be faster for the same cost. Otherwise it'd stay on the same physical piece of hardware it was allocated to when new.

Amusing is a good description.

"Hardware degradation detected, please turn it off and back on again"

I could do a migration with zero downtime in VMware for a decade but they can't seamlessly move my VM to a machine that works in 2024? Great, thanks. Amusing.

Re: SSDs have become fast, except in the cloud

#94

Post by Nextgrid »

jeffbee wrote: post #53
siliconc0w wrote: post #13

Core count plus modern nvme actually make a great case for moving away from the cloud- before it was, "your data probably fits into memory". These are so fast that they're close enough to memory so it's "your data surely fits on disk". This reduces the complexity of a lot of workloads so you can just buy a beefy server and do pretty insane caching/calculation/serving with just a single box or two for redundancy.

"I will simply have another box for redundancy" is already a system so complex that having it in or out of the cloud won't make a difference.

It really depends on business requirements. Real-time redundancy is hard. Taking backups at 15-min intervals and having the standby box merely pull down the last backup when starting up is much easier, and this may actually be fine for a lot of applications.

Unfortunately very few actually think about failure modes, set realistic targets, and actually test the process. Everyone thinks they need 100% uptime and consistency, few actually achieve it in practice (many think they do, but when shit hits the fan it uncovers an edge-case they haven't thought of), but it turns out that in most cases it doesn't matter and they could've saved themselves a lot of trouble and complexity.

Re: SSDs have become fast, except in the cloud

#95

Post by baq »

adrr wrote: post #38
pclmulqdq wrote: post #11

This was a huge technical problem I worked on at Google, and is sort of fundamental to a cloud. I believe this is actually a big deal that drives peoples' technology directions. SSDs in the cloud are attached over a network, and fundamentally have to be. The problem is that this network is so large and slow that it can't give you anywhere near the performance of a local SSD. This wasn't a problem for hard drives, whi…

If the local drives are network drives(eg: SAN) then why are they ephemeral?

live vm migrations, perhaps

Re: SSDs have become fast, except in the cloud

#96

Post by ssl-3 »

supriyo-biswas wrote: post #47
ssl-3 wrote: post #30

Earlier quoted context omitted.

A network adds more points of failure. It does not reduce them.

A network attached, replicated storage hedges against data loss but increases latency; however most customers usually prefer higher latency to data loss. As an example, see the highly upvoted fly.io thread[1] with customers complaining about the same thing. [1] https://news.ycombinator.com/item?id=36808296

Locally-attached, replicated storage also hedges against data loss.

Re: SSDs have become fast, except in the cloud

#97

Post by tw04 »

pclmulqdq wrote: post #11

This was a huge technical problem I worked on at Google, and is sort of fundamental to a cloud. I believe this is actually a big deal that drives peoples' technology directions. SSDs in the cloud are attached over a network, and fundamentally have to be. The problem is that this network is so large and slow that it can't give you anywhere near the performance of a local SSD. This wasn't a problem for hard drives, whi…

>The problem is that this network is so large and slow that it can't give you anywhere near the performance of a local SSD.

*As implemented in the public cloud providers.

You can absolutely get better than local disk speeds from SAN devices and we've been doing it for decades. To do it on-prem with flash devices will require NVMe over FC or Ethernet and an appropriate storage array. Modern all-flash array performance is measured in millions of IOPS.

Will there be a slight uptick in latency? Sure, but it's well worth it for the data services and capacity of an external array for nearly every workload.

Re: SSDs have become fast, except in the cloud

#98

Post by Nextgrid »

sprachspiel wrote: post #88
nik_0_0 wrote: post #75

Earlier quoted context omitted.

Is it? The line preceding the bullet list on that page seems to state otherwise: “” Each storage volume can deliver the following performance (all measured using 4 KiB blocks): * Up to 8000 MB/second of sequential reads “”

Just tested a i4i.32xlarge: $ lsblk NAME MAJ:MIN RM SIZE RO TYPE MOUNTPOINTS loop0 7:0 0 24.9M 1 loop /snap/amazon-ssm-agent/7628 loop1 7:1 0 55.7M 1 loop /snap/core18/2812 loop2 7:2 0 63.5M 1 loop /snap/core20/2015 loop3 7:3 0 111.9M 1 loop /snap/lxd/24322 loop4 7:4 0 40.9M 1 loop /snap/snapd/20290 nvme0n1 259:0 0 8G 0 disk ├─nvme0n1p1 259:1 0 7.9G 0 part / ├─nvme0n1p14 259:2 0 4M 0 part └─nvme0n1p15 259:3 0 106M 0…

If you still have this machine, I wonder if you can get this bandwidth in parallel across all SSDs? There could be some hypervisor-level or host-level bottleneck that means while any SSD in isolation will give you the observed bandwidth, you can't actually reach that if you try to access them all in parallel?

Re: SSDs have become fast, except in the cloud

#99

Post by rbranson »

It's awfully similar to a lane of PCIe 4.0, if we're talking about instance storage. It reads like they've chosen to map each physical device to a single PCIe lane. Surely the AWS Nitro hardware platform has longer cycle times than PCIe. Note that once the instance type has multiple block devices exposed (i.e. im4gn.8xlarge or higher), striping across these will reach higher throughput (2 devices yields 4G/s, 4 yields 8G/s).

Re: SSDs have become fast, except in the cloud

#100

Post by kwillets »

colechristensen wrote: post #66
pclmulqdq wrote: post #42

Earlier quoted context omitted.

It's because the "local" SSDs are not actually physically attached and there's a network protocol in the way.

For AWS there are EBS volumes attached through a custom hardware NVMe interface and then there's Instance Store which is actually local SSD storage. These are different things. https://docs.aws.amazon.com/AWSEC2/latest/UserGuide/Instance...

EBS is also slower than local NVMe mounts on i3's.

Also, both features use Nitro SSD cards, according to AWS docs. The Nitro architecture is all locally attached -- instance storage to the instance, EBS to the EBS server.

Post reply on HN