Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

SSDs have become fast, except in the cloud

databasearchitects.blogspot.com

Post reply on HN
Print view View original

81–90 of 427 posts

Re: SSDs have become fast, except in the cloud

#81

Post by Nextgrid »

crazygringo wrote: post #69
jsnell wrote: post #28

Earlier quoted context omitted.

According to the submitted article, the numbers are from AWS instance types where the SSD is "physically attached" to the host, not about SSD-backed NAS solutions. Also, the article isn't just about SSDs being no faster than a network. It's about SSDs being two orders of magnitude slower than datacenter networks.

> It's about SSDs being two orders of magnitude slower than datacenter networks. Could that have to do with every operation requiring a round trip, rather than being able to queue up operations in a buffer to saturate throughput? It seems plausible if the interface protocol was built for a device it assumed was physically local and so waited for confirmation after each operation before performing the next. In this ca…

Underlying protocol limitations wouldn't be an issue - the cloud provider's implementation can work around that. They're unlikely to be sending sequential SCSI/NVMe commands over the wire - instead, the hypervisor pretends to be the NVME device, but then converts to some internal protocol (that's less chatty and can coalesce requests without waiting on individual ACKs) before sending that to the storage server.

The problem is that ultimately your application often requires the outcome of a given IO operation to decide which operation to perform next - let's say when it comes to a database, it should first read the index (and wait for that to complete) before it knows the on-disk location of the actual row data which it needs to be able to issue the next IO operation.

In this case, there's no other solution than to move that application closer to the data itself. Instead of the networked storage node being a dumb blob storage returning bytes, the networked "storage" node is your database itself, returning query results. I believe that's what RDS Aurora does for example, every storage node can itself understand query predicates.

Re: SSDs have become fast, except in the cloud

#83

Post by doubled112 »

silverquiet wrote: post #78
_Rabs_ wrote: post #15

Earlier quoted context omitted.

So much of this. The amount of times I've seen someone complain about slow DB performance when they're trying to connect to it from a different VPC, and bottlenecking themselves to 100Mbits is stupidly high. Literally depending on where things are in a data center... If you're looking for closely coupled and on a 10G line on the same switch, going to the same server rack. I bet you performance will be so much more co…

> Literally depending on where things are in a data center I thought cloud was supposed to abstract this away? That's a bit of a sarcastic question from a long-time cloud skeptic, but... wasn't it?

Reality always beats the abstraction. After all, it's just somebody else's computer in somebody else's data center.

Re: SSDs have become fast, except in the cloud

#84

Post by bombcar »

_Rabs_ wrote: post #8
bombcar wrote: post #3

I think the obvious answer is there's not much demand, and keeping it "low" allows trickery and funny business with the virtualization layer (think: SAN, etc) that you can't do with "raw hardware speed".

Sure, but it does make me wonder what kind of speeds we are paying for if we can't even get raw hardware speeds... Sounds like one more excuse for AWS to obfuscate any meaning in their billing structure and take control of the narrative. How much are they getting away with by virtualization. (Think how banks use your money for loans and stuff) You actually don't get to really see the internals other than IOPS which d…

The biggest "scam" if you can call it that is reducing all factors of CPU performance to "cores".

Re: SSDs have become fast, except in the cloud

#85

Post by supriyo-biswas »

mkoubaa wrote: post #25
pclmulqdq wrote: post #11

This was a huge technical problem I worked on at Google, and is sort of fundamental to a cloud. I believe this is actually a big deal that drives peoples' technology directions. SSDs in the cloud are attached over a network, and fundamentally have to be. The problem is that this network is so large and slow that it can't give you anywhere near the performance of a local SSD. This wasn't a problem for hard drives, whi…

Dumb question. Why does the network have to be slow? If the SSDs are two feet away from the motherboard and there's an optical connection to it, shouldn't it be fast? Are data centers putting SSDs super far away from motherboards?

It’s not the network being slow, but dividing the available network bandwidth amongst all users, while also distributing the written data to multiple nodes reliably so that one tenant doesn’t hog resources is quite challenging. The pricing structure is meant to control resource usage; a discussion of the exact prices and how much profit AWS or any other cloud provider makes is a separate discussion.

Re: SSDs have become fast, except in the cloud

#86

Post by rfoo »

pclmulqdq wrote: post #63
candiddevmike wrote: post #55

Earlier quoted context omitted.

Depends on the cloud provider. Local SSDs are physically attached to the host on GCP, but that makes them only useful for temporary storage.

If you're at G, you should read the internal docs on exactly how this happens and it will be interesting.

Why would I lose all data on these SSDs when I initiate a power off of the VM on console, then?

I believe local SSDs are definitely attached to the host. They are just not exposed via NVMe ZNS hence the performance hit.

Re: SSDs have become fast, except in the cloud

#87

Post by bombcar »

doubled112 wrote: post #83
silverquiet wrote: post #78

Earlier quoted context omitted.

> Literally depending on where things are in a data center I thought cloud was supposed to abstract this away? That's a bit of a sarcastic question from a long-time cloud skeptic, but... wasn't it?

Reality always beats the abstraction. After all, it's just somebody else's computer in somebody else's data center.

Which can cause considerable "amusement" depending on the provider - one I won't name directly but is much more centered on actual renting racks than their (now) cloud offering - if you had a virtual machine older than a year or so, deleting and restoring it would get you on a newer "host" and you'd be faster for the same cost.

Otherwise it'd stay on the same physical piece of hardware it was allocated to when new.

Re: SSDs have become fast, except in the cloud

#88

Post by sprachspiel »

nik_0_0 wrote: post #75
sprachspiel wrote: post #44

Earlier quoted context omitted.

This is for 8 SSDs and a single modern PCIe 5.0 has better specs than this.

Is it? The line preceding the bullet list on that page seems to state otherwise: “” Each storage volume can deliver the following performance (all measured using 4 KiB blocks): * Up to 8000 MB/second of sequential reads “”

Just tested a i4i.32xlarge:

  $ lsblk
  NAME         MAJ:MIN RM   SIZE RO TYPE MOUNTPOINTS
  loop0          7:0    0  24.9M  1 loop /snap/amazon-ssm-agent/7628
  loop1          7:1    0  55.7M  1 loop /snap/core18/2812
  loop2          7:2    0  63.5M  1 loop /snap/core20/2015
  loop3          7:3    0 111.9M  1 loop /snap/lxd/24322
  loop4          7:4    0  40.9M  1 loop /snap/snapd/20290
  nvme0n1      259:0    0     8G  0 disk 
  ├─nvme0n1p1  259:1    0   7.9G  0 part /
  ├─nvme0n1p14 259:2    0     4M  0 part 
  └─nvme0n1p15 259:3    0   106M  0 part /boot/efi
  nvme2n1      259:4    0   3.4T  0 disk 
  nvme4n1      259:5    0   3.4T  0 disk 
  nvme1n1      259:6    0   3.4T  0 disk 
  nvme5n1      259:7    0   3.4T  0 disk 
  nvme7n1      259:8    0   3.4T  0 disk 
  nvme6n1      259:9    0   3.4T  0 disk 
  nvme3n1      259:10   0   3.4T  0 disk 
  nvme8n1      259:11   0   3.4T  0 disk
Since nvme0n1 is the EBS boot volume, we have 8 SSDs. And here's the read bandwidth for one of them:

  $ sudo fio --name=bla --filename=/dev/nvme2n1 --rw=read --iodepth=128 --ioengine=libaio --direct=1 --blocksize=16m
  bla: (g=0): rw=read, bs=(R) 16.0MiB-16.0MiB, (W) 16.0MiB-16.0MiB, (T) 16.0MiB-16.0MiB, ioengine=libaio, iodepth=128
  fio-3.28
  Starting 1 process
  ^Cbs: 1 (f=1): [R(1)][0.5%][r=2704MiB/s][r=169 IOPS][eta 20m:17s]
So we should have a total bandwidth of 2.7*8=21 GB/s. Not that great for 2024.

Re: SSDs have become fast, except in the cloud

#89

Post by baq »

jalk wrote: post #56
SteveNuts wrote: post #35

Earlier quoted context omitted.

Because even if you can squeeze 100TB or more of SSD/NVMe in a server, and there are 10 tenants using the machine, you're limited to 10TB as a hard ceiling. What happens when one tenant needs 200TB attached to a server? Cloud providers are starting to offer local SSD/NVMe, but you're renting the entire machine, and you're still limited to exactly what's installed in that server.

How is that different from how cores, mem and network bandwidth is allotted to tenants?

It isn't. You could ask for network-attached CPUs or RAM. You'd be the only one, though, so in practice only network-attached storage makes sense business-wise. It also makes sense if you need to provision larger-than-usual amounts like tens of TB - these are usually hard to come by in a single server, but quite mundane for storage appliances.

Re: SSDs have become fast, except in the cloud

#90

Post by bombcar »

crazygringo wrote: post #79
ssl-3 wrote: post #30

Earlier quoted context omitted.

A network adds more points of failure. It does not reduce them.

A network adds more points of failures but also reduces user-facing failures overall when properly architected. If one CPU attached to storage dies, another can take over and reattach -- or vice-versa. If one network link dies, it can be rerouted around.

Using a SAN (which is what networked storage is, after all) also lets you get various "tricks" such as snapshots, instant migration, etc for "free".
Post reply on HN