Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

SSDs have become fast, except in the cloud

databasearchitects.blogspot.com

Post reply on HN
Print view View original

61–70 of 427 posts

Re: SSDs have become fast, except in the cloud

#61

Post by nixass »

bugbuddy wrote: post #27
_Rabs_ wrote: post #15

Earlier quoted context omitted.

So much of this. The amount of times I've seen someone complain about slow DB performance when they're trying to connect to it from a different VPC, and bottlenecking themselves to 100Mbits is stupidly high. Literally depending on where things are in a data center... If you're looking for closely coupled and on a 10G line on the same switch, going to the same server rack. I bet you performance will be so much more co…

Aren’t 10G and 100G connections standard nowadays in data centers? Heck, I thought they were standard 10 years ago.

400G is fairly normal thing in DCs nowadays

Re: SSDs have become fast, except in the cloud

#62

Post by pixl97 »

bugbuddy wrote: post #27
_Rabs_ wrote: post #15

Earlier quoted context omitted.

So much of this. The amount of times I've seen someone complain about slow DB performance when they're trying to connect to it from a different VPC, and bottlenecking themselves to 100Mbits is stupidly high. Literally depending on where things are in a data center... If you're looking for closely coupled and on a 10G line on the same switch, going to the same server rack. I bet you performance will be so much more co…

Aren’t 10G and 100G connections standard nowadays in data centers? Heck, I thought they were standard 10 years ago.

Bandwidth delay product does not help serialized transactions. If you're reaching out to disk for results, or if you have locking transactions on a table the achievable operations drops dramatically as latency between the host and the disk increases.

Re: SSDs have become fast, except in the cloud

#63

Post by pclmulqdq »

candiddevmike wrote: post #55
pclmulqdq wrote: post #42

Earlier quoted context omitted.

It's because the "local" SSDs are not actually physically attached and there's a network protocol in the way.

Depends on the cloud provider. Local SSDs are physically attached to the host on GCP, but that makes them only useful for temporary storage.

If you're at G, you should read the internal docs on exactly how this happens and it will be interesting.

Re: SSDs have become fast, except in the cloud

#64

Post by [deleted] »

pclmulqdq wrote: post #11

This was a huge technical problem I worked on at Google, and is sort of fundamental to a cloud. I believe this is actually a big deal that drives peoples' technology directions. SSDs in the cloud are attached over a network, and fundamentally have to be. The problem is that this network is so large and slow that it can't give you anywhere near the performance of a local SSD. This wasn't a problem for hard drives, whi…

[deleted]

Re: SSDs have become fast, except in the cloud

#65

Post by pixl97 »

jalk wrote: post #56
SteveNuts wrote: post #35

Earlier quoted context omitted.

Because even if you can squeeze 100TB or more of SSD/NVMe in a server, and there are 10 tenants using the machine, you're limited to 10TB as a hard ceiling. What happens when one tenant needs 200TB attached to a server? Cloud providers are starting to offer local SSD/NVMe, but you're renting the entire machine, and you're still limited to exactly what's installed in that server.

How is that different from how cores, mem and network bandwidth is allotted to tenants?

Because a fair number of customers spin up another image when cores/mem/bandwidth run low. Dedicated storage breaks that paradigm.

Also, adding, if I am on an 8 core machine and need 16, network storage can be detached from host A and connected to host B. In dedicated storage it must be fully copied over first.

Re: SSDs have become fast, except in the cloud

#66

Post by colechristensen »

pclmulqdq wrote: post #42
jsnell wrote: post #28

Earlier quoted context omitted.

According to the submitted article, the numbers are from AWS instance types where the SSD is "physically attached" to the host, not about SSD-backed NAS solutions. Also, the article isn't just about SSDs being no faster than a network. It's about SSDs being two orders of magnitude slower than datacenter networks.

It's because the "local" SSDs are not actually physically attached and there's a network protocol in the way.

For AWS there are EBS volumes attached through a custom hardware NVMe interface and then there's Instance Store which is actually local SSD storage. These are different things.

https://docs.aws.amazon.com/AWSEC2/latest/UserGuide/Instance...

Re: SSDs have become fast, except in the cloud

#67

Post by jeffbee »

sprachspiel wrote: post #44
zokier wrote: post #16

> Since then, several NVMe instance types, including i4i and im4gn, have been launched. Surprisingly, however, the performance has not increased; seven years after the i3 launch, we are still stuck with 2 GB/s per SSD. AWS marketing claims otherwise: Up to 800K random write IOPS Up to 1 million random read IOPS Up to 5600 MB/second of sequential writes Up to 8000 MB/second of sequential reads https://aws.amazon.com/b…

This is for 8 SSDs and a single modern PCIe 5.0 has better specs than this.

Those claims are per device. There isn't even an instance in that family with 8 devices.

Re: SSDs have become fast, except in the cloud

#68

Post by jrullman »

pclmulqdq wrote: post #42
jsnell wrote: post #28

Earlier quoted context omitted.

According to the submitted article, the numbers are from AWS instance types where the SSD is "physically attached" to the host, not about SSD-backed NAS solutions. Also, the article isn't just about SSDs being no faster than a network. It's about SSDs being two orders of magnitude slower than datacenter networks.

It's because the "local" SSDs are not actually physically attached and there's a network protocol in the way.

I can attest to the fact that on EC2, "instance store" volumes are actually physically attached.

Re: SSDs have become fast, except in the cloud

#69

Post by crazygringo »

jsnell wrote: post #28
pclmulqdq wrote: post #11

This was a huge technical problem I worked on at Google, and is sort of fundamental to a cloud. I believe this is actually a big deal that drives peoples' technology directions. SSDs in the cloud are attached over a network, and fundamentally have to be. The problem is that this network is so large and slow that it can't give you anywhere near the performance of a local SSD. This wasn't a problem for hard drives, whi…

According to the submitted article, the numbers are from AWS instance types where the SSD is "physically attached" to the host, not about SSD-backed NAS solutions. Also, the article isn't just about SSDs being no faster than a network. It's about SSDs being two orders of magnitude slower than datacenter networks.

> It's about SSDs being two orders of magnitude slower than datacenter networks.

Could that have to do with every operation requiring a round trip, rather than being able to queue up operations in a buffer to saturate throughput?

It seems plausible if the interface protocol was built for a device it assumed was physically local and so waited for confirmation after each operation before performing the next.

In this case it's not so much the throughput rate that matters, but the latency -- which can also be heavily affected by buffering of other network traffic.

Post reply on HN