Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

SSDs have become fast, except in the cloud

databasearchitects.blogspot.com

Post reply on HN
Print view View original

201–210 of 427 posts

Re: SSDs have become fast, except in the cloud

#201

Post by hathawsh »

pclmulqdq wrote: post #42
jsnell wrote: post #28

Earlier quoted context omitted.

According to the submitted article, the numbers are from AWS instance types where the SSD is "physically attached" to the host, not about SSD-backed NAS solutions. Also, the article isn't just about SSDs being no faster than a network. It's about SSDs being two orders of magnitude slower than datacenter networks.

It's because the "local" SSDs are not actually physically attached and there's a network protocol in the way.

That seems like a big opportunity for other cloud providers. They could provide SSDs that are actually physically attached and boast (rightfully) that their SSDs are a lot faster, drawing away business from older cloud providers.

Re: SSDs have become fast, except in the cloud

#202

Post by computerdork »

jiggawatts wrote: post #187

There's a lot of talk about cloud network and disk performance in this thread. I recently benchmarked both Azure and AWS and found that: - Azure network latency is about 85 microseconds. - AWS network latency is about 55 microseconds. - Both can do better, but only in special circumstances such as RDMA NICs in HPC clusters. - Cross-VPC or cross-VNET is basically identical. Some people were saying it's terribly slow,…

Interesting. And would you happen to have the numbers on the performance of the local SSD? Is it's read and write throughput up to the level of modern SSD's?

Re: SSDs have become fast, except in the cloud

#203

Post by magicalhippo »

queuebert wrote: post #165
20after4 wrote: post #80

Earlier quoted context omitted.

If the SSD is installed in the host server, doesn't that still allow for it to be shared among many instances running on said host? I can imagine that a compute node has just a handful of SSDs and many hundreds of instances sharing the I/O bandwidth.

How do these machines manage the sharing of one local SSD across multiple VMs? Is there some wrapper around the I/O stack? Does it appear as a network share? Geniuinely curious...

In say VirtualBox you can create a file backed on the physical disk, and attach it to the VM so the VM sees it as a NVMe drive.

In my experience this is also orders of magnitude slower that true direct access, ie PCIe pass-through, as all access has to pass through the VM storage driver and so could explain what is happening.

Re: SSDs have become fast, except in the cloud

#204

Post by Salgat »

pclmulqdq wrote: post #42
jsnell wrote: post #28

Earlier quoted context omitted.

According to the submitted article, the numbers are from AWS instance types where the SSD is "physically attached" to the host, not about SSD-backed NAS solutions. Also, the article isn't just about SSDs being no faster than a network. It's about SSDs being two orders of magnitude slower than datacenter networks.

It's because the "local" SSDs are not actually physically attached and there's a network protocol in the way.

At first you'd think maybe they can do a volume copy from a snapshot to a local drive on instance creation but even at 100gbps you're looking at almost 3 minutes for a 2TB drive.

Re: SSDs have become fast, except in the cloud

#205

Post by dekhn »

mint2 wrote: post #166
dekhn wrote: post #126

Earlier quoted context omitted.

I suspect you must be conflating several different storage products. Are you saying https://cloud.google.com/compute/docs/disks/local-ssd devices talk to the host through a network (say, ethernet with some layer on top)? Because the documentation very clearly says otherwise, "This is because Local SSD disks are physically attached to the server that hosts your VM. For this same reason, Local SSD disks can only provid…

I don’t really agree with assuming the form of physical attachment and interaction unless it is spelled out. If that’s what’s meant it will be stated in some fine print, if it’s not stated anywhere then there is no guarantee what the term means, except I would guess they may want people to infer things that may not necessarily be true.

"Physically attached" has had a fairly well defined meaning and i don't normally expect a cloud provider to play word salad to convince me a network drive is locally attached (like I said, if true, I would need to have a chat with my TAM about it).

Physically attached for servers, for the past 20+ years, has meant a direct electrical connection to a host bus (such as the PCI bus attached to the front-side bus). I'd like to see some alternative examples that violate that convention.

Re: SSDs have become fast, except in the cloud

#206

Post by wmf »

KaiserPro wrote: post #172
scottlamb wrote: post #117

Earlier quoted context omitted.

> The problem is that this network is so large and slow that it can't give you anywhere near the performance of a local SSD. This wasn't a problem for hard drives, which was the backing technology when a lot of these network attached storage systems were invented, because they are fundamentally slow compared to networks, but it is a problem for SSD. Certainly true that SSD bandwidth and latency improvements are hard…

Networks are not reliable, despite what you hear, so latency is used to mask re-tries and delays. The other thing to note about big inter-DC links are heavily QoS'd and contented, because they are both expensive and a bollock to maintain. Also, from what I recall, 40gig links are just parallel 10 gig links, so have no lower latency. I'm not sure if 100/400 gigs are ten/fourty lines of ten gigs in parallel or actually…

40gig links are just parallel 10 gig links, so have no lower latency

That's not correct. Higher link speeds do have lower serialization latency, although that's a small fraction of overall network latency.

Re: SSDs have become fast, except in the cloud

#207

Post by wiredfool »

ianburrell wrote: post #118
mike_hearn wrote: post #58

Earlier quoted context omitted.

They do this because they want SSDs to be in a physically separate part of the building for operational reasons, or what's the point in giving you a "local" SSD that isn't actually plugged into the real machine?

The reason for having most instances use network storage is that it makes possible migrating instances to other hosts. If the host fails, the network storage can be pointed at the new host with a reboot. AWS sends out notices regularly when they are going to reboot or migrate instances. Their probably should be more local instance storage types for using with instances that can be recreated without loss. But it is si…

Are you saying that a reboot wipes the ephemeral disks? Or a stop the instance and start the instance from AWS console/api?

Re: SSDs have become fast, except in the cloud

#208

Post by justsomehnguy »

kstrauser wrote: post #106
fabioyy wrote: post #70

it is not worth to use cloud if you need a lot of iops/bandwidth heck, its not worth for anything besides scalability dedicated servers are wayyyy cheaper

I'm not certain that's true if you look at TCO. Yes, you can probably buy a server for less than the yearly rent on the equivalent EC2 instance. But then you've got to put that server somewhere, with reliable power and probably redundant Internet connections. You have to pay someone's salary to set it up and load it to the point that a user can SSH in and configure it. You have to maintain an inventory of spares, and…

> as cheaply as possible, reliability be damned. As soon as you start caring about keeping it in a business-ready state, costs start to skyrocket.

The demand for five-nines is greatly exaggerated.

Re: SSDs have become fast, except in the cloud

#209

Post by malfist »

disqard wrote: post #171
malfist wrote: post #115

Earlier quoted context omitted.

I totally hear you about that. I work for FAANG, and I'm working on a service that has to be capable of sending 1.6m text messages in less than 10 minutes. The amount of complexity the architecture has because of those constraints is insane. When I worked at my previous job, management kept asking for that scale of designs for less than 1/1000 of the throughput and I was constantly pushing back. There's real costs to…

I'm trying to guess what "OE" stands for... over engineering? operating expenditure? I'd love to know what you meant :)

Sorry, thought it was a common term. Operational Excellence. All the effort and time it takes to keep a service online, on call included

Re: SSDs have become fast, except in the cloud

#210

Post by ianburrell »

wiredfool wrote: post #207
ianburrell wrote: post #118

Earlier quoted context omitted.

The reason for having most instances use network storage is that it makes possible migrating instances to other hosts. If the host fails, the network storage can be pointed at the new host with a reboot. AWS sends out notices regularly when they are going to reboot or migrate instances. Their probably should be more local instance storage types for using with instances that can be recreated without loss. But it is si…

Are you saying that a reboot wipes the ephemeral disks? Or a stop the instance and start the instance from AWS console/api?

Reboot keeps the instance storage volumes. Restarting wipes them. Starting frequently migrates to new host. And the "restart" notices AWS sends are likely cause the host has a problem and need to migrate it.
Post reply on HN