Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

SSDs have become fast, except in the cloud

databasearchitects.blogspot.com

Post reply on HN
Print view View original

161–170 of 427 posts

Re: SSDs have become fast, except in the cloud

#161

Post by Dylan16807 »

bugbuddy wrote: post #133
Dylan16807 wrote: post #76

Earlier quoted context omitted.

But the sentence right after undermines it. > However, this does not explain why the read bandwidth is stuck at 2 GB/s. Faster read speeds would give them a more enticing product without wearing drives out.

They may be limiting the read artificially to increase your resource utilization else where. If you have disk bottleneck then you would be more likely to use more instances. It is still about the bottom line.

That could be. But it's a completely different reason. If you summarize everything as "bottom line", you lose all the valuable information.

Re: SSDs have become fast, except in the cloud

#162

Post by crotchfire »

pclmulqdq wrote: post #42
jsnell wrote: post #28

Earlier quoted context omitted.

According to the submitted article, the numbers are from AWS instance types where the SSD is "physically attached" to the host, not about SSD-backed NAS solutions. Also, the article isn't just about SSDs being no faster than a network. It's about SSDs being two orders of magnitude slower than datacenter networks.

It's because the "local" SSDs are not actually physically attached and there's a network protocol in the way.

This is incorrect.

Amazon offers both locally-attached storage devices as well as instance-attached storage devices. The article is about the latter kind.

Re: SSDs have become fast, except in the cloud

#163

Post by dijit »

mike_hearn wrote: post #137
supriyo-biswas wrote: post #109

Earlier quoted context omitted.

RAID rebuild times make it an unviable option and customers typically expect problematic VMs to be live-migrated to other hosts with the disks still having their intended data. The self hosted version of this is GlusterFS and Ceph, which have the same dynamics as EBS and its equivalents in other cloud providers.

With NVMe SSDs? What makes RAID unviable in that environment?

This depends, like all things.

When you say RAID, what level? Software-raid or hardware raid? What controller?

Let's take best-case:

RAID10, small enough (but many) NVMe drives and an LVM/Software RAID like ZFS, which is data aware so only rebuilds actual data: rebuilds will degrade performance enough potentially that your application can become unavailable if your IOPS are 70%+ of maximum.

That's an ideal scenario, if you use hardware raid which is not data-aware then your rebuild times depend entirely on the size of the drive being rebuilt and it can punish IOPs even more during the rebuild. But it will affect your CPU less.

There's no panacea. Most people opt for higher latency distributed storage where the RAID is spread across an enormous amount of drives, which makes rebuilds much less painful.

Re: SSDs have become fast, except in the cloud

#164

Post by pixl97 »

bee_rider wrote: post #120
pixl97 wrote: post #62

Earlier quoted context omitted.

Bandwidth delay product does not help serialized transactions. If you're reaching out to disk for results, or if you have locking transactions on a table the achievable operations drops dramatically as latency between the host and the disk increases.

The typical way to trade bandwidth away for latency would, I guess, be speculative requests. In the CPU world at least. I wonder if any cloud providers have some sort of framework built around speculative disk reads (or maybe it is a totally crazy trade to make in this context)?

I mean we already have readahead in the kernel.

This said the problem can get more complex than this really fast. Write barriers for example and dirty caches. Any application that forces writes and the writes are enforced by the kernel are going to suffer.

The same is true for SSD settings. There are a number of tweakable values on SSDs when it comes to write commit and cache usage which can affect performance. Desktop OS's tend to play more fast and loose with these settings and servers defaults tend to be more conservative.

Re: SSDs have become fast, except in the cloud

#165

Post by queuebert »

20after4 wrote: post #80
jsnell wrote: post #71

Earlier quoted context omitted.

I think you're wrong about that. AWS calls this class of storage "instance storage" [0], and defines it as: > Many Amazon EC2 instances can also include storage from devices that are located inside the host computer, referred to as instance storage. There might be some wiggle room in "physically attached", but there's none in "storage devices located inside the host computer". It's not some kind of AWS-only thing eit…

If the SSD is installed in the host server, doesn't that still allow for it to be shared among many instances running on said host? I can imagine that a compute node has just a handful of SSDs and many hundreds of instances sharing the I/O bandwidth.

How do these machines manage the sharing of one local SSD across multiple VMs? Is there some wrapper around the I/O stack? Does it appear as a network share? Geniuinely curious...

Re: SSDs have become fast, except in the cloud

#166

Post by mint2 »

dekhn wrote: post #126
pclmulqdq wrote: post #42

Earlier quoted context omitted.

It's because the "local" SSDs are not actually physically attached and there's a network protocol in the way.

I suspect you must be conflating several different storage products. Are you saying https://cloud.google.com/compute/docs/disks/local-ssd devices talk to the host through a network (say, ethernet with some layer on top)? Because the documentation very clearly says otherwise, "This is because Local SSD disks are physically attached to the server that hosts your VM. For this same reason, Local SSD disks can only provid…

I don’t really agree with assuming the form of physical attachment and interaction unless it is spelled out.

If that’s what’s meant it will be stated in some fine print, if it’s not stated anywhere then there is no guarantee what the term means, except I would guess they may want people to infer things that may not necessarily be true.

Re: SSDs have become fast, except in the cloud

#167

Post by KaiserPro »

bugbuddy wrote: post #27
_Rabs_ wrote: post #15

Earlier quoted context omitted.

So much of this. The amount of times I've seen someone complain about slow DB performance when they're trying to connect to it from a different VPC, and bottlenecking themselves to 100Mbits is stupidly high. Literally depending on where things are in a data center... If you're looking for closely coupled and on a 10G line on the same switch, going to the same server rack. I bet you performance will be so much more co…

Aren’t 10G and 100G connections standard nowadays in data centers? Heck, I thought they were standard 10 years ago.

Yes, but you have to think about contention. Whilst the Top of rack might have 2x400 gig links to the core, thats shared with the entire rack, and all the other machines trying to shout at the core switching infra.

Then stuff goes away, or route congested, etc, etc, etc.

Re: SSDs have become fast, except in the cloud

#168

Post by taneq »

SteveNuts wrote: post #35
vlovich123 wrote: post #14

Earlier quoted context omitted.

Why do they fundamentally need to be network attached storage instead of local to the VM?

Because even if you can squeeze 100TB or more of SSD/NVMe in a server, and there are 10 tenants using the machine, you're limited to 10TB as a hard ceiling. What happens when one tenant needs 200TB attached to a server? Cloud providers are starting to offer local SSD/NVMe, but you're renting the entire machine, and you're still limited to exactly what's installed in that server.

> What happens when one tenant needs 200TB attached to a server?

Link to this mythical hosting service that expects far less than 200TB of data per client but just pulls a sad face and takes the extra cost on board when a client demands it. :D

Re: SSDs have become fast, except in the cloud

#169

Post by samstave »

pclmulqdq wrote: post #11

This was a huge technical problem I worked on at Google, and is sort of fundamental to a cloud. I believe this is actually a big deal that drives peoples' technology directions. SSDs in the cloud are attached over a network, and fundamentally have to be. The problem is that this network is so large and slow that it can't give you anywhere near the performance of a local SSD. This wasn't a problem for hard drives, whi…

Forgive if stupid, but when netflix was doing all their edge content boxes, where they were putting machines much more latency-close to customers... is/does/can this model kinda work for SSDs in a SScDN type of network (client---->CDN->SScDN-------SSD?

Re: SSDs have become fast, except in the cloud

#170

Post by bfncieezo »

pclmulqdq wrote: post #42
jsnell wrote: post #28

Earlier quoted context omitted.

According to the submitted article, the numbers are from AWS instance types where the SSD is "physically attached" to the host, not about SSD-backed NAS solutions. Also, the article isn't just about SSDs being no faster than a network. It's about SSDs being two orders of magnitude slower than datacenter networks.

It's because the "local" SSDs are not actually physically attached and there's a network protocol in the way.

instances can have block storage which is network attached, or local attached ssd/nvme. its 2 separate things.
Post reply on HN