Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

SSDs have become fast, except in the cloud

databasearchitects.blogspot.com

Post reply on HN
Print view View original

311–320 of 427 posts

Re: SSDs have become fast, except in the cloud

#311

Post by vlovich123 »

pclmulqdq wrote: post #23
vlovich123 wrote: post #14

Earlier quoted context omitted.

Why do they fundamentally need to be network attached storage instead of local to the VM?

Reliability. SSDs break and screw up a lot more frequently and more quickly than CPUs. Amazon has published a lot on the architecture of EBS, and they go through a good analysis of this. If you have a broken disk and you locally attach, you have a broken machine. RAID helps you locally, but fundamentally relies on locality and low latency (and maybe custom hardware) to minimize the time window where you get true data…

Sure, but there's plenty of software that's written to use distributed unreliable storage similar to how cloud providers write their own software (e.g. Kafka). I can understand if many applications are just need something like EBS that's durable but looks like a normal disk, but not so sure it's a fundamentally required abstraction.

Re: SSDs have become fast, except in the cloud

#312

Post by SteveNuts »

adgjlsfhk1 wrote: post #232
dekhn wrote: post #205

Earlier quoted context omitted.

"Physically attached" has had a fairly well defined meaning and i don't normally expect a cloud provider to play word salad to convince me a network drive is locally attached (like I said, if true, I would need to have a chat with my TAM about it). Physically attached for servers, for the past 20+ years, has meant a direct electrical connection to a host bus (such as the PCI bus attached to the front-side bus). I'd l…

Ethernet cables are physical...

If that’s the game we’re going to play then technically my driveway is on the same road as the White House.

Re: SSDs have become fast, except in the cloud

#313

Post by amir734jj »

pclmulqdq wrote: post #11

This was a huge technical problem I worked on at Google, and is sort of fundamental to a cloud. I believe this is actually a big deal that drives peoples' technology directions. SSDs in the cloud are attached over a network, and fundamentally have to be. The problem is that this network is so large and slow that it can't give you anywhere near the performance of a local SSD. This wasn't a problem for hard drives, whi…

Kind of funny but we use similar idea in Azure.

Re: SSDs have become fast, except in the cloud

#314

Post by jsolson »

pclmulqdq wrote: post #11

This was a huge technical problem I worked on at Google, and is sort of fundamental to a cloud. I believe this is actually a big deal that drives peoples' technology directions. SSDs in the cloud are attached over a network, and fundamentally have to be. The problem is that this network is so large and slow that it can't give you anywhere near the performance of a local SSD. This wasn't a problem for hard drives, whi…

This is untrue of Local SSD (https://cloud.google.com/local-ssd) in Google Cloud. Local SSDs are PCIe peripherals, not network attached.

There are also multiple Persistent Disk (https://cloud.google.com/persistent-disk) offerings that are backed by SSDs over the network.

(I'm an engineer on GCE. I work directly on the physical hardware that backs our virtualization platform.)

Re: SSDs have become fast, except in the cloud

#315

Post by pzb »

pclmulqdq wrote: post #294
dekhn wrote: post #218

Earlier quoted context omitted.

it sounds like you're trying to say "PCI switch" without saying "PCI switch" (I worked at Google for over a decade, including hardware division).

That is what I am trying to say without actually giving it out. PCIe switches are very much not transparent devices. Apparently AWS has not published anything about this, and doesn't have Nitro moderating access to "local" SSD, though - that I did get confused with EBS.

AWS has stated that there is a "Nitro Card for Instance Storage"[0][1] which is a NVMe PCIe controller that implements transparent encryption[2].

I don't have access to an EC2 instance to check, but you should be able to see the PCIe topology to determine how many physical cards are likely in i4i and im4gn and their PCIe connections. i4i claims to have 8 x 3,750 AWS Nitro SSD, but it isn't clear how many PCIe lanes are used.

Also, AWS claims "Traditionally, SSDs maximize the peak read and write I/O performance. AWS Nitro SSDs are architected to minimize latency and latency variability of I/O intensive workloads [...] which continuously read and write from the SSDs in a sustained manner, for fast and more predictable performance. AWS Nitro SSDs deliver up to 60% lower storage I/O latency and up to 75% reduced storage I/O latency variability [...]"

This could explain the findings in the article - they only meared peak r/w, not predictability.

[0] https://perspectives.mvdirona.com/2019/02/aws-nitro-system/ [1] https://aws.amazon.com/ec2/nitro/ [2] https://d1.awsstatic.com/events/reinvent/2019/REPEAT_2_Power...

Re: SSDs have become fast, except in the cloud

#316

Post by jsolson »

AlgorithmicTime wrote: post #123
pclmulqdq wrote: post #11

This was a huge technical problem I worked on at Google, and is sort of fundamental to a cloud. I believe this is actually a big deal that drives peoples' technology directions. SSDs in the cloud are attached over a network, and fundamentally have to be. The problem is that this network is so large and slow that it can't give you anywhere near the performance of a local SSD. This wasn't a problem for hard drives, whi…

> SSDs in the cloud are attached over a network, and fundamentally have to be. No they don't. I work for a cloud provider and I can guarantee that your SSD is local to your VM.

This is also true for GCE Local SSD: https://cloud.google.com/local-ssd

The GP is incorrect.

Re: SSDs have become fast, except in the cloud

#317

Post by jsolson »

zokier wrote: post #22
pclmulqdq wrote: post #11

This was a huge technical problem I worked on at Google, and is sort of fundamental to a cloud. I believe this is actually a big deal that drives peoples' technology directions. SSDs in the cloud are attached over a network, and fundamentally have to be. The problem is that this network is so large and slow that it can't give you anywhere near the performance of a local SSD. This wasn't a problem for hard drives, whi…

> SSDs in the cloud are attached over a network, and fundamentally have to be Not on AWS. Instance stores (what the article is about) are physical local disks.

Same for Google (the GP is incorrect about how GCE Local SSD works).

Re: SSDs have become fast, except in the cloud

#318

Post by jsolson »

pclmulqdq wrote: post #63
candiddevmike wrote: post #55

Earlier quoted context omitted.

Depends on the cloud provider. Local SSDs are physically attached to the host on GCP, but that makes them only useful for temporary storage.

If you're at G, you should read the internal docs on exactly how this happens and it will be interesting.

In most cases, they're physically plugged into a PCIe CEM slot in the host.

There is no network in the way, you are either misinformed or thinking of a different product.

Re: SSDs have become fast, except in the cloud

#319

Post by rwiggins »

deadmutex wrote: post #141
mike_hearn wrote: post #134

Earlier quoted context omitted.

Sure, I understand that, but this user is claiming that on GCP even local SSDs aren't really local, which raises the question of why not. I suspect the answer is something to do with their manufacturing processes/rack designs. When I worked there (pre GCP) machines had only a tiny disk used for booting and they wanted to get rid of that. Storage was handled by "diskful" machines that had dedicated trays of HDDs conne…

The GCE claim is unverified. OP seems to be referring to PD-SSD and not LocalSSD

GCE local SSDs absolutely are on the same host as the VM. The docs [0] are pretty clear on this, I think:

> Local SSD disks are physically attached to the server that hosts your VM.

Disclosure: I work on GCE.

[0] https://cloud.google.com/compute/docs/disks/local-ssd

Re: SSDs have become fast, except in the cloud

#320

Post by adgjlsfhk1 »

SteveNuts wrote: post #312
adgjlsfhk1 wrote: post #232

Earlier quoted context omitted.

Ethernet cables are physical...

If that’s the game we’re going to play then technically my driveway is on the same road as the White House.

exactly. it's not about what's good for the consumer, it's about what they can do without losing a lawsuit for false advertising.
Post reply on HN