Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

SSDs have become fast, except in the cloud

databasearchitects.blogspot.com

Post reply on HN
Print view View original

381–390 of 427 posts

Re: SSDs have become fast, except in the cloud

#381

Post by MichaelZuo »

stingraycharles wrote: post #293
jsnell wrote: post #199

Earlier quoted context omitted.

I've linked to public documentation that is pretty clearly in conflict with what you said. There's no wiggle room in how AWS describes their service without it being false advertising. There's no "ah, but what if we define the entire building to be the host computer, then the networked SSDs really are inside the host computer" sleight of hand to pull off here. You've provided cryptic hints and a suggestion to watch s…

You are correct, and the parent you’re replying to is confused. Nitro is for EBS, not the i3 local NVMe instances. Those i3 instances lose your data whenever you stop and start them again (ie migrate to a different host machine), there’s absolutely no reason they would use network. EBS itself uses a different network than the “normal” internet, if I were to guess it’s a converged Ethernet network optimized for iSCSI.…

The argument could also be resolved by just getting the latency numbers for both cases and compare them, on bare metal it shouldn't be more than a few hundred nanoseconds.

Re: SSDs have become fast, except in the cloud

#383

Post by abadpoli »

driftnet wrote: post #379
arrakeenrevived wrote: post #299

Earlier quoted context omitted.

Clouds (like, the big fluffy things in the sky) are made up of many droplets of liquid. Using "droplet" to refer to the things that make up cloud computing is a pretty natural nickname for any cloud provider, not just DO. I do imagine that DO uses "droplet" as a public product branding because it works well with their "Ocean" brand, though. ...now I'm actually interested in knowing if "droplet" is derived from "ocean…

Clouds are water vapor, not droplets.

“Cloud: Visible mass of liquid droplets or frozen crystals suspended in the atmosphere“

https://en.wikipedia.org/wiki/Cloud

Re: SSDs have become fast, except in the cloud

#384

Post by c0l0 »

tumult wrote: post #362
c0l0 wrote: post #359

Earlier quoted context omitted.

This happens for you after you invoked an explicit sync() (et al.) before the power cut?

Yep.

That is highly interesting and contrary to a number of reports I've read about the Samsung 970 EVO Plus Series (and experienced for myself) specifically! Can you share more details about your particular setup and methodology? (Specific model name/capacity, Firmware release, Kernel version, filesystem, mkfs and mount options, any relevant block layer funny business you are conciously setting would be of greatest interest.) Do you have more than one drive where this can happen?

Re: SSDs have become fast, except in the cloud

#385

Post by icedchai »

jethro_tell wrote: post #380
icedchai wrote: post #225

Earlier quoted context omitted.

With Linux and KVM/QEMU, you can map an entire physical disk, disk partition, or file to a block device in the VM. For my own VM hosts, I use LVM and map a logical volume to the VM. I assumed cloud providers did something conceptually similar, only much more sophisticated.

Heh, you'd probably be surprised, there's some really cool cutting edge stuff being done in those data centers but a lot of what is done is just plan old standard server management without much in the way of tricks. Its just someone else does it instead of you and the billing department is counting milliseconds.

Do cloud providers document these internals anywhere? I'd love to read about that sort of thing.

Re: SSDs have become fast, except in the cloud

#386

Post by jsolson »

barrkel wrote: post #369
jiggawatts wrote: post #336

Earlier quoted context omitted.

It's notable that your second link has a screenshot for 24(!) NVMe SSDs totalling 9 terabytes, but the aggregate performance is 2.4M IOPS and 9.3 GB/s for reads. In other words, just 100K/400MB per individual SSD, which is very low these days. For comparison, a single 1 TB consumer SSD can deliver comparable numbers (lower IOPS but higher throughput). If I plugged 24 consumer SSDs into a box, I would expect over 30M…

There's a quadrant of the market which is poorly served by the Cloud model of elastic compute: persistent local SSDs across shutdown and restart. Elastic compute means you want to be able to treat compute hardware as fungible. Persistent local storage makes that a lot harder because the Cloud provider wants to hand out that compute to someone else after shutdown, so the local storage needs to be wiped. So you either…

GCE fares a little better than this:

- VMs with SSDs can (in general -- there are exceptions for things like GPUs and exceptionally large instances) live migrate with contents preserved.

- GCE supports a timeboxed "restart in place" feature where the VM stays in limbo ("REPAIRING") for some amount of time waiting for the host to return to service: https://cloud.google.com/compute/docs/instances/host-mainten.... This mostly only applies to transient failures like power-loss beyond battery/generator sustaining thresholds, software crashes, etc.

- There is a related feature, also controlled by the `--discard-local-ssd=` flag, which allows preservation of local SSD data on a customer initiated VM stop.

Re: SSDs have become fast, except in the cloud

#387

Post by rcarmo »

silverquiet wrote: post #375
rcarmo wrote: post #372

Earlier quoted context omitted.

As someone who works with Azure daily, I am amazed not just at the initial consultant's conclusion (that is, alas, typical of folk who do not understand database engines), but also to your struggle with NVMe storage (I have some pretty large SQLite databases on my personal projects). You should not have needed an Ebsv5 (memory-optimised) instance. For that kind of thing, you should only have needed a D-series VM with…

> Anyway, many people fail to understand that Azure Storage works more like a SAN than a directly attached disk--when you attach a disk volume to the VM, you are actually attaching a _replica set_ of that storage that is at least three-way replicated and distributed across the datacenter to avoid data loss. You get RAID for free, if you will. I've said this a bit more sarcastically elsewhere in this thread, but basic…

Well, in short, people need to understand that cloud is not their computer. It is resource allocation with underlying assumptions around availability, redundancy and performance at a scale well beyond what they would experience in their own datacenter.

And they absolutely must understand this to avoid mis-designing things. Failure to do so is just bad engineering, and a LOT of time is spent educating customers on these differences.

A case in point that aligns with is that I used to work with Hadoop clusters, where you would use data replication for both redundancy and distributed processing. Moving Hadoop to Azure and maintaining conventional design rules (i.e., tripling the amount of disks) is the wrong way do do things, because it isn't required neither for redundancy nor for performance (they are both catered for by the storage resources).

(Of course there are better solutions than Hadoop these days - Spark being one that is very nice from a cloud resource perspective - but many people have nine times the storage they need allocated in their cloud Hadoop clusters because of lack of understanding...)

Re: SSDs have become fast, except in the cloud

#388

Post by tumult »

c0l0 wrote: post #384
tumult wrote: post #362

Earlier quoted context omitted.

Yep.

That is highly interesting and contrary to a number of reports I've read about the Samsung 970 EVO Plus Series (and experienced for myself) specifically! Can you share more details about your particular setup and methodology? (Specific model name/capacity, Firmware release, Kernel version, filesystem, mkfs and mount options, any relevant block layer funny business you are conciously setting would be of greatest inter…

Yeah, it happens on two of the 970 EVO Plus models. One on the older revision, and one on the newer. (I think there are only two?) It happens on both Linux and Windows. Uhh, I'm not sure about the kernel versions. I don't remember what I had booted at the time. On Windows I've seen it happen as far back as 1607 and as recently as 21H2. I've also seen it happen on someone else's computer (laptop.)

It's really easy to reproduce (at least for me?) and I'm pretty sure anyone can do it if they try to on purpose.

Re: SSDs have become fast, except in the cloud

#389

Post by jethro_tell »

icedchai wrote: post #385
jethro_tell wrote: post #380

Earlier quoted context omitted.

Heh, you'd probably be surprised, there's some really cool cutting edge stuff being done in those data centers but a lot of what is done is just plan old standard server management without much in the way of tricks. Its just someone else does it instead of you and the billing department is counting milliseconds.

Do cloud providers document these internals anywhere? I'd love to read about that sort of thing.

Not generally, especially not the super generic stuff. Where they really excel is having the guy that wrote the kernel driver or hypervisor on staff. But a lot of it is just an automated version of what you'd do on a smaller scale

Re: SSDs have become fast, except in the cloud

#390

Post by silverquiet »

rcarmo wrote: post #387
silverquiet wrote: post #375

Earlier quoted context omitted.

> Anyway, many people fail to understand that Azure Storage works more like a SAN than a directly attached disk--when you attach a disk volume to the VM, you are actually attaching a _replica set_ of that storage that is at least three-way replicated and distributed across the datacenter to avoid data loss. You get RAID for free, if you will. I've said this a bit more sarcastically elsewhere in this thread, but basic…

Well, in short, people need to understand that cloud is not their computer. It is resource allocation with underlying assumptions around availability, redundancy and performance at a scale well beyond what they would experience in their own datacenter. And they absolutely must understand this to avoid mis-designing things. Failure to do so is just bad engineering, and a LOT of time is spent educating customers on the…

I would think that lifting and shifting a Hadoop setup into the cloud would be considered an anti-pattern anyway; typically you would be told to find a managed, cloud-native solution.
Post reply on HN