Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

SSDs have become fast, except in the cloud

databasearchitects.blogspot.com

Post reply on HN
Print view View original

371–380 of 427 posts

Re: SSDs have become fast, except in the cloud

#371

Post by bcaxis »

wistlo wrote: post #364

At my job at a telco, I had a 13 billion record file to scan and index for duplicates and bad addresses. Consultants brought in to move our apps (some of which were Excel macros, others SAS scripts running on old desktop) to Azure. The Azure architects identified Postgres as the best tool. Consultants attempted to create a Postgres index in a small Azure instance but their tests would fail without completion (they we…

This matches my experience. Something about cloud systems makes them incredibly slow compared to real hardware. Not just the disk, the CPU is more limited too.

I fire up vCPU or dedicated or bare metal in the cloud, doesn't matter, I simply cannot match the equivalent compute of real hardware and it's not even close.

Re: SSDs have become fast, except in the cloud

#372

Post by rcarmo »

wistlo wrote: post #364

At my job at a telco, I had a 13 billion record file to scan and index for duplicates and bad addresses. Consultants brought in to move our apps (some of which were Excel macros, others SAS scripts running on old desktop) to Azure. The Azure architects identified Postgres as the best tool. Consultants attempted to create a Postgres index in a small Azure instance but their tests would fail without completion (they we…

As someone who works with Azure daily, I am amazed not just at the initial consultant's conclusion (that is, alas, typical of folk who do not understand database engines), but also to your struggle with NVMe storage (I have some pretty large SQLite databases on my personal projects).

You should not have needed an Ebsv5 (memory-optimised) instance. For that kind of thing, you should only have needed a D-series VM with a premium storage data disk (or, if you wanted a hypervisor-adjacent, very low latency volume, a temp volume in another SKU).

Anyway, many people fail to understand that Azure Storage works more like a SAN than a directly attached disk--when you attach a disk volume to the VM, you are actually attaching a _replica set_ of that storage that is at least three-way replicated and distributed across the datacenter to avoid data loss. You get RAID for free, if you will.

That is inherently slower than a hypervisor-adjacent (i.e., on-board) volume.

Re: SSDs have become fast, except in the cloud

#373

Post by rcarmo »

jng wrote: post #159
pclmulqdq wrote: post #143

Earlier quoted context omitted.

That's the abstraction they want you to work with, yes. That doesn't mean it's what is actually happening - at least not in the same way that you're thinking. As a hint for you, I said " a network", not " the network." You can also look at public presentations about how Nitro works.

Nitro "virtual NVME" device are mostly (only?) for EBS -- remote network storage, transparently managed, using a separate network backbone, and presented to the host as a regular local NVME device. SSD drives in instances such as i4i, etc. are physically attached in a different way -- but physically, unlike EBS, they are ephemeral and the content becomes unavaiable as you stop the instance, and when you restart, you…

This is the way Azure temporary volumes work as well. They are scrubbed off the hardware once the VM that accesses them is dead. Everything else is over the network.

Re: SSDs have become fast, except in the cloud

#374

Post by silent_cal »

bcaxis wrote: post #371
wistlo wrote: post #364

At my job at a telco, I had a 13 billion record file to scan and index for duplicates and bad addresses. Consultants brought in to move our apps (some of which were Excel macros, others SAS scripts running on old desktop) to Azure. The Azure architects identified Postgres as the best tool. Consultants attempted to create a Postgres index in a small Azure instance but their tests would fail without completion (they we…

This matches my experience. Something about cloud systems makes them incredibly slow compared to real hardware. Not just the disk, the CPU is more limited too. I fire up vCPU or dedicated or bare metal in the cloud, doesn't matter, I simply cannot match the equivalent compute of real hardware and it's not even close.

Isn't that expected? I would assume cloud stuff is slower because it's essentially an emulation of the real thing.

Re: SSDs have become fast, except in the cloud

#375

Post by silverquiet »

rcarmo wrote: post #372
wistlo wrote: post #364

At my job at a telco, I had a 13 billion record file to scan and index for duplicates and bad addresses. Consultants brought in to move our apps (some of which were Excel macros, others SAS scripts running on old desktop) to Azure. The Azure architects identified Postgres as the best tool. Consultants attempted to create a Postgres index in a small Azure instance but their tests would fail without completion (they we…

As someone who works with Azure daily, I am amazed not just at the initial consultant's conclusion (that is, alas, typical of folk who do not understand database engines), but also to your struggle with NVMe storage (I have some pretty large SQLite databases on my personal projects). You should not have needed an Ebsv5 (memory-optimised) instance. For that kind of thing, you should only have needed a D-series VM with…

> Anyway, many people fail to understand that Azure Storage works more like a SAN than a directly attached disk--when you attach a disk volume to the VM, you are actually attaching a _replica set_ of that storage that is at least three-way replicated and distributed across the datacenter to avoid data loss. You get RAID for free, if you will.

I've said this a bit more sarcastically elsewhere in this thread, but basically, why would you expect people to understand this? Cloud is sold as abstracting away hardware details and giving performance SLAs billed by the hour (or minute, second, whatever). If you need to know significant details of their implementation, then you're getting to the point where you might as well buy your own hardware and save a bunch of money (which seems to be gaining some steam in a minor but noticeable cloud repatriation movement).

Re: SSDs have become fast, except in the cloud

#376

Post by cduzz »

Retric wrote: post #150
cduzz wrote: post #116

Earlier quoted context omitted.

I've seen SANs get nuked by operator error or by environmental issues (overheated DC == SAN shuts itself down). Distributed clusters of things can work just fine on ephemeral local storage (aka local storage ). A kafka cluster or an opensearch cluster will be fine using instance local storage, for instance. As with everything else.... "it depends"

Sure distributed clusters get back to network/workload limitations.

These days it's likely that your SAN is actually just a cluster of commodity hardware where the disks/SSDs have custom firmware and some fancy block shoveling software.

Re: SSDs have become fast, except in the cloud

#377

Post by wistlo »

rcarmo wrote: post #372
wistlo wrote: post #364

At my job at a telco, I had a 13 billion record file to scan and index for duplicates and bad addresses. Consultants brought in to move our apps (some of which were Excel macros, others SAS scripts running on old desktop) to Azure. The Azure architects identified Postgres as the best tool. Consultants attempted to create a Postgres index in a small Azure instance but their tests would fail without completion (they we…

As someone who works with Azure daily, I am amazed not just at the initial consultant's conclusion (that is, alas, typical of folk who do not understand database engines), but also to your struggle with NVMe storage (I have some pretty large SQLite databases on my personal projects). You should not have needed an Ebsv5 (memory-optimised) instance. For that kind of thing, you should only have needed a D-series VM with…

I may have the "Ebsv5" series code incorrect. I'd look it up, but I don't have access to the subscription any longer.

What I chose ultimately was definitely "nVME attached" and definitely pricey. The "hypervisor-adjacent, very low latency volume" was not an obvious choice.

The best performing configuration did come from me--the db admin learning Azure on the fly--and not the four Azure architects nor the half dozen consultants with Azure credentials brought onto the project.

Re: SSDs have become fast, except in the cloud

#378

Post by namibj »

bcaxis wrote: post #281
jauntywundrkind wrote: post #254

Earlier quoted context omitted.

NVMe has been ridiculously great. I'm excited to see what happens to prices as E1 form factor ramps up! Much physically bigger drives allows for consolidation of parts, a higher ratio of flash chips to everything else, which seems promising. It's more a value line, but Intel's P5315 is 15TB at a quite low $0.9/GB. It might not help much with oops though. Amazing that we have PCIe 5.0 16GB/s and already are so near th…

The main problem with consumer drives is the missing power loss protection (plp). M.2 drives just don't have space for the caps like an enterprise 2.5 u.2/u.3 drive will have. This matters when the DB calls a sync and it's expecting the data to be written safely to disk before it returns. A consumer drive basically stops everything until it can report success and your IOPS falls to like 1/100th of what the drive is c…

Sadly the solution, a firmware variant with ZNS instead of the normal random write block device, just isn't on offer (please tell if I'm wrong; I'd love one!). Because with ZNS you can get away with tiny caps, large enough to complete the in-flight blocks (not the buffered ones, just those that are already at the flash chip itself), plus one metadata journal/ring buffer page to store write pointers and zone status for all zones touched since the last metadata write happened. Given that this should take about 100 μs, I don't see unannounced power loss really that problematic to deal with.

In theory the ATX PSU reports imminent power loss with a mandatory notice of no less than 1ms; this would easily be enough to finish in-flight writes and record the zone state.

Re: SSDs have become fast, except in the cloud

#379

Post by driftnet »

arrakeenrevived wrote: post #299
chatmasta wrote: post #295

Earlier quoted context omitted.

Surely "droplet" is a derivative of "ocean?"

Clouds (like, the big fluffy things in the sky) are made up of many droplets of liquid. Using "droplet" to refer to the things that make up cloud computing is a pretty natural nickname for any cloud provider, not just DO. I do imagine that DO uses "droplet" as a public product branding because it works well with their "Ocean" brand, though. ...now I'm actually interested in knowing if "droplet" is derived from "ocean…

Clouds are water vapor, not droplets.

Re: SSDs have become fast, except in the cloud

#380

Post by jethro_tell »

icedchai wrote: post #225
queuebert wrote: post #165

Earlier quoted context omitted.

How do these machines manage the sharing of one local SSD across multiple VMs? Is there some wrapper around the I/O stack? Does it appear as a network share? Geniuinely curious...

With Linux and KVM/QEMU, you can map an entire physical disk, disk partition, or file to a block device in the VM. For my own VM hosts, I use LVM and map a logical volume to the VM. I assumed cloud providers did something conceptually similar, only much more sophisticated.

Heh, you'd probably be surprised, there's some really cool cutting edge stuff being done in those data centers but a lot of what is done is just plan old standard server management without much in the way of tricks. Its just someone else does it instead of you and the billing department is counting milliseconds.
Post reply on HN