Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

SSDs have become fast, except in the cloud

databasearchitects.blogspot.com

Post reply on HN
Print view View original

101–110 of 427 posts

Re: SSDs have become fast, except in the cloud

#101

Post by tjoff »

nullindividual wrote: post #10
ThrowawayTestr wrote: post #5

Earlier quoted context omitted.

Buy a server

I don't like this answer. When I look at cloud, I get to think "finally! No more hardware to manage. No OS to manage". It's the best thing about the cloud, provided your workload is amenable to PaaS. It's great because I don't have to manage Windows or IIS. Microsoft does that part for me and significantly cheaper than it would be to employ me to do that work.

And now you have to manage the cloud instead. Which turns out to be more hassle and with no overlap with the actual problem you are trying to solve.

So not only do you spend time on the wrong thing you don't even know how it works. And the providers goals are not aligned either as all they care about is locking you in.

How is that better?

Re: SSDs have become fast, except in the cloud

#102

Post by Aachen »

sprachspiel wrote: post #88
nik_0_0 wrote: post #75

Earlier quoted context omitted.

Is it? The line preceding the bullet list on that page seems to state otherwise: “” Each storage volume can deliver the following performance (all measured using 4 KiB blocks): * Up to 8000 MB/second of sequential reads “”

Just tested a i4i.32xlarge: $ lsblk NAME MAJ:MIN RM SIZE RO TYPE MOUNTPOINTS loop0 7:0 0 24.9M 1 loop /snap/amazon-ssm-agent/7628 loop1 7:1 0 55.7M 1 loop /snap/core18/2812 loop2 7:2 0 63.5M 1 loop /snap/core20/2015 loop3 7:3 0 111.9M 1 loop /snap/lxd/24322 loop4 7:4 0 40.9M 1 loop /snap/snapd/20290 nvme0n1 259:0 0 8G 0 disk ├─nvme0n1p1 259:1 0 7.9G 0 part / ├─nvme0n1p14 259:2 0 4M 0 part └─nvme0n1p15 259:3 0 106M 0…

So if I'm reading it right, the quote from the original article that started this thread was ballpark correct?

> we are still stuck with 2 GB/s per SSD

Versus the ~2.7 GiB/s your benchmark shows (bit hard to know where to look on mobile with all that line-wrapped output, and when not familiar with the fio tool; not your fault but that's why I'm double checking my conclusion)

Re: SSDs have become fast, except in the cloud

#103

Post by amluto »

candiddevmike wrote: post #55
pclmulqdq wrote: post #42

Earlier quoted context omitted.

It's because the "local" SSDs are not actually physically attached and there's a network protocol in the way.

Depends on the cloud provider. Local SSDs are physically attached to the host on GCP, but that makes them only useful for temporary storage.

Which is a weird sort of limitation. For any sort of you-own-the-hardware arrangement, NVMe disks are fine for long term storage. (Obviously one should have backups, but that’s a separate issue. One should have a DR plan for data on EBS, too.)

You need to migrate that data if you replace an entire server, but this usually isn’t a very big deal.

Re: SSDs have become fast, except in the cloud

#104

Post by aeyes »

20after4 wrote: post #80
jsnell wrote: post #71

Earlier quoted context omitted.

I think you're wrong about that. AWS calls this class of storage "instance storage" [0], and defines it as: > Many Amazon EC2 instances can also include storage from devices that are located inside the host computer, referred to as instance storage. There might be some wiggle room in "physically attached", but there's none in "storage devices located inside the host computer". It's not some kind of AWS-only thing eit…

If the SSD is installed in the host server, doesn't that still allow for it to be shared among many instances running on said host? I can imagine that a compute node has just a handful of SSDs and many hundreds of instances sharing the I/O bandwidth.

On AWS yes, the older instances which I am familiar with had 900GB drives and they sliced that up into volumes of 600, 450, 300, 150, 75GB depending on instance size.

But they also tell you how much IOPS you get: https://docs.aws.amazon.com/AWSEC2/latest/WindowsGuide/stora...

Re: SSDs have become fast, except in the cloud

#105

Post by vel0city »

SteveNuts wrote: post #35
vlovich123 wrote: post #14

Earlier quoted context omitted.

Why do they fundamentally need to be network attached storage instead of local to the VM?

Because even if you can squeeze 100TB or more of SSD/NVMe in a server, and there are 10 tenants using the machine, you're limited to 10TB as a hard ceiling. What happens when one tenant needs 200TB attached to a server? Cloud providers are starting to offer local SSD/NVMe, but you're renting the entire machine, and you're still limited to exactly what's installed in that server.

Given AWS and GCP offer multiple sizes for the same processor version with local SSDs, I don't think you have to rent the entire machine.

Search for i3en API names and you'll see:

i3en.large, 2x CPU, 1250GB SSD

i3en.xlarge, 4x CPU, 2500GB SSD

i3en.2xlarge, 8x CPU, 2x2500GB SSD

i3en.3xlarge, 12x CPU, 7500GB SSD

i3en.6xlarge, 24x CPU, 2x7500GB SSD

i3en.12xlarge, 48x CPU, 4x7500GB SSD

i3en.24xlarge, 96x CPU, 8x7500GB SSD

i3en.metal, 96x CPU, 8x7500GB SSD

So they've got servers with 96 CPUs and 8x7500GB SSDs. You can get a slice of one, or you can get the whole one. All of these are the ratio of 625GB of local SSD per CPU core.

https://instances.vantage.sh/

On GCP you can get a 2-core N2 instance type and attach multiple local SSDs. I doubt they have many physical 2-core Xeons in their datacenters.

Re: SSDs have become fast, except in the cloud

#106

Post by kstrauser »

fabioyy wrote: post #70

it is not worth to use cloud if you need a lot of iops/bandwidth heck, its not worth for anything besides scalability dedicated servers are wayyyy cheaper

I'm not certain that's true if you look at TCO. Yes, you can probably buy a server for less than the yearly rent on the equivalent EC2 instance. But then you've got to put that server somewhere, with reliable power and probably redundant Internet connections. You have to pay someone's salary to set it up and load it to the point that a user can SSH in and configure it. You have to maintain an inventory of spares, and pay someone to swap it out if it breaks. You have to pay to put its backups somewhere.

Yeah, you can skip a lot of that if your goal is to get a server online as cheaply as possible, reliability be damned. As soon as you start caring about keeping it in a business-ready state, costs start to skyrocket.

I've worn the sysadmin hat. If AWS burned down, I'd be ready and willing to recreate the important parts locally so that my company could stay in business. But wow, would they ever be in for some sticker shock.

Re: SSDs have become fast, except in the cloud

#107

Post by Aachen »

bombcar wrote: post #3

I think the obvious answer is there's not much demand, and keeping it "low" allows trickery and funny business with the virtualization layer (think: SAN, etc) that you can't do with "raw hardware speed".

I ended up buying a SATA SSD for 50 euros to stick in an old laptop that I was already using as server and, my god, it is so much faster than the thing I was trying to run on digitalocean. The DO VPS barely beat the old 5400 rpm spinning rust that was in the laptop originally (the reason why I was trying to rent a fast, advertised-with-SSD, server). Doing this i/o task effectively in the cloud, at least with DO, seems to require putting it in RAM which was a bit expensive for the few hundred gigabytes of data I wanted to process into an indexed format

So there is demand, but I'm certainly not interested in paying many multiples of 50 euros over an expected lifespan of a few years, so it may not make economic sense for them to offer it to users like me at least. On the other hand, for the couple hours this should have taken (rather than the days it initially did), I'd certainly have been willing to pay that cloud premium and that's why I tried to get me one of these allegedly SSD-backed VPSes... but now that I have a fast system permanently, I don't think that was a wise decision of past me

Re: SSDs have become fast, except in the cloud

#108

Post by malfist »

avg_dev wrote: post #59
malfist wrote: post #45

Earlier quoted context omitted.

I keep hearing that, but that's simply not true. SSDs are fast, but they're several orders of magnitude slower than RAM, which is orders of magnitude slower than CPU Cache. Samsung 990 Pro 2TB has a latency of 40 μs DDR4-2133 with a CAS 15 has a latency of 14 nano seconds. DDR4 latency is 0.035% of one of the fastest SSDs, or to put it another way, DDR4 is 2,857x faster than an SSD. L1 cache is typically accessible i…

pretty cool comparisons. quite some differences there. tangent, I remember reading some post called something like "Latency numbers every programmer should know" and being slightly ashamed when I could not internalize it.

Oh don't feel bad. I had to look up every one of those numbers

Re: SSDs have become fast, except in the cloud

#109

Post by supriyo-biswas »

ssl-3 wrote: post #96
supriyo-biswas wrote: post #47

Earlier quoted context omitted.

A network attached, replicated storage hedges against data loss but increases latency; however most customers usually prefer higher latency to data loss. As an example, see the highly upvoted fly.io thread[1] with customers complaining about the same thing. [1] https://news.ycombinator.com/item?id=36808296

Locally-attached, replicated storage also hedges against data loss.

RAID rebuild times make it an unviable option and customers typically expect problematic VMs to be live-migrated to other hosts with the disks still having their intended data.

The self hosted version of this is GlusterFS and Ceph, which have the same dynamics as EBS and its equivalents in other cloud providers.

Re: SSDs have become fast, except in the cloud

#110

Post by ownagefool »

20after4 wrote: post #80
jsnell wrote: post #71

Earlier quoted context omitted.

I think you're wrong about that. AWS calls this class of storage "instance storage" [0], and defines it as: > Many Amazon EC2 instances can also include storage from devices that are located inside the host computer, referred to as instance storage. There might be some wiggle room in "physically attached", but there's none in "storage devices located inside the host computer". It's not some kind of AWS-only thing eit…

If the SSD is installed in the host server, doesn't that still allow for it to be shared among many instances running on said host? I can imagine that a compute node has just a handful of SSDs and many hundreds of instances sharing the I/O bandwidth.

PCI bus, etc too
Post reply on HN