Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

SSDs have become fast, except in the cloud

databasearchitects.blogspot.com

Post reply on HN
Print view View original

151–160 of 427 posts

Re: SSDs have become fast, except in the cloud

#151

Post by deathanatos »

nullindividual wrote: post #10
ThrowawayTestr wrote: post #5

Earlier quoted context omitted.

Buy a server

I don't like this answer. When I look at cloud, I get to think "finally! No more hardware to manage. No OS to manage". It's the best thing about the cloud, provided your workload is amenable to PaaS. It's great because I don't have to manage Windows or IIS. Microsoft does that part for me and significantly cheaper than it would be to employ me to do that work.

> No more hardware to manage. No OS to manage

We must be using different clouds.

For some of the much higher-level services … maybe some semblance of that statement holds. But for VMs? Definitely not "no OS to manage" … the OS is usually on the customer. There might be OS-level agents from your cloud of choice that make certain operations easier … but I'm still on the hook for updates.

Even "No machine" is a stretch, though I've found this is much more dependent on cloud. AWS typically notices failures before I do, and by the time I notice something is up, the VM has been migrated to a new host and I'm none the wiser sans the reboot that cost. But other clouds I've been less lucky with: we've caught host failures well before the cloud provider, to an extent where I've wished there was a "vote of no confidence" API call I could make to say "give me new HW, and I personally think this HW is suss".

Even on higher level services like RDS, or S3, I've noticed failures prior to AWS … or even to the extent that I don't know that AWS would have noticed those failures unless we had opened the support ticket. (E.g., in the S3 case, even though we clearly reported the problem, and the problem was occurring on basically every request, we still had to provide example request IDs before they'd believe us. The service was basically in an outage as far as we could tell … though I think AWS ended up claiming it was "just us".)

That said, S3 in particular is still an excellent service, and I'd happily use it again. But cloud == 0 time on my part. It depends heavily on the cloud, and less heavily on the service how much time, and sometimes, it is still worthwhile.

Re: SSDs have become fast, except in the cloud

#152

Post by catherinecodes »

infamia wrote: post #135
catherinecodes wrote: post #36

Earlier quoted context omitted.

Hetzner and Entrywan are pure-play cloud companies with good prices and support. Hetzner is based in Germany and Entrywan in the US.

Thanks for mentioning Entrywan, they look great from what I can tell on their site. Have you used their services? If so, I'm curious about your experiences with them.

My irc bouncer and two kubernetes clusters are running there. So far the service has been good.

Re: SSDs have become fast, except in the cloud

#153

Post by wmf »

doubled112 wrote: post #93
bombcar wrote: post #87

Earlier quoted context omitted.

Which can cause considerable "amusement" depending on the provider - one I won't name directly but is much more centered on actual renting racks than their (now) cloud offering - if you had a virtual machine older than a year or so, deleting and restoring it would get you on a newer "host" and you'd be faster for the same cost. Otherwise it'd stay on the same physical piece of hardware it was allocated to when new.

Amusing is a good description. "Hardware degradation detected, please turn it off and back on again" I could do a migration with zero downtime in VMware for a decade but they can't seamlessly move my VM to a machine that works in 2024? Great, thanks. Amusing.

Cloud providers have live migration now but I guess they don't want to guarantee anything.

Re: SSDs have become fast, except in the cloud

#154

Post by Szpadel »

LeifCarrotson wrote: post #73
malfist wrote: post #45

Earlier quoted context omitted.

I keep hearing that, but that's simply not true. SSDs are fast, but they're several orders of magnitude slower than RAM, which is orders of magnitude slower than CPU Cache. Samsung 990 Pro 2TB has a latency of 40 μs DDR4-2133 with a CAS 15 has a latency of 14 nano seconds. DDR4 latency is 0.035% of one of the fastest SSDs, or to put it another way, DDR4 is 2,857x faster than an SSD. L1 cache is typically accessible i…

I wonder how many people have built failed businesses that never had enough customer data to exceed the DDR4 in the average developer laptop, and never had so many simultaneous queries it couldn't be handled by a single core running SQLite, but built the software architecture on a distributed cloud system just in case it eventually scaled to hundreds of terabytes and billions of simultaneous queries.

In may day job I often see systems that have the opposite. Especially for database queries, developers tested on local machine with 100s of records and everything was quick and snappy and on production with mere millions of records I often see queries taking minutes up to a hour just because some developer didn't see need for creating indexes or created query in a way there is no way to even create any index that would work

Re: SSDs have become fast, except in the cloud

#155

Post by kwillets »

AWS docs and blogs describe the Nitro SSD architecture, which is locally attached with custom firmware.

> The Nitro Cards are physically connected to the system main board and its processors via PCIe, but are otherwise logically isolated from the system main board that runs customer workloads.

https://docs.aws.amazon.com/whitepapers/latest/security-desi...

> In order to make the [SSD] devices last as long as possible, the firmware is responsible for a process known as wear leveling.... There’s some housekeeping (a form of garbage collection) involved in this process, and garden-variety SSDs can slow down (creating latency spikes) at unpredictable times when dealing with a barrage of writes. We also took advantage of our database expertise and built a very sophisticated, power-fail-safe journal-based database into the SSD firmware.

https://aws.amazon.com/blogs/aws/aws-nitro-ssd-high-performa...

This firmware layer seems like a good candidate for the slowdown.

Re: SSDs have become fast, except in the cloud

#156

Post by wmf »

0cf8612b2e1e wrote: post #111

Serious question, for a consumer does it make any sense to compare SSD benchmarks? I assume the best and worst models give a user an identical experience in 99% of cases, and it is only prosumer activities (video? sustained writes?) which would differentiate them.

Yeah, that's pretty much the case. Cheap SSDs provide good enough performance for desktop use.

Re: SSDs have become fast, except in the cloud

#157

Post by throwawaaarrgh »

pclmulqdq wrote: post #11

This was a huge technical problem I worked on at Google, and is sort of fundamental to a cloud. I believe this is actually a big deal that drives peoples' technology directions. SSDs in the cloud are attached over a network, and fundamentally have to be. The problem is that this network is so large and slow that it can't give you anywhere near the performance of a local SSD. This wasn't a problem for hard drives, whi…

> SSDs in the cloud are attached over a network, and fundamentally have to be

SANs can still be quite fast, and instance storage is fast, both of which are available in cloud providers

Re: SSDs have become fast, except in the cloud

#158

Post by throwawaaarrgh »

20after4 wrote: post #80
jsnell wrote: post #71

Earlier quoted context omitted.

I think you're wrong about that. AWS calls this class of storage "instance storage" [0], and defines it as: > Many Amazon EC2 instances can also include storage from devices that are located inside the host computer, referred to as instance storage. There might be some wiggle room in "physically attached", but there's none in "storage devices located inside the host computer". It's not some kind of AWS-only thing eit…

If the SSD is installed in the host server, doesn't that still allow for it to be shared among many instances running on said host? I can imagine that a compute node has just a handful of SSDs and many hundreds of instances sharing the I/O bandwidth.

Instance storage is not networked. That's why it's there.

Re: SSDs have become fast, except in the cloud

#159

Post by jng »

pclmulqdq wrote: post #143
jsnell wrote: post #71

Earlier quoted context omitted.

I think you're wrong about that. AWS calls this class of storage "instance storage" [0], and defines it as: > Many Amazon EC2 instances can also include storage from devices that are located inside the host computer, referred to as instance storage. There might be some wiggle room in "physically attached", but there's none in "storage devices located inside the host computer". It's not some kind of AWS-only thing eit…

That's the abstraction they want you to work with, yes. That doesn't mean it's what is actually happening - at least not in the same way that you're thinking. As a hint for you, I said " a network", not " the network." You can also look at public presentations about how Nitro works.

Nitro "virtual NVME" device are mostly (only?) for EBS -- remote network storage, transparently managed, using a separate network backbone, and presented to the host as a regular local NVME device. SSD drives in instances such as i4i, etc. are physically attached in a different way -- but physically, unlike EBS, they are ephemeral and the content becomes unavaiable as you stop the instance, and when you restart, you get a new "blank slate". Their performance is 1 order of magnitude faster than standard-level EBS, and the cost structure is completely different (and many orders of magnitude more affordable than EBS volumes configured to have comparable I/O performance).

Re: SSDs have become fast, except in the cloud

#160

Post by throwawaaarrgh »

candiddevmike wrote: post #55
pclmulqdq wrote: post #42

Earlier quoted context omitted.

It's because the "local" SSDs are not actually physically attached and there's a network protocol in the way.

Depends on the cloud provider. Local SSDs are physically attached to the host on GCP, but that makes them only useful for temporary storage.

Yes, that's what their purpose is in cloud applications: temporary high performance storage only.

If you want long term local storage you'll have to reserve an instance host.

Post reply on HN