Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

SSDs have become fast, except in the cloud

databasearchitects.blogspot.com

Post reply on HN
Print view View original

191–200 of 427 posts

Re: SSDs have become fast, except in the cloud

#191

Post by layer8 »

Szpadel wrote: post #154
LeifCarrotson wrote: post #73

Earlier quoted context omitted.

I wonder how many people have built failed businesses that never had enough customer data to exceed the DDR4 in the average developer laptop, and never had so many simultaneous queries it couldn't be handled by a single core running SQLite, but built the software architecture on a distributed cloud system just in case it eventually scaled to hundreds of terabytes and billions of simultaneous queries.

In may day job I often see systems that have the opposite. Especially for database queries, developers tested on local machine with 100s of records and everything was quick and snappy and on production with mere millions of records I often see queries taking minutes up to a hour just because some developer didn't see need for creating indexes or created query in a way there is no way to even create any index that wou…

That’s true, but has little to do with distributed cloud architecture vs. single local instance.

Re: SSDs have become fast, except in the cloud

#192

Post by treflop »

bee_rider wrote: post #120
pixl97 wrote: post #62

Earlier quoted context omitted.

Bandwidth delay product does not help serialized transactions. If you're reaching out to disk for results, or if you have locking transactions on a table the achievable operations drops dramatically as latency between the host and the disk increases.

The typical way to trade bandwidth away for latency would, I guess, be speculative requests. In the CPU world at least. I wonder if any cloud providers have some sort of framework built around speculative disk reads (or maybe it is a totally crazy trade to make in this context)?

Often times it’s the app (or something high level) that would need speculative requests, which it may not be possible in the given domain.

I don’t think it’s possible in most domains.

Re: SSDs have become fast, except in the cloud

#193

Post by treflop »

silverquiet wrote: post #78
_Rabs_ wrote: post #15

Earlier quoted context omitted.

So much of this. The amount of times I've seen someone complain about slow DB performance when they're trying to connect to it from a different VPC, and bottlenecking themselves to 100Mbits is stupidly high. Literally depending on where things are in a data center... If you're looking for closely coupled and on a 10G line on the same switch, going to the same server rack. I bet you performance will be so much more co…

> Literally depending on where things are in a data center I thought cloud was supposed to abstract this away? That's a bit of a sarcastic question from a long-time cloud skeptic, but... wasn't it?

Cloud makes provisioning more servers quicker because you are paying someone to basically have a bunch of servers ready to go right away with an API call instead of a phone call, maintained by a team that isn’t yours, with economies of scale working for the provider.

Cloud does not do anything else.

None of these latency/speed problems are cloud-specific. If you have on-premise servers and you are storing your data on network-attached storage, you have the exact same problems (and also the same advantages).

Unfortunately the gap between local and network storage is wide. You win some, you lose some.

Re: SSDs have become fast, except in the cloud

#194

Post by kccqzy »

scottlamb wrote: post #145
kccqzy wrote: post #138

Earlier quoted context omitted.

That document is probably deliberately on the pessimistic side to encourage your code to be portable across all kinds of "data centers" (however that is defined). When I previously worked at Google, the standard RPC system definitely offered 50 microseconds of round trip latency at the median (I measured it myself in a real application), and their advanced user-space implementation called Snap could offer about 10 mi…

Interesting. I worked at Google until January 2021. I see 2019 dates on that PDF, but I wasn't aware of snap when I left. There was some alternate RPC approach (Pony Express, maybe? I get the names mixed up) that claimed 10 µs or so but was advertised as experimental (iirc had some bad failure modes at the time in practice) and was simply unavailable in many of the datacenters I needed to deploy in. Maybe they're two…

Snap was the external name for the internal project known as User Space Packet Service (abbreviated USPS) so naturally they renamed it prior to publication. I deployed an app using Pony Express in 2023 and it was available in the majority of cells worldwide. Pony Express supported more than just RPC though. The alternate RPC approach that you spoke of was called Void. It had been experimental for a long time and indeed it wasn't well known even inside Google.

> but I remember it wasn't that much better than 0.5 ms.

If you and I still worked at Google I'd just give you an automon dashboard link showing latency an order of magnitude better than that to prove myself…

Re: SSDs have become fast, except in the cloud

#195

Post by yolovoe »

pclmulqdq wrote: post #42
jsnell wrote: post #28

Earlier quoted context omitted.

According to the submitted article, the numbers are from AWS instance types where the SSD is "physically attached" to the host, not about SSD-backed NAS solutions. Also, the article isn't just about SSDs being no faster than a network. It's about SSDs being two orders of magnitude slower than datacenter networks.

It's because the "local" SSDs are not actually physically attached and there's a network protocol in the way.

You’re wrong. Instance local means SSD is physically attached to the droplet and is inside the server chassis, connected via PCIe.

Sourece: I work on nitro cards.

Re: SSDs have become fast, except in the cloud

#196

Post by yolovoe »

mike_hearn wrote: post #58
pclmulqdq wrote: post #42

Earlier quoted context omitted.

It's because the "local" SSDs are not actually physically attached and there's a network protocol in the way.

They do this because they want SSDs to be in a physically separate part of the building for operational reasons, or what's the point in giving you a "local" SSD that isn't actually plugged into the real machine?

The comment you’re responding to is wrong. AWS offers many kinds of storage. Instance local storage is physically attached to the droplet. EBS isn’t but that’s a separate thing entirely.

I literally work in EC2 Nitro.

Re: SSDs have become fast, except in the cloud

#197

Post by kwillets »

dan-robertson wrote: post #189
kwillets wrote: post #155

AWS docs and blogs describe the Nitro SSD architecture, which is locally attached with custom firmware. > The Nitro Cards are physically connected to the system main board and its processors via PCIe, but are otherwise logically isolated from the system main board that runs customer workloads. https://docs.aws.amazon.com/whitepapers/latest/security-desi... > In order to make the [SSD] devices last as long as possible…

Yeah, I’m curious how they would respond to the claims in the article. In [1], they talk about aiming for low latency, for consistent performance (apparently other SSDs could stall at inopportune times), and support on-disk encryption. Latency is often in direct conflict with throughput (eg batching usually trades one for the other), and also matters a lot for plenty of filesystem or database tasks (indeed the OP lin…

The Nitro chipset claims 100 GB/s encryption, so that doesn't seem to be the reason.

Re: SSDs have become fast, except in the cloud

#198

Post by layer8 »

kstrauser wrote: post #106
fabioyy wrote: post #70

it is not worth to use cloud if you need a lot of iops/bandwidth heck, its not worth for anything besides scalability dedicated servers are wayyyy cheaper

I'm not certain that's true if you look at TCO. Yes, you can probably buy a server for less than the yearly rent on the equivalent EC2 instance. But then you've got to put that server somewhere, with reliable power and probably redundant Internet connections. You have to pay someone's salary to set it up and load it to the point that a user can SSH in and configure it. You have to maintain an inventory of spares, and…

Where I live, a number of SMEs are doing this. It’s really not that costly, unless you are a tiny startup I guess.

Re: SSDs have become fast, except in the cloud

#199

Post by jsnell »

pclmulqdq wrote: post #143
jsnell wrote: post #71

Earlier quoted context omitted.

I think you're wrong about that. AWS calls this class of storage "instance storage" [0], and defines it as: > Many Amazon EC2 instances can also include storage from devices that are located inside the host computer, referred to as instance storage. There might be some wiggle room in "physically attached", but there's none in "storage devices located inside the host computer". It's not some kind of AWS-only thing eit…

That's the abstraction they want you to work with, yes. That doesn't mean it's what is actually happening - at least not in the same way that you're thinking. As a hint for you, I said " a network", not " the network." You can also look at public presentations about how Nitro works.

I've linked to public documentation that is pretty clearly in conflict with what you said. There's no wiggle room in how AWS describes their service without it being false advertising. There's no "ah, but what if we define the entire building to be the host computer, then the networked SSDs really are inside the host computer" sleight of hand to pull off here.

You've provided cryptic hints and a suggestion to watch some unnamed presentation.

At this point I really think the burden of proof is on you.

Re: SSDs have become fast, except in the cloud

#200

Post by goguy »

malfist wrote: post #115
LeifCarrotson wrote: post #73

Earlier quoted context omitted.

I wonder how many people have built failed businesses that never had enough customer data to exceed the DDR4 in the average developer laptop, and never had so many simultaneous queries it couldn't be handled by a single core running SQLite, but built the software architecture on a distributed cloud system just in case it eventually scaled to hundreds of terabytes and billions of simultaneous queries.

I totally hear you about that. I work for FAANG, and I'm working on a service that has to be capable of sending 1.6m text messages in less than 10 minutes. The amount of complexity the architecture has because of those constraints is insane. When I worked at my previous job, management kept asking for that scale of designs for less than 1/1000 of the throughput and I was constantly pushing back. There's real costs to…

That really doesn't require that much complexity.

I used to send something like 250k a minute complete with delivery report processing from a single machine running a bunch of other services like 10 years ago.

Post reply on HN