Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

SSDs have become fast, except in the cloud

databasearchitects.blogspot.com

Post reply on HN
Print view View original

251–260 of 427 posts

Re: SSDs have become fast, except in the cloud

#251

Post by computerdork »

jiggawatts wrote: post #219
computerdork wrote: post #202

Earlier quoted context omitted.

Interesting. And would you happen to have the numbers on the performance of the local SSD? Is it's read and write throughput up to the level of modern SSD's?

It's pretty much like how the article said. The cloud local SSDs are notably slower than what you'd get in an ordinary laptop, let alone a high-end server. I'm not an insider and don't have any exclusive knowledge, but from reading a lot about the topic my impression is that the issue in both clouds is the virtualization overheads. That is, having the networking or storage go through any hypervisor software layer is…

Makes a lot of sense! Yeah, seems like for the OP's performance-issue, you pretty much have the reason why it's happening (VM overhead) and solutions for it (bypassing the software layer using custom hardware like Azure Boost).

Thanks for the info!

Re: SSDs have become fast, except in the cloud

#252

Post by Nextgrid »

Ericson2314 wrote: post #240
Nextgrid wrote: post #51

Earlier quoted context omitted.

The problem isn't necessarily speed, it's random access latency. What makes SSDs fast and "magical" is their low random-access latency compared to a spinning disk. The sequential-access read speed is merely a bonus. Networked storage negates that significantly, absolutely killing performance for certain applications. You could have a 100Gbps network and it still won't match a direct-attached SSD in terms of latency (…

Um... why the hell does the network care whether I am doing random or sequential access? Your left that part out of your argument.

Ah sorry, my bad. You are correct that you can fire off many random access operations in parallel and get good throughput that way.

The problem is that this is not possible when the next IO request depends on the result of a previous one, like in a database where you must first read the index to know the location of the row data itself.

Re: SSDs have become fast, except in the cloud

#253

Post by mike_hearn »

davkan wrote: post #190
nostrademons wrote: post #77

Earlier quoted context omitted.

Makes me wonder if we're on the crux of a shift back to client-based software. Historically changes in the relative cost of computing components have driven most of the shifts in the computing industry. Cheap teletypes & peripherals fueled the shift from batch-processing mainframes to timesharing minicomputers. Cheap CPUs & RAM fueled the shift from minicomputers to microcomputers. Cheap and fast networking fueled th…

I think a major limiting factor here for many applications is that mobile users are a huge portion of the user base. In that space storage, and more importantly battery life, are still at a premium. Granted the storage cost just seems to be gouging from my layman’s point of view, so industry needs might force a shift upwards.

Mobile devices are the desktop computers of the 2010s though. They are mostly used with very thick clients.

Re: SSDs have become fast, except in the cloud

#254

Post by jauntywundrkind »

c0l0 wrote: post #48

Seeing the really just puny "provisioned IOPS" numbers on hugely expensive cloud instances made me chuckle (first in disbelief, then in horror) when I joined a "cloud-first" enterprise shop in 2020 (having come from a company that hosted their own hardware at a colo). It's no wonder that many people nowadays, esp. those who are so young that they've never experienced anything but cloud instances, seem to have little…

NVMe has been ridiculously great. I'm excited to see what happens to prices as E1 form factor ramps up! Much physically bigger drives allows for consolidation of parts, a higher ratio of flash chips to everything else, which seems promising. It's more a value line, but Intel's P5315 is 15TB at a quite low $0.9/GB.

It might not help much with oops though. Amazing that we have PCIe 5.0 16GB/s and already are so near theoretical max (some lost to overhead), even on consumer cards.

Going enterprise for the drive-writes-per-day (DWPD) is 100% worth it for most folks, but I am morbidly curious how different the performance profile would be running enterprise vs non these days. But reciprocally the high DWPD drives (Kioxia CD8P-V for example is DWPD of 3) seems to often come with somewhat more mild sustained 4k write oops, making me think maybe there's a speed vs reliability tradeoff that could be taken advantage of from consumer drives in some cases; not sure who wants tons of iops but doesn't actually intend to hit their Total Drive Writes, but it save you some iops/$ if so. That said, I'm shocked to see the enterprise premium is a lot less absurd than it used to be! (If you can find stock.)

Re: SSDs have become fast, except in the cloud

#255

Post by lolc »

solardev wrote: post #223
hathawsh wrote: post #201

Earlier quoted context omitted.

That seems like a big opportunity for other cloud providers. They could provide SSDs that are actually physically attached and boast (rightfully) that their SSDs are a lot faster, drawing away business from older cloud providers.

For what kind of workloads would a slower SSD be a significant bottleneck?

I tend some workloads that transform data grids of varying sizes. The grids are anon mmaps so that when mem runs out, they get paged out. This means processing stays mostly in-mem yet won't abort when mem runs tight. The processes that get hit by paging slow to a crawl though. Getting faster SSD means they're still crawling but crawling faster. Doubling SSD throughput would pretty much half the tail latency.

Re: SSDs have become fast, except in the cloud

#256

Post by rdoherty »

kuschku wrote: post #186
malfist wrote: post #115

Earlier quoted context omitted.

I totally hear you about that. I work for FAANG, and I'm working on a service that has to be capable of sending 1.6m text messages in less than 10 minutes. The amount of complexity the architecture has because of those constraints is insane. When I worked at my previous job, management kept asking for that scale of designs for less than 1/1000 of the throughput and I was constantly pushing back. There's real costs to…

Maybe I'm misunderstanding something, but that's about 2700 a second. Or about 3Mbps. Even a very unoptimized application running on a dev laptop can serve 1Gbps nowadays without issues. So what are the constraints that demand a complex architecture?

I'm not the OP but a few things:

* Reading/fetching the data - usernames, phone number, message, etc.

* Generating the content for each message - it might be custom per person

* This is using a 3rd party API that might take anywhere from 100ms to 2s to respond, and you need to leave a connection open.

* Retries on errors, rescheduling, backoffs

* At least once or at most once sends? Each has tradeoffs

* Stopping/starting that many messages at any time

* Rate limits on some services you might be using alongside your service (network gateway, database, etc)

* Recordkeeping - did the message send? When?

Re: SSDs have become fast, except in the cloud

#257

Post by Nextgrid »

bee_rider wrote: post #120
pixl97 wrote: post #62

Earlier quoted context omitted.

Bandwidth delay product does not help serialized transactions. If you're reaching out to disk for results, or if you have locking transactions on a table the achievable operations drops dramatically as latency between the host and the disk increases.

The typical way to trade bandwidth away for latency would, I guess, be speculative requests. In the CPU world at least. I wonder if any cloud providers have some sort of framework built around speculative disk reads (or maybe it is a totally crazy trade to make in this context)?

You'd need the whole stack to understand your data format in order to make speculative requests useful. It wouldn't surprise me if cloud providers indeed do speculative reads but there isn't much they can do to understand your data format, so chances are they're just reading a few extra blocks beyond where your OS read and are hoping that the next OS-initiated read will fall there so it can be serviced using this prefetched data. Because of full-disk-encryption, the storage stack may not be privy to the actual data so it couldn't make smarter, data-aware decisions even if it wanted to, limiting it to primitive readahead or maybe statistics based on previously-seen patterns (if it sees that a request for block X is often followed by block Y, it may choose to prefetch that next time it sees block X accessed).

A problem in applications such as databases is when the outcome of an IO operation is required to initiate the next one - for example, you must first read an index to know the on-disk location of the actual row data. This is where the higher latency absolutely tanks performance.

A solution could be to make the storage drives smarter - have an NVME command that could say like "search in between this range for this byte pattern" and one that can say "use the outcome of the previous command as a the start address and read N bytes from there". This could help speed up the aforementioned scenario (effectively your drive will do the index scan & row retrieval for you), but would require cooperation between the application, the filesystem and the encryption system (typical, current FDE would break this).

Re: SSDs have become fast, except in the cloud

#258

Post by kikimora »

cynicalsecurity wrote: post #46
nullindividual wrote: post #10

Earlier quoted context omitted.

I don't like this answer. When I look at cloud, I get to think "finally! No more hardware to manage. No OS to manage". It's the best thing about the cloud, provided your workload is amenable to PaaS. It's great because I don't have to manage Windows or IIS. Microsoft does that part for me and significantly cheaper than it would be to employ me to do that work.

A cloud is just someone else's computer. When you rent a bare metal server, you don't manage your hardware either. The failed parts are replaced for you. Unless you can't figure out what hardware configuration you need - which would be a really big red flag for your level of expertise.

A cloud is network and tools, less hardware. For example see this [1] Reddit thread discussing network in Hetzner. Any other bare-metal would have same challenges. Once you solve network security you have to deal with server access. People hired and fired, hardcoded SSH keys is a bad idea. Once you solve access you likely have AD, LDAP and SSO of some sort. Then backups, and automated test suite + periodical test recoveries. Then database and backups. Then secrets, does all members of your team know production db password? And so on and on.

Maybe TCO still favors bare-metal but you have to spend a lot of time on configuration.

[1] https://www.reddit.com/r/hetzner/comments/rjuzcs/securing_ne...

Re: SSDs have become fast, except in the cloud

#259

Post by zokier »

jeffbee wrote: post #241
zokier wrote: post #230

Earlier quoted context omitted.

somehow I4g drives don't like to get formatted # nvme format /dev/nvme1 -n1 -f NVMe status: INVALID_OPCODE: The associated command opcode field is not valid(0x2001) # nvme id-ctrl /dev/nvme1 | grep oacs oacs : 0 but the LBA format indeed is sus: LBA Format 0 : Metadata Size: 0 bytes - Data Size: 512 bytes - Relative Performance: 0 Best (in use)

It's a shame. The recent "datacenter nvme" standards involving fb, goog, et al mandate 4K LBA support.

it'd be great if you'd manage to throw together quick blogpost about i4g io perf, there obviously something funny going on and I imagine you guys could figure it out much easier than anybody else, especially if you are already having some figures in the marketing.

Re: SSDs have become fast, except in the cloud

#260

Post by wmf »

wstuartcl wrote: post #226
jsnell wrote: post #71

Earlier quoted context omitted.

I think you're wrong about that. AWS calls this class of storage "instance storage" [0], and defines it as: > Many Amazon EC2 instances can also include storage from devices that are located inside the host computer, referred to as instance storage. There might be some wiggle room in "physically attached", but there's none in "storage devices located inside the host computer". It's not some kind of AWS-only thing eit…

the tests were for these local (metal direct connect ssds). The issue is not network overhead -- its that just like everything else in cloud the performance of 10 years ago was used as the baseline that carries over today with upcharges to buy back the gains. there is a reason why vcpu performance is still locked to the typical core from 10 years ago when every core on a machine today in those data scenters is 3-5x o…

vcpu performance is still locked to the typical core from 10 years ago

No. In some cases I think AWS actually buys special processors that are clocked higher than the ones you can buy.

Post reply on HN