Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

SSDs have become fast, except in the cloud

databasearchitects.blogspot.com

Post reply on HN
Print view View original

351–360 of 427 posts

Re: SSDs have become fast, except in the cloud

#351

Post by kilmanio »

cogman10 wrote: post #26

There's a 4th option. Cost. The fastest SSDs tend to also be MLC which tend to have much lower write life vs other technologies. This isn't unusual, increasing data density generally also makes it easier to increase performance. However, it's at the cost that the writes are typically done for a block/cell in memory rather than for single bits. So if one cell goes bad, they all fail. But even if that's not the problem…

Sure, MLC write life is worse than SLC, but significantly better than TLC and QLC

-edit: this comment was purely focused at your first sentence:

>The fastest SSDs tend to also be MLC which tend to have much lower write life vs other technologies.

I'm not sure what you mean with "other technologies" in this case, SLC is indeed truly expensive for a significantly higher write life, and HDDs are debatable for their lifespan.

Re: SSDs have become fast, except in the cloud

#352

Post by dijit »

dboreham wrote: post #306
c0l0 wrote: post #48

Seeing the really just puny "provisioned IOPS" numbers on hugely expensive cloud instances made me chuckle (first in disbelief, then in horror) when I joined a "cloud-first" enterprise shop in 2020 (having come from a company that hosted their own hardware at a colo). It's no wonder that many people nowadays, esp. those who are so young that they've never experienced anything but cloud instances, seem to have little…

Some of us are making a good living offboarding workloads from cloud onto bare metal with on-node NVMe storage.

Really? I'd like to do this as a job.

Are you hiring?

Cloud is great for prototyping or randomly elastic workloads, but it feels like people are pushing highly static workloads from on-prem to cloud. I'd love to be part of the change going the other way. Especially since the skills for doing so seem to have dried up completely.

Re: SSDs have become fast, except in the cloud

#353

Post by dijit »

bcaxis wrote: post #302
tumult wrote: post #288

Earlier quoted context omitted.

My experience lately is that consumer drives will also lie and use a cache, but then drop your data on the floor if the power is lost or there’s a kernel panic / BSOD. (Samsung and others.)

Rumors of that. I've never actually seen it myself.

Eh, I've definitely seen it.

I buy Samsung drives relatively exclusively if that makes any difference.

All that to say though: this is why things like journalling and write-ahead systems exist. OS design is mostly about working around physical (often physics related) limitations of hardware and one of those is what to do if you get caught in a situation where something is incomplete.

The prevailing methodology is to paper over it with some atomic actions. For example: Copy-on-Write or POSIX move semantics (rename(2)).

Then some spiffy young dev comes along and turns off all of those guarantees and says they made something ultra fast (*cough*mongodb*cough*) then maybe claims those guarantees are somewhere up the stack instead. This is almost always a lie.

Also: Beware any database that only syncs to VFS.

Re: SSDs have become fast, except in the cloud

#354

Post by pooper »

Szpadel wrote: post #154
LeifCarrotson wrote: post #73

Earlier quoted context omitted.

I wonder how many people have built failed businesses that never had enough customer data to exceed the DDR4 in the average developer laptop, and never had so many simultaneous queries it couldn't be handled by a single core running SQLite, but built the software architecture on a distributed cloud system just in case it eventually scaled to hundreds of terabytes and billions of simultaneous queries.

In may day job I often see systems that have the opposite. Especially for database queries, developers tested on local machine with 100s of records and everything was quick and snappy and on production with mere millions of records I often see queries taking minutes up to a hour just because some developer didn't see need for creating indexes or created query in a way there is no way to even create any index that wou…

This is a different topic and not always a skills issue. The stupid match for "productivity" and "velocity" means you have to cut corners.

Also Sometimes, it is poor communication. Just yesterday I saw some code that requests auth token before every request even though each bearer token comes with expires in (about twelve hours).

Re: SSDs have become fast, except in the cloud

#355

Post by joshstrange »

EB66 wrote: post #282
Aurornis wrote: post #249

Earlier quoted context omitted.

> It's no wonder that many people nowadays, esp. those who are so young that they've never experienced anything but cloud instances, seem to have little idea of how much performance you can actually pack in just one or two RUs today. On the contrary, young people often show up having learned on their super fast Apple SSD or a top of the line gaming machine with NVMe SSD. Many know what hardware can do. There’s no nee…

> because beyond simple single-purpose applications with flexible uptime requirements, that’s not a realistic option. I frequently hear this point expressed in cloud vs colo debates. The notion that you can't achieve high availability with simple colo deploys is just nonsense. Two colo deploys in two geographically distinct datacenters, two active physical servers with identical builds (RAIDed drives, dual NICs, A+B…

I doubt you’ll find anyone who disagrees that colo is much cheaper and that it’s possible to have failover with little to no downtime. Same with higher performance on bare metal vs a public cloud. Or at least I’ve never thought differently.

The difference is setting up all of that and maintaining it/debugging when something goes wrong is not a small task IMHO.

For some companies with that experience in-house I can understand doing it all yourself. As a solo founder and an employee of a small company we don’t have the bandwidth to do all of that without hiring 1+ more people which are more expensive than the cloud costs.

If we were drive-speed-constrained and getting that speed just wasn’t possible then maybe the math would shift further in favor of colo but we aren’t. Also upgrading the hardware our servers run on is fairly straightforward vs replacing a server on a rack or dealing with failing/older hardware.

Re: SSDs have become fast, except in the cloud

#356

Post by joshstrange »

rdoherty wrote: post #256
kuschku wrote: post #186

Earlier quoted context omitted.

Maybe I'm misunderstanding something, but that's about 2700 a second. Or about 3Mbps. Even a very unoptimized application running on a dev laptop can serve 1Gbps nowadays without issues. So what are the constraints that demand a complex architecture?

I'm not the OP but a few things: * Reading/fetching the data - usernames, phone number, message, etc. * Generating the content for each message - it might be custom per person * This is using a 3rd party API that might take anywhere from 100ms to 2s to respond, and you need to leave a connection open. * Retries on errors, rescheduling, backoffs * At least once or at most once sends? Each has tradeoffs * Stopping/star…

I literally spent the last week speccing out a system just like this and you are completely correct. You’ve touched on almost every single thing we ran into.

Re: SSDs have become fast, except in the cloud

#357

Post by Nemo_bis »

kccqzy wrote: post #138
scottlamb wrote: post #117

Earlier quoted context omitted.

> The problem is that this network is so large and slow that it can't give you anywhere near the performance of a local SSD. This wasn't a problem for hard drives, which was the backing technology when a lot of these network attached storage systems were invented, because they are fundamentally slow compared to networks, but it is a problem for SSD. Certainly true that SSD bandwidth and latency improvements are hard…

That document is probably deliberately on the pessimistic side to encourage your code to be portable across all kinds of "data centers" (however that is defined). When I previously worked at Google, the standard RPC system definitely offered 50 microseconds of round trip latency at the median (I measured it myself in a real application), and their advanced user-space implementation called Snap could offer about 10 mi…

50 microseconds is a lot. I'm looking at disk read latency on a bunch of baremetal servers (nothing fancy, just node_disk_read.* metrics from node-exporter) and one of the slowest fleets has a median disk read latency barely above 1 microsecond. (And that's with rather slow HDDs.)

Re: SSDs have become fast, except in the cloud

#359

Post by c0l0 »

tumult wrote: post #350
bcaxis wrote: post #302

Earlier quoted context omitted.

Rumors of that. I've never actually seen it myself.

I can get it to happen easily. 970 Evo Plus. Write a text file and kill the power within 20 seconds or so, assuming not much other write activity. File will be zeroes or garbage, or not present on the filesystem, after reboot.

This happens for you after you invoked an explicit sync() (et al.) before the power cut?

Re: SSDs have become fast, except in the cloud

#360

Post by kuschku »

rdoherty wrote: post #256
kuschku wrote: post #186

Earlier quoted context omitted.

Maybe I'm misunderstanding something, but that's about 2700 a second. Or about 3Mbps. Even a very unoptimized application running on a dev laptop can serve 1Gbps nowadays without issues. So what are the constraints that demand a complex architecture?

I'm not the OP but a few things: * Reading/fetching the data - usernames, phone number, message, etc. * Generating the content for each message - it might be custom per person * This is using a 3rd party API that might take anywhere from 100ms to 2s to respond, and you need to leave a connection open. * Retries on errors, rescheduling, backoffs * At least once or at most once sends? Each has tradeoffs * Stopping/star…

Oh, I absolutely agree that the complexity is in these topics. I'm just sceptic that they're enough to turn a task that could run on a laptop into one that requires an entire cluster of machines.

The third party API is the part that has the potential to turn this straightforward task into a byzantine mess, though, so I suspect that's the missing piece of information.

I'm comparing this to my own experience with IRC, where handling the same or larger streams of messages is common. And that's with receiving this in real time, storing the messages, matching and potentially reacting to them, and doing all that while running on a raspberry pi.

Post reply on HN