Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Modern storage is plenty fast, but the APIs are bad

itnext.io

Post reply on HN
Print view View original

91–100 of 168 posts

Re: Modern storage is plenty fast, but the APIs are bad

#91

Post by selectodude »

blindm wrote: post #26
Felk wrote: post #9

Earlier quoted context omitted.

I think having a backup solution is the better choice here. You can use your SSDs until they die or become too slow, and you won't lose your data if it breaks before you replace it after a year

> I think having a backup solution is the better choice here Any particular provider you would recommend? I've looked into backblaze but it seems a bit pricey. Also: I am aware that cloud based backup solutions have very little failure rate in terms of drives since they're probably using RAID

$6/mo?

Re: Modern storage is plenty fast, but the APIs are bad

#92

Post by tpurves »

mikepurvis wrote: post #67
danuker wrote: post #58

Earlier quoted context omitted.

> network storage Do you mean cloud storage? as in, other people's computers?

It's a pretty common pattern to have a fleet of big beefy VM hosts all backed by a single giant SAN on a 10gbe switch. This lets you do things like seamlessly migrate a VM from one host to another, or do a high availability thing with multiple synchronized instances and automatic failover (VMWare called this all "vMotion"). In any case, lots of bandwidth to the storage, but also high latency, at least relative to a l…

Why not consider nVME in this case then as cheaper than RAM, slower than RAM, but faster than network storage? I don't know how you handle concurrency btwn VMs or virtualize that storage, but there must be some standard for that?

Re: Modern storage is plenty fast, but the APIs are bad

#93

Post by wtallis »

ggm wrote: post #89
blindm wrote: post #4

Also: Modern storage is plenty fast, but also not reliable for long term use. That is why I buy a new SSD every year and clone my current (worn out) SSD to the new one. I have several old SSDs that started to get unhealthy, well, according to my S.M.A.R.T utility that I used to check them. I could probably get away with using an SSD for another year, but will not risk the data loss. Anyone else do this?

Can somebody please write up modern SSD and state of the world regarding data retention, modes, applicability for SSD replacing spinning rust "on the shelf" offline...

SSDs make no sense for offline archival. They're more expensive than hard drives and will be for the foreseeable future. You don't need the improved random IO performance or power efficiency for a drive that's mostly sitting on a shelf.

Re: Modern storage is plenty fast, but the APIs are bad

#95

Post by stingraycharles »

Miraste wrote: post #81
arka2147483647 wrote: post #55

Earlier quoted context omitted.

I read that as that it was what he had at hand when running the tests. Also, if the speed and features are available as professional grade devices today, it will be available everywhere in a few years.

Optane has been commercially available for five years already and it's not used in any device I'm aware of. Assuming it will find broad adoption at this point seems like a bad bet.

I know a few optane deployments in finance, but other than that, it seems incredibly difficult to justify the steep price.

Re: Modern storage is plenty fast, but the APIs are bad

#96

Post by wtallis »

fibers wrote: post #88
eightysixfour wrote: post #79

Earlier quoted context omitted.

Yes, using PCIe expansion cards. I know of an AMD board that ships with 5 (3 on the board, 2 with a PCIe card). Could easily add more.

is this with threadripper boards?

3 M.2 slots is common even on AMD's mainstream X570 and B550 platforms. I don't know if any of those motherboards also bundle riser cards for further M.2 PCIe SSDs, but they do support PCIe bifurcation so you can run your GPU at PCIe 4.0 x8 and use the second x16/x8 slot to run two more SSDs in a passive riser purchased separately.

Re: Modern storage is plenty fast, but the APIs are bad

#97

Post by mikepurvis »

tpurves wrote: post #92
mikepurvis wrote: post #67

Earlier quoted context omitted.

It's a pretty common pattern to have a fleet of big beefy VM hosts all backed by a single giant SAN on a 10gbe switch. This lets you do things like seamlessly migrate a VM from one host to another, or do a high availability thing with multiple synchronized instances and automatic failover (VMWare called this all "vMotion"). In any case, lots of bandwidth to the storage, but also high latency, at least relative to a l…

Why not consider nVME in this case then as cheaper than RAM, slower than RAM, but faster than network storage? I don't know how you handle concurrency btwn VMs or virtualize that storage, but there must be some standard for that?

I think a lot of it depends what the machines are used for. I'm not actually the IT department, but I believe in my org, we started out with a SAN-backed high availability cluster, because the immediate initial need was getting basic infrastructure (wiki, source control, etc) off of a dedicated machine that was a single point of failure.

But then down the road a different set of hosts were brought online that had fast local storage, and those were used for short term, throwaway environments like Jenkins builders, where performance was far more important than redundancy.

Re: Modern storage is plenty fast, but the APIs are bad

#98

Post by echlebek »

Miraste wrote: post #81
arka2147483647 wrote: post #55

Earlier quoted context omitted.

I read that as that it was what he had at hand when running the tests. Also, if the speed and features are available as professional grade devices today, it will be available everywhere in a few years.

Optane has been commercially available for five years already and it's not used in any device I'm aware of. Assuming it will find broad adoption at this point seems like a bad bet.

Consumer NVMe devices can deliver GB/s I/O and hundreds of thousands of iops. The article's point doesn't hinge on Optane at all.

Re: Modern storage is plenty fast, but the APIs are bad

#99

Post by PaulDavisThe1st »

rtkwe wrote: post #57
mehrdadn wrote: post #32

I liked most of the piece, but some bits rubbed me the wrong way: > I was taken by surprise by the fact that although every one of my peers is certainly extremely bright, most of them carried misconceptions about how to best exploit the performance of modern storage technology leading to suboptimal designs, even if they were aware of the increasing improvements in storage technology. > In the process of writing this…

Yeah how many people are running apps on servers served at all or even partially by NVMe SSDs? Where I work for our on prem stuff it's basically all network storage.

There's more to life than "apps on servers".

People doing audio/video work with lots of input streams can also max out disk I/O throughput (quite easily before NVMe SSDs; not so much anymore).

Re: Modern storage is plenty fast, but the APIs are bad

#100

Post by hinkley »

ivoras wrote: post #2

> ...misconceptions... Yet if you skim through specs of modern NVMe devices you see commodity devices with latencies in the microseconds range and several GB/s of throughput supporting several hundred thousands random IOPS. So where’s the disconnect? Whoa there... let's not compare devices with 20+ GB/s and latencies in nanosecond ranges which translate to half a dozen giga-ops per second (aka RAM) with any kind of f…

We’ve been using Ethernet cards for storage because the network round trip to RAM over TCP/IP on another machine in the same rack is far cheaper than accessing local storage. Latency compared to that option is likely the most noteworthy performance gain.

My understanding of distributed computing history is that the last time network>local storage happened was in the 80’s, and most of the rest of the history of computing, moving the data physically closer to the point of usage has always been faster.

Just as then, we’ve taken a pronounced software architecture detour. This one has lasted much longer, but it can’t really last forever. With this new generation of storage, we’ll probably see a lot of people trotting out 90’s era system designs as if they are new ideas rather than just regression to the mean.

Same as it ever was.

Post reply on HN