Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Where's my petabyte disk drive?

bit-player.org

Post reply on HN
Print view View original

51–60 of 101 posts

Re: Where's my petabyte disk drive?

#51

Post by greglindahl »

digi_owl wrote: post #46
evmar wrote: post #9

Massive hard drives only useful for archival purposes, under this argument: if your hard drive is being used for live queries, you want to access all the data on it. Even if you have have all the pieces in place to stream 1gig per sec off of the drive, a 1pb drive would still take 1 million seconds = ~11.5 days to read. So in practice in production it's more useful to have smaller hard drives in more places to work o…

I seem to recall reading some worries that current sizes are already pushing the limits of RAID setups. This in that it takes so much time to rebuild a drive given current interfaces that you risk one of the others in the array to fail during the process, thus making recovery impossible.

... which is why many big storage systems keep 3 replicas of each chunk of data stored across different servers. A failed device's chunks can be rebuilt in parallel across the entire cluster.

Or, if you care capacity-constrained, you can use cross-server erasure coding. And even call it cross-server RAID, if you like.

Re: Where's my petabyte disk drive?

#52

Post by kyberias »

aninhumer wrote: post #16
wtbob wrote: post #12

Why did he write '\(2\frac{1}{2}\) or \(3\frac{1}{2}\)' (which presumably uses JavaScript to be rendered), when he could have just written '2½ or 3½'? It'd have taken 8 bytes to store instead of 36 — and that's not even counting the size of the JavaScript! As long as folks keep on reinventing the wheel, only bigger, hard drives are going to have to keep increasing in size.

If you already have the maths rendering script in place for more complicated uses, and you don't already know a quick way to insert the ½ character, then it's just the path of least resistance.

I was wondering what you're talking about but then realized that the US keyboard layout does not have the ½ symbol right there in the keyboard like we have in Europe.

Re: Where's my petabyte disk drive?

#53

Post by animex »

Could the reality be that storage needs for the average user have plateau'd for a combination of reasons such as network/internet speed limitations, rise of cloud computing, rise of streamed movie subscriptions (netflix, etc) thus muting the financial benefit to develop new technology to drive exponential drive storage technology forward. I think once we have a reason to grow our storage again, we could see storage technology pick up again.

Re: Where's my petabyte disk drive?

#54

Post by aab0 »

You could also ask what ended the kink. The timing looks like it matches up with the Thai floods, which killed price decreases for years afterwards. The experience curve means you need bulk to achieve improvements, and the floods shattered a key part of the supply chain.

Re: Where's my petabyte disk drive?

#55

Post by Razengan »

jsprogrammer wrote: post #29
rgbrenner wrote: post #10

The growth in hard drive space is called Kryder's Law [0] (like Moore's law). There's a paper from 2012 on the cost of long-term storage [1].. and here's a quote from the researchers blog[2]: Here is a graph I got from Dave Anderson [director of strategic planning at Seagate] years ago. It shows that what looks like a smooth Kryder's Law curve is actually the superposition of a series of S-curves, one for each succes…

So, the rate if increase (in density) is slowed by lack of demand for storage?

> lack of demand for storage

and in turn, demand for storage is slowed by lack of need for storage.

The real question should be: Where's my 4K video? My 10-bit-channel images? My lossless audio?

Yes they are here but not nearly as common as content delivered in ancient formats from over a decade ago. For example go on a wallpapers subreddit, the vast majority is still in 8-bit 1920x1080 JPEG. Video is still mostly 1080. The leading music services still deliver in 2-channel lossy formats. And this is 2016.

Most of this I suppose is because of the general slowness of the internet and the usage caps in many areas of the world.

So maybe improve internet service -> create more detailed content and let people save it -> people will want to demand more storage to store all that in.

Re: Where's my petabyte disk drive?

#56

Post by Kenji »

mrob wrote: post #48
13of40 wrote: post #43

Earlier quoted context omitted.

I'd assert that it could make a better prediction for the majority of the files stored on your drive. For example, even if it was technically feasable to do with separate drives, do you know which files in the global assembly cache should go on SSD versus HD? Which pieces of your registry hive files? Which bits of your browser cache?

I don't use Windows so I don't have GAC or registry files. Browser cache always goes on the SSD because it's highly latency sensitive.

You say browser cache always goes on the SSD (I do that too, whenever there's no configuration to put it into RAM, which I prefer). Aren't you concerned about wear? If you stream high resolution video files from, say, youtube or twitch, you write gigabyte after gigabyte into that SSD.

Re: Where's my petabyte disk drive?

#57

Post by vegabook »

drostie wrote: post #7

We appear to be slashdotting the server; here's some cached copies: Just text (loads instantly): http://webcache.googleusercontent.com/search?q=cache:Jr34hZX... Images etc. loaded from the site (seems to work, albeit slowly): http://webcache.googleusercontent.com/search?q=cache:Jr34hZX...

nice old-time "slashdotting" jargo, though I doubt that Slashdot does any slashdotting anymore (sadly). "HackerNewsing" just doesn't do it, does it.

Re: Where's my petabyte disk drive?

#58

Post by ladzoppelin »

ChrisDutrow wrote: post #38
archiebunker wrote: post #4

Excellent post. Great information. I have a question about SSDs, though. Google has published their information about hard drive and SSD survival in their data centers. It can be viewed here: http://www.datacenterdynamics.com/servers-storage/googles-ss... So my question is how we mere mortals can deal with all the maintenance. It may well be that spinning hard drives are best for us.

I'm not sure spinning hard drives are more reliable outside of data centers - at least not for laptops because you are carrying them around (anecdotaly, I recently had en employee break a hard drive in a laptop after dropping it). It probably doesn't really matter as much as your intuition might suggest anyway because each drive can and will fail. And they fail at similar rates (as opposed to an exponential differenc…

"The I put all the information I don't want to lose in dropbox." I don't think its healthy to consider cloud storage backup. For example I am pretty sure you could loose everything to Ransomware.

Re: Where's my petabyte disk drive?

#59

Post by aninhumer »

kyberias wrote: post #52
aninhumer wrote: post #16

Earlier quoted context omitted.

If you already have the maths rendering script in place for more complicated uses, and you don't already know a quick way to insert the ½ character, then it's just the path of least resistance.

I was wondering what you're talking about but then realized that the US keyboard layout does not have the ½ symbol right there in the keyboard like we have in Europe.

There are a wide variety of keyboard layouts in use in Europe, and a quick investigation suggests few include ½.

What country are you from?

Re: Where's my petabyte disk drive?

#60

Post by mrob »

Kenji wrote: post #56
mrob wrote: post #48

Earlier quoted context omitted.

I don't use Windows so I don't have GAC or registry files. Browser cache always goes on the SSD because it's highly latency sensitive.

You say browser cache always goes on the SSD (I do that too, whenever there's no configuration to put it into RAM, which I prefer). Aren't you concerned about wear? If you stream high resolution video files from, say, youtube or twitch, you write gigabyte after gigabyte into that SSD.

Modern SSDs will survive hundreds of TBs of writes, sometimes even PBs. Techreport did a long endurance test with multiple drives:

http://techreport.com/review/27909/the-ssd-endurance-experim... "All of the drives surpassed their official endurance specifications by writing hundreds of terabytes without issue."

I expect to upgrade for increased capacity long before I reach that.

Post reply on HN