Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Where's my petabyte disk drive?

bit-player.org

Post reply on HN
Print view View original

71–80 of 101 posts

Re: Where's my petabyte disk drive?

#71

Post by Viper007Bond »

digi_owl wrote: post #46
evmar wrote: post #9

Massive hard drives only useful for archival purposes, under this argument: if your hard drive is being used for live queries, you want to access all the data on it. Even if you have have all the pieces in place to stream 1gig per sec off of the drive, a 1pb drive would still take 1 million seconds = ~11.5 days to read. So in practice in production it's more useful to have smaller hard drives in more places to work o…

I seem to recall reading some worries that current sizes are already pushing the limits of RAID setups. This in that it takes so much time to rebuild a drive given current interfaces that you risk one of the others in the array to fail during the process, thus making recovery impossible.

My consumer storage RAIDs already take 1-2 days to rebuild. I can't imagine larger enterprise ones!

Re: Where's my petabyte disk drive?

#72

Post by abecedarius »

capitalsigma wrote: post #62
abecedarius wrote: post #25

> Is this notion of merging memory and storage an attractive prospect or a nightmare? I’m not sure. There are some huge potential problems. For safety and sanity we generally want to limit which programs can alter which documents. Those rules are enforced by the file system, and they would have to be re-engineered to work in the memory-mapped environment. This was done back in the 80s in http://www.cis.upenn.edu/~Key…

I'm not sure his idea about "merging memory and storage" really makes sense. He says that he wants load instructions to be able to hit the disk in order to avoid "calls to input/output routines in the operating system." But you can't avoid the input/output routines --- he's effectively saying that we should hardcode our filesystems into a single machine instruction and let the processor figure it out. If anything, we…

saying that we should hardcode our filesystems into a single machine instruction and let the processor figure it out

I think he was rather saying that the OS could do it: persistent virtual memory as the primary abstraction. In Unix, files and processes are different kinds of things; in KeyKOS there were only processes; RAM was effectively a cache. As Unix directories have links to files, KeyKOS processes could be given capabilities to invoke other processes (passing capabilities and data as arguments). The different security model makes this analogy misleading, but you can see how you could emulate a filesystem.

What assumptions do you mean?

Re: Where's my petabyte disk drive?

#73

Post by icelancer »

ChrisDutrow wrote: post #34
icelancer wrote: post #28

Earlier quoted context omitted.

How people use Wordpress without a caching plugin today... sigh.

What caching plugin do you recommend?

W3 Total Cache generally works fine. I use it with memcached and PHP's new built-in caching module (replacement for APC) and get my site blitzed from time to time due to major mentions in the media with few problems. SSD on Digital Ocean probably helps with disk caching too.

I use Apache but falcolas' point of nginx (often in combination with Varnish and potentially hhvm; I've used this combo before with great success) is worth considering as well.

No replacement for good configuration of your database (try MySQLtuner.pl if you use MySQL/MariaDB) of course.

Re: Where's my petabyte disk drive?

#74

Post by Chronic51 »

coldtea wrote: post #67
Razengan wrote: post #55

Earlier quoted context omitted.

> lack of demand for storage and in turn, demand for storage is slowed by lack of need for storage. The real question should be: Where's my 4K video? My 10-bit-channel images? My lossless audio? Yes they are here but not nearly as common as content delivered in ancient formats from over a decade ago. For example go on a wallpapers subreddit, the vast majority is still in 8-bit 1920x1080 JPEG. Video is still mostly 10…

> Yes they are here but not nearly as common as content delivered in ancient formats from over a decade ago. For example go on a wallpapers subreddit, the vast majority is still in 8-bit 1920x1080 JPEG. Video is still mostly 1080. The leading music services still deliver in 2-channel lossy formats. And this is 2016. Ever experienced YouTube buffering on an average connection? And that's for the metropolitan US -- con…

The rest of the world doesn't matter. If there is sufficient demand for 4K video and faster connection speeds, there will eventually be a product for them.

Re: Where's my petabyte disk drive?

#75

Post by TrickedOut »

Razengan wrote: post #55
jsprogrammer wrote: post #29

Earlier quoted context omitted.

So, the rate if increase (in density) is slowed by lack of demand for storage?

> lack of demand for storage and in turn, demand for storage is slowed by lack of need for storage. The real question should be: Where's my 4K video? My 10-bit-channel images? My lossless audio? Yes they are here but not nearly as common as content delivered in ancient formats from over a decade ago. For example go on a wallpapers subreddit, the vast majority is still in 8-bit 1920x1080 JPEG. Video is still mostly 10…

Sure, perhaps there isnt strong consumer demand. But with analysis and value-extraction from big-data becoming easier, and machine learning becoming more accessible, i think there will be plenty of commercial and industrial demand which can drive up need.

Re: Where's my petabyte disk drive?

#77

Post by TheOtherHobbes »

dalke wrote: post #27
evmar wrote: post #9

Massive hard drives only useful for archival purposes, under this argument: if your hard drive is being used for live queries, you want to access all the data on it. Even if you have have all the pieces in place to stream 1gig per sec off of the drive, a 1pb drive would still take 1 million seconds = ~11.5 days to read. So in practice in production it's more useful to have smaller hard drives in more places to work o…

I've toyed with the idea that if you buy a 1PB hard disk it will come pre-loaded with, say, a copy of archive.org and search indices, or a large collection of movies. Also, if everyone had the same reference data set then (handwaving!) it could make for some fantastic compression methods. I don't think the economics works out, since at 1G/s it probably takes too long to load the data, and as this essay points out mos…

The problem is that new content is being generated at an increasing rate.

There was an estimate in 2011 that total storage of everything everywhere by everyone was more than 250 Exabytes, increasing by around 25% annually.

There's going to be a lot of duplication in that, and a lot of it won't be public. So as a ballpark guess a complete collection of public-only sources - including all available commercial content of all possible kinds ever recorded, academic papers, Wikis, news sites, forums, and such - is going to need 25-50 Exabytes, with maybe 25% compound of new content every year.

So you could get the entire Internet delivered by truck or two, but you probably wouldn't have anywhere to put it.

Re: Where's my petabyte disk drive?

#78

Post by jimothyhalpert7 »

At least you don't have to worry about erasing your childhood to store 80GB of data, when you apply for that courier position in 4 years. [1][2]

[1] Johnny Mnemonic - Official Trailer - https://www.youtube.com/watch?v=Uwl5MBzTCRQ

[2] Also, I guess Toshiba is in a lawsuit with Apple over their new VR headset - the EyePhone - https://www.youtube.com/watch?v=vXSqN7qXwpU

Re: Where's my petabyte disk drive?

#79

Post by roywiggins »

douche wrote: post #61
Razengan wrote: post #55

Earlier quoted context omitted.

> lack of demand for storage and in turn, demand for storage is slowed by lack of need for storage. The real question should be: Where's my 4K video? My 10-bit-channel images? My lossless audio? Yes they are here but not nearly as common as content delivered in ancient formats from over a decade ago. For example go on a wallpapers subreddit, the vast majority is still in 8-bit 1920x1080 JPEG. Video is still mostly 10…

For a lot of things, that extra resolution is not really any benefit. Particularly for content that wasn't ever recorded at that level of fidelity anyway - I don't need 4k versions of Seinfeld re-runs, or SpongeBob SquarePants for the kiddos. Most of that stuff doesn't even need to be 720p. Besides, pirated video content was really the only thing that most normal people could fill up their hard drives with (okay, may…

Also people who shoot raw photos. They're like 35MB each, an order of magnitude bigger than jpegs. They can eat up gigabytes (though to be fair, not terabytes) pretty fast. I have about 525GB of photos and videos that I've taken over years. Something like 300GB is jpegs. If I'd exclusively shot raw files I would need 3-4TB to store everything.

Re: Where's my petabyte disk drive?

#80

Post by hga »

abecedarius wrote: post #25

> Is this notion of merging memory and storage an attractive prospect or a nightmare? I’m not sure. There are some huge potential problems. For safety and sanity we generally want to limit which programs can alter which documents. Those rules are enforced by the file system, and they would have to be re-engineered to work in the memory-mapped environment. This was done back in the 80s in http://www.cis.upenn.edu/~Key…

First done as far as I can remember with Multics in the 1960s. The people who worked on IBM's canceled successor to the System/360 used it for System/38 in 1979, with capabilities, a feature which was dropped for the successor AS/400/iSeries/System i.
Post reply on HN