Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Where's my petabyte disk drive?

bit-player.org

Post reply on HN
Print view View original

21–30 of 101 posts

Re: Where's my petabyte disk drive?

#21

Post by 13of40 »

Kenji wrote: post #8

The reason why spinning disks still stick around is not just because of the price. Often, it is the case that you do not need 200-500MBit/s and ~100+ are enough. I have my OS on a (small) SSD and my spinning disks filled with my documents, music, movies. All of this does not need high speed, so it would be wasted money to buy an SSD for it. I'd rather buy twice the storage and do RAID1 (which I did).

It's a mystery to me that hybrid SSD+HD drives aren't more ubiquitous. I can guess what data is going to be read off the drive more frequently, but the computer can collect statistics and make a way more accurate prediction than I can.

Re: Where's my petabyte disk drive?

#22

Post by derefr »

wtbob wrote: post #12

Why did he write '\(2\frac{1}{2}\) or \(3\frac{1}{2}\)' (which presumably uses JavaScript to be rendered), when he could have just written '2½ or 3½'? It'd have taken 8 bytes to store instead of 36 — and that's not even counting the size of the JavaScript! As long as folks keep on reinventing the wheel, only bigger, hard drives are going to have to keep increasing in size.

Besides laziness, perhaps there's a slight semantic justification—I imagine there exists at least a few mathematical solvers that can ingest data of the TeX form and work with it, but probably fewer that understand how to ingest Unicode codepoints.

Re: Where's my petabyte disk drive?

#23

Post by dalke »

PaulHoule wrote: post #2

Note in the early 1990s you could read hard drive content pretty easily with a scanning probe microscope, in fact the bits were lozenge shaped and if you looked close you might find the edges of bits that had been written before and weren't perfectly aligned. (i.e. scientists had tools in their labs that were better at reading a hard drive than the read head) Back then it was possible you could smash the plates and s…

"in the early 1990s you could read hard drive content pretty easily with a scanning probe microscope"

I believe that is an urban legend. http://all.net/ForensicsPapers/2012-12-07-OverwrittenMagneti... describes attempts to track down such cases:

> To date I have found no example of any instance in which digital data recorded on a hard disk drive and subsequently overwritten was recovered from such a drive since 1985, when about 15% of the overwritten data was claimed to have been recovered from an modified frequency modulation (MFM) disk drive.

It cites "Overwriting Hard Drive Data: The Great Wiping Controversy" at http://www.vidarholen.net/~vidar/overwriting_hard_drive_data... which gives a best case example of a pristine hard drive, written once and then wiped once, and where you know the data is located before hand. Even then nearly all of the data had disappeared. If the drive was not pristine, it was not possible to recover the data. Quoting from it (emphasis mine):

> The purpose of this paper was a categorical settlement to the controversy surrounding the misconceptions involving the belief that data can be recovered following a wipe procedure. This study has demonstrated that correctly wiped data cannot reasonably be retrieved even if it is of a small size or found only over small parts of the hard drive. Not even with the use of a MFM or other known methods. The belief that a tool can be developed to retrieve gigabytes or terabytes of information from a wiped drive is in error.

> Although there is a good chance of recovery for any individual bit from a drive, the chances of recovery of any amount of data from a drive using an electron microscope are negligible. Even speculating on the possible recovery of an old drive, there is no likelihood that any data would be recoverable from the drive. The forensic recovery of data using electron microscopy is infeasible. This was true both on old drives and has become more difficult over time. Further, there is a need for the data to have been written and then wiped on a raw unused drive for there to be any hope of any level of recovery even at the bit level, which does not reflect real situations. It is unlikely that a recovered drive will have not been used for a period of time and the interaction of defragmentation, file copies and general use that overwrites data areas negates any chance of data recovery. The fallacy that data can be forensically recovered using an electron microscope or related means needs to be put to rest.

Re: Where's my petabyte disk drive?

#24

Post by ChuckMcM »

bit-player wrote: post #11
evmar wrote: post #9

Massive hard drives only useful for archival purposes, under this argument: if your hard drive is being used for live queries, you want to access all the data on it. Even if you have have all the pieces in place to stream 1gig per sec off of the drive, a 1pb drive would still take 1 million seconds = ~11.5 days to read. So in practice in production it's more useful to have smaller hard drives in more places to work o…

This is an important point. The Google white paper does address it, though only in the abstract: They want to maximize both capacity and IO bandwidth. One can imagine large disks with multiple independently actuated head assemblies. But quite possibly you're right that it's not worth the both.

The interesting thing about Google's storage infrastructure was that teams were optimizing IOPs per drive and talking to thousands of them over a gigabit link. I had an interesting conversation with Sean about that one day, asking him if he got 100 IOPs per drive, and had 10,000 drives, and a gigabit ethernet port, how much data on the drive could be part of any service being provided over that gigabit link? Plot that over a RPS (generic 'requests per second').

In my case I was trying to get him to sign off on powering down some of the drives that could not be reached to save power. But even with the data staring him in the face he could not go there. Network bandwidth gets better, and that exposes more data to the pipeline, but if you want < 500mS request response you have to balance the system.

Re: Where's my petabyte disk drive?

#25

Post by abecedarius »

> Is this notion of merging memory and storage an attractive prospect or a nightmare? I’m not sure. There are some huge potential problems. For safety and sanity we generally want to limit which programs can alter which documents. Those rules are enforced by the file system, and they would have to be re-engineered to work in the memory-mapped environment.

This was done back in the 80s in http://www.cis.upenn.edu/~KeyKOS/ . A favorite demo reportedly was to pull the plug on a running computer then start up again. They took the need to redesign security as an opportunity to make it better.

Re: Where's my petabyte disk drive?

#26

Post by mikhail-g-kan »

I suppose the need of volume of consumer disks will depend on future networks qualities. With low latency and high speed users won't need big local disks, services like cloud storage/streaming will be enough. We will always find ways to use higher capacity disks, i.e. high-def VR content, or biochemistry data of our bodies for personal healthcare/fitness

Re: Where's my petabyte disk drive?

#27

Post by dalke »

evmar wrote: post #9

Massive hard drives only useful for archival purposes, under this argument: if your hard drive is being used for live queries, you want to access all the data on it. Even if you have have all the pieces in place to stream 1gig per sec off of the drive, a 1pb drive would still take 1 million seconds = ~11.5 days to read. So in practice in production it's more useful to have smaller hard drives in more places to work o…

I've toyed with the idea that if you buy a 1PB hard disk it will come pre-loaded with, say, a copy of archive.org and search indices, or a large collection of movies. Also, if everyone had the same reference data set then (handwaving!) it could make for some fantastic compression methods.

I don't think the economics works out, since at 1G/s it probably takes too long to load the data, and as this essay points out most people will stream what they want on demand. I also doubt there will be a standard content set which is around long enough to assure that my imagined on-the-fly compression model-building-by-corpus-reference will take root.

Re: Where's my petabyte disk drive?

#28

Post by icelancer »

lucb1e wrote: post #14
drostie wrote: post #7

We appear to be slashdotting the server; here's some cached copies: Just text (loads instantly): http://webcache.googleusercontent.com/search?q=cache:Jr34hZX... Images etc. loaded from the site (seems to work, albeit slowly): http://webcache.googleusercontent.com/search?q=cache:Jr34hZX...

> I found the problem.

How people use Wordpress without a caching plugin today... sigh.

Re: Where's my petabyte disk drive?

#29

Post by jsprogrammer »

rgbrenner wrote: post #10

The growth in hard drive space is called Kryder's Law [0] (like Moore's law). There's a paper from 2012 on the cost of long-term storage [1].. and here's a quote from the researchers blog[2]: Here is a graph I got from Dave Anderson [director of strategic planning at Seagate] years ago. It shows that what looks like a smooth Kryder's Law curve is actually the superposition of a series of S-curves, one for each succes…

So, the rate if increase (in density) is slowed by lack of demand for storage?

Re: Where's my petabyte disk drive?

#30

Post by pilsetnieks »

spullara wrote: post #20

He is wrong about 6TB being the biggest hard drive in the market. Now that SSDs have become the standard they are taking up the curve. Samsung announced a 16TB SSD last year. I don't expect people to devote as much effort to making spinning disks bigger. http://arstechnica.com/gadgets/2015/08/samsung-unveils-2-5-i...

He mentioned that:

> As the pace of magnetic disk development slackens, an alternative storage medium is coming on strong. Flash memory, a semiconductor technology, has recently surpassed magnetic disk in areal density; Micron Technologies reports a laboratory demonstration of 2.7 terabits per square inch. And Samsung has announced a flash-based solid-state drive (SSD) with 15 terabytes of capacity, larger than any mechanical disk drive now on the market. SSDs are still much more expensive than mechanical disks—by a factor of 5 or 10—but they offer higher speed and lower power consumption. They also offer the virtue of total silence, which I find truly golden.

Post reply on HN