Live data from Hacker News

Cost of storing all human audio-visual experiences

lists.w3.org

21–30 of 40 posts

Re: Cost of storing all human audio-visual experiences

#21
post #6

Earlier quoted context omitted.

Have we learnt from the middle ages, or do we find the circumstances too different from now to draw authoritative conclusions? (honest question)

Yes I think in some regards we definitely have - like taking science as fact rather than religion as fact - is mostly the status quo. That certainly wasnt the case 1000 years ago.

Depends where you live. I've toured entire counties that seem to think Al Gore is the devil for daring to say their hummers are melting ice caps.

But on principle, those are the kinds of people that didn't study history. Or much of anything, at all. So we do have evidence we learn from our history, just that we don't inspire enough people to learn about it.

Re: Cost of storing all human audio-visual experiences

#22

assuming somekind of google glass attached to a giant world wide data storage. Could it be compressed? (which got me to thinking about compression and the larger the size of the data stored the more likely the chance of larger and larger streams being duplicate. How many parts of the image do I have to store of my desk. Also, could we limit it to interesting moments(how many times do I have to record me eating cheeri…

Presumably you could also use lossy compression across users, especially for the boring parts. I imagine that, within the confines of my bowl, my milk and cheerios look pretty similar to yours. Do the actual differences matter, as a record of some particular breakfast experience? I suppose only if they float into some surprising configuration.

Re: Cost of storing all human audio-visual experiences

#23

"If storage prices continue to develop as they have since the 1980s," bzzzt and thanks for playing. Remember this truism, "In real life, exponentials are s-curves." That said, there are folks today who record 24x7 stereo 44khz as a 'life diary.' It isn't that hard to store these days and compress down. One could imagine adding video to the mix.

>bzzzt and thanks for playing. There may be tens or hundreds or even thousands of years before that exponential curve looks like an s-curve, making his point more relevant that yours.

Well I've been pretty involved in storage for the last 10 years, 5 of them at Network Appliance and then later another 4 at Google dealing with their storage scale. And worked with both Seagate, Fujitsu, and to a lesser extent Hitachi as they have worked to increase the density of what can be reliably recalled. More interesting has been watching the struggle of thermal noise and the ability to push past areal densities of 300 - 400 Gbits/inch^2.

There are certainly nano-scale technologies which seek to store information in the 'spin' of electrons (I'd be hard pressed to see you get better than that) but may become impractical if the network bandwidth gets to the point where the size doesn't matter any more. Specifically, what matters to the end consumer is that they can get what they stored into something which can use that information. If it comes from disk across the room that is just as good as a local on board disk if the bandwidth is the same.

The effect this reality is having on storage is that fatter (and increasingly more fragile) drives, are becoming less useful to consumers than larger but more reliable storage attached via a protocol (be it iSCSI, iCloud, S3, or NFS).

So once people are unwilling to pay to carry it around, the ability to recoup your investment in making it possible to carry around a device that can read electron spin is less and less likely. And your s-curve will become clear, when for the last 5 years the drives have all been at most 6 maybe 10TB and the price of those doesn't seem to fall all that much.

I realize disk growth has been phenomenal "your whole life" as CPU performance growth was for most of mine, but CPU performance growth has kinda sputtered big time. We've been multi-coring for a relatively long time now. Storage systems (especially random access read/write systems) are in the same boat. But the glory days are over.

I am curious to see if storage will have its 'multi-core' moment. That will be interesting if it happens.

Re: Cost of storing all human audio-visual experiences

#24

Once whatever successor to Google Glass becomes ubiquitous, I can see this becoming a reality. This is an advertising research agency's wet dream come true, unfortunately.

I wonder, with personal income on the decline this past decade, will advertisers spend less money on adverts or more on people who have more disposable income from mined data?

Doubt it. If firms could analyze and deconstruct the purchasing decisions and preferences from such 24/7 feeds, it will just allow them to more efficiently funnel money from the consumer to the top. The lower income market is intensely profitable, mostly due to the limitation of choice a disenfranchised person has.

It's disconcerting that, literally knowing everything about you, puts advertisers in a position to accurately target you in insidious ways.

Advertising is incredibly good at convincing people that they need [product] as it is. Assuming the 'rational actor' theory ever treaded water, this sinks it.

It does seem like the logical progression for Google and friends. What will be interesting is how such invasive monitoring will be pitched as innocuous. People seem to be a-okay with corporations mining their data, though.

Re: Cost of storing all human audio-visual experiences

#25

It's funny, because in 1000 years, this is an anthropologists dream come true (assuming the storage can be preserved that long by eventually copying to a less volatile medium). Is it worth sacrificing our privacy now, so that their perfect glimpse of us in "the stupid ages" teaches them enough to learn from all the mistakes we keep repeating?

Man, this brings up a lot of privacy questions. Especially around who gets to look at what. I mean is this a personal archive that only unlocks after your death? Is it open to anyone? Is it available to law enforcement?

This about this:

Your entire sex life is replayable. This includes being with someone or solo action. Do you want you grandkids watching that?

No longer are we contemplating whether someone committed a crime, we would just watch the replay. Would you even commit a crime if you knew it was being recorded? You wouldn't need witnesses, just their feeds. Heck, you don't need forensic evidence really - if you never find a murder weapon who cares? It's on video! The innocent always have an air-tight alibi. It's on video!

Would not allowing law enforcement to look at your 'life feed' automatically make you a suspect?

Re: Cost of storing all human audio-visual experiences

#26

assuming somekind of google glass attached to a giant world wide data storage. Could it be compressed? (which got me to thinking about compression and the larger the size of the data stored the more likely the chance of larger and larger streams being duplicate. How many parts of the image do I have to store of my desk. Also, could we limit it to interesting moments(how many times do I have to record me eating cheeri…

Presumably you could also use lossy compression across users, especially for the boring parts. I imagine that, within the confines of my bowl, my milk and cheerios look pretty similar to yours. Do the actual differences matter, as a record of some particular breakfast experience? I suppose only if they float into some surprising configuration.

Dedupe for life experiences.

Re: Cost of storing all human audio-visual experiences

#27

Earlier quoted context omitted.

I wonder, with personal income on the decline this past decade, will advertisers spend less money on adverts or more on people who have more disposable income from mined data?

Doubt it. If firms could analyze and deconstruct the purchasing decisions and preferences from such 24/7 feeds, it will just allow them to more efficiently funnel money from the consumer to the top. The lower income market is intensely profitable, mostly due to the limitation of choice a disenfranchised person has. It's disconcerting that, literally knowing everything about you, puts advertisers in a position to accu…

>The lower income market is intensely profitable, mostly due to the limitation of choice a disenfranchised person has.

From what I've seen, this has actually started to attract foreign owned corporations in some industries like food (snacks in particular), I'm sure that's not the only industry.

>It does seem like the logical progression for Google and friends. What will be interesting is how such invasive monitoring will be pitched as innocuous. People seem to be a-okay with corporations mining their data, though.

Agreed. From someone trying to get into the data mining space, I feel like the "positive" direction to go in is to basically admitting that there is no putting the lid back onto this box (simply too valuable to many services and provides utility for people can't be botherd to learn more outside of how to press buttons) and trying to make data more valuable to people on an individual level. (instead of just for sigint agencies and their outsourced private equity owners portfolio companies like BAH).

Re: Cost of storing all human audio-visual experiences

#28

Earlier quoted context omitted.

>bzzzt and thanks for playing. There may be tens or hundreds or even thousands of years before that exponential curve looks like an s-curve, making his point more relevant that yours.

Well I've been pretty involved in storage for the last 10 years, 5 of them at Network Appliance and then later another 4 at Google dealing with their storage scale. And worked with both Seagate, Fujitsu, and to a lesser extent Hitachi as they have worked to increase the density of what can be reliably recalled. More interesting has been watching the struggle of thermal noise and the ability to push past areal densiti…

>I am curious to see if storage will have its 'multi-core' moment. That will be interesting if it happens.

Thank you for the insight and explanation.

I suppose I took issue with your above comment as I saw it "missing the point" of the author; if storage capability keeps growing (even +/- its current rate), it will be extremely cheap by modern economic standards to record...everything.

Re: Cost of storing all human audio-visual experiences

#29

I think a typical DVD is more like 4.5 Mbps...

That's true, but the OP said "Typical DVD-quality SD bitrate". DVDs use MPEG-2 encoding, but there are much better codecs available today, such as H.264, which can achieve equivalent quality at substantially lower bitrates.

Opinions vary on what the "best" bitrate for transcoding DVDs to H.264 is, and it depends on what your content is too. In a quick search I found people claiming as low as 500kbps(+128kpbs for audio)[1], which is half what the OP uses, all the way up to 2500kbps[2], double his figure. All of which leads me to believe that 1.25Mbps is a pretty reasonable napkin number.

[1]http://forums.macrumors.com/showthread.php?t=842536

[2]http://www.networkedmediatank.com/showthread.php?tid=6867

Re: Cost of storing all human audio-visual experiences

#30

Charlie Brooker did an episode of Black Mirror[0] based on the premises that everyone records every waking moment of their life. [0] http://en.wikipedia.org/wiki/Black_Mirror_(TV_series)

I wanted to do something like that but it didn't work out very well. Mostly "everything" meant on my computer (audio files are massive!) or phone and I couldn't wear a video camera at work or scan every piece of paper I came into contact with.

But still I find it interesting it's almost possible.

Post reply on HN