Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Data centers contain 90% crap data

gerrymcgovern.com

Post reply on HN
Print view View original

81–90 of 146 posts

Re: Data centers contain 90% crap data

#81

Post by jl6 »

AStonesThrow wrote: post #36
ivraatiems wrote: post #27

Earlier quoted context omitted.

This is exactly the problem. Consider an order processing system that processes a million orders a day, and retains them for ninety days. What percentage of those 89-day-old orders are actually needed at 89 days old? It could be quite low, maybe a couple thousand out of a million. But if those orders aren't there, shit hits the fan. PCI compliance audits fail. The ability for customers to reconcile their charges with…

I was just pondering this today, in terms of how much data and objects I create in a Google or Microsoft account, for example, and then they create a burden of cost and maintenance for me down the road. Especially deleting old emails and photos. That's arduous and sometimes poignant as I flush away my personal life story. Cloud services make difficult and sometimes even byzantine processes for deleting stuff, and it'…

Unfortunately a lot of data retention policies aren’t so mechanical, but are of the form “delete after 7 years unless there is a legal hold in force”, which is usually just rare enough of an edge case that orgs evaluate it manually and hence only do periodic manual purges. But probably the main reason auto-delete isn’t popular is because a process that can delete your old data is one bug/misconfig away from deleting your new data too.

Re: Data centers contain 90% crap data

#82

Post by Borg3 »

wvenable wrote: post #57

I hate searching for vintage computer stuff only to find broken hyperlinks to lost pages that once contained useful information or software but is now gone forever.

Yeah, because we do NOT mirror usefull information. Only crap :) I remember years ago (20 years, even 10 years ago) it was easy to mirror stuff. These days, web pages are loaded w/ JS junk.. Is hard, so I just extract interesting text and put it into my storage.

Re: Data centers contain 90% crap data

#83

Post by InsideOutSanta »

danpalmer wrote: post #11

> One organization I knew of had 1,500 terabytes of data, with less than 2% ever having been accessed after it was first stored. On a related note, probably a similar percentage of people claim on their car insurance. If only the rest realised they had "crap insurance" and were paying for nothing, they could save so much money! This is obviously sarcasm, but I think it's important to remember that much of the data is…

The article touches on this issue by pointing out that:

>The Cloud is what happens when the cost of storing data is less than the cost of figuring out what to do with the crap

But I think that's wrong. The actual issue is that you often can't figure out "what to do with the crap" because the difference between useful data and crap data is determined at the point in time when you need it, not when you store it.

I'm relatively careful with deleting data, but even so, there were countless instances where I thought something was no longer needed and deleted it, only to figure out that, a month later, I needed it after all.

Also, the article has a few anecdotes like this:

>Scottish Enterprise had 753 pages on its website, with 47 pages getting 80% of visits

But that is completely orthogonal to the question of what is "crap." The fact that some data is rarely needed does not mean it's crap. If anything, the fact that it is rarely needed might make it more valuable because it might make it less likely that the same data can be found elsewhere.

Re: Data centers contain 90% crap data

#84

Post by foobahify »

2muchcoffeeman wrote: post #59
danpalmer wrote: post #11

> One organization I knew of had 1,500 terabytes of data, with less than 2% ever having been accessed after it was first stored. On a related note, probably a similar percentage of people claim on their car insurance. If only the rest realised they had "crap insurance" and were paying for nothing, they could save so much money! This is obviously sarcasm, but I think it's important to remember that much of the data is…

Photos of kids? Maybe that one will be The One that we end up framing? Look through the pictures with your kids (and with your kids) and pick out the best ones. Delete the rest.

[flagged]

Re: Data centers contain 90% crap data

#85

Post by InsideOutSanta »

jongjong wrote: post #44
danpalmer wrote: post #11

> One organization I knew of had 1,500 terabytes of data, with less than 2% ever having been accessed after it was first stored. On a related note, probably a similar percentage of people claim on their car insurance. If only the rest realised they had "crap insurance" and were paying for nothing, they could save so much money! This is obviously sarcasm, but I think it's important to remember that much of the data is…

True, when it comes to data, there is a strong case to be a compulsive hoarder. It's cheap to store; often free and data is often marginally more useful and interesting than most generic mass-produced old junk which sits in garage gathering dust.

Technology for sorting through unordered piles of data also gets better over time. When I started storing digital photographs, face detection wasn't a thing, but it is now, so those tens of thousands of photos from the 90s suddenly became more useful to me.

Re: Data centers contain 90% crap data

#86

Post by foobahify »

otterley wrote: post #75

90% of libraries consist of books that are never opened. These books were all produced by destroying and processing trees, sometimes with toxic chemicals, and their information density is orders of magnitude lower than that of a hard disk or SSD. Same with photo processing, where 90% of photos taken are discarded, and the toxicity of the chemicals is even higher. So the question isn't simply whether storage is wasted…

Yes. Rather than compare to paper, compare to yesterday's HDDs. Do we need more HDDs today than then?

Re: Data centers contain 90% crap data

#88

Post by hoseja »

"We’re destroying our environment to (...)"

No we're not. I really dislike this "environmental" anti-technologist angle. A single steel plant in china has tenfold "environmental impact" than all photos stored on a platter everywhere.

Would you prefer the photos are a cocktail of weird chemicals on a negative and printed on glossy photo paper?

Digital data is the most ephemeral we are able to make it through vast effort.

Re: Data centers contain 90% crap data

#90

Post by ghaff »

InsideOutSanta wrote: post #83
danpalmer wrote: post #11

> One organization I knew of had 1,500 terabytes of data, with less than 2% ever having been accessed after it was first stored. On a related note, probably a similar percentage of people claim on their car insurance. If only the rest realised they had "crap insurance" and were paying for nothing, they could save so much money! This is obviously sarcasm, but I think it's important to remember that much of the data is…

The article touches on this issue by pointing out that: >The Cloud is what happens when the cost of storing data is less than the cost of figuring out what to do with the crap But I think that's wrong. The actual issue is that you often can't figure out "what to do with the crap" because the difference between useful data and crap data is determined at the point in time when you need it, not when you store it. I'm re…

I do cull photos and docs but it takes effort and you will make mistakes. On the other hand, intelligent culling makes it easier to find things.

It’s a trade off especially with digital where there’s not that much cost associated with holding onto things relative to the physical world. My whole house is basically in storage because of a kitchen fire and I’m planning to be pretty selective about what I move back in.

Post reply on HN