Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Data centers contain 90% crap data

gerrymcgovern.com

Post reply on HN
Print view View original

61–70 of 146 posts

Re: Data centers contain 90% crap data

#61

Post by philips »

2muchcoffeeman wrote: post #59
danpalmer wrote: post #11

> One organization I knew of had 1,500 terabytes of data, with less than 2% ever having been accessed after it was first stored. On a related note, probably a similar percentage of people claim on their car insurance. If only the rest realised they had "crap insurance" and were paying for nothing, they could save so much money! This is obviously sarcasm, but I think it's important to remember that much of the data is…

Photos of kids? Maybe that one will be The One that we end up framing? Look through the pictures with your kids (and with your kids) and pick out the best ones. Delete the rest.

The photo apps really don't have a good "sorting through the shoe box" UX. It would be great if I could in Google Photos, for example:

1. Bulk favorite photos in Google Photos for long term archiving

2. Set a retention policy of months/years for photos based on metadata.

3. Have a UX to quickly sort the week or months photos via a swipe left/right UX.

Re: Data centers contain 90% crap data

#63

Post by userbinator »

danpalmer wrote: post #11

> One organization I knew of had 1,500 terabytes of data, with less than 2% ever having been accessed after it was first stored. On a related note, probably a similar percentage of people claim on their car insurance. If only the rest realised they had "crap insurance" and were paying for nothing, they could save so much money! This is obviously sarcasm, but I think it's important to remember that much of the data is…

"Better to have and not need, than need and not have."

Storage is cheap, very cheap.

Re: Data centers contain 90% crap data

#64

Post by stephen_g »

2muchcoffeeman wrote: post #59
danpalmer wrote: post #11

> One organization I knew of had 1,500 terabytes of data, with less than 2% ever having been accessed after it was first stored. On a related note, probably a similar percentage of people claim on their car insurance. If only the rest realised they had "crap insurance" and were paying for nothing, they could save so much money! This is obviously sarcasm, but I think it's important to remember that much of the data is…

Photos of kids? Maybe that one will be The One that we end up framing? Look through the pictures with your kids (and with your kids) and pick out the best ones. Delete the rest.

I go back and look at old photos all the time (like when I think of a trip, and look back to it, and then just keep scrolling for a while). The pictures bring back more memories and that's to me a great joy - and on the other hand I find no value in deleting the photos that are not "the best" (I'm talking about good enough photos, I delete blurry, useless or duplicate ones immediately) - what would it gain? A few dollars less storage cost over the next decade? The satisfaction of digital cleaning?

Re: Data centers contain 90% crap data

#65

Post by prawn »

mrexroad wrote: post #22
danpalmer wrote: post #11

> One organization I knew of had 1,500 terabytes of data, with less than 2% ever having been accessed after it was first stored. On a related note, probably a similar percentage of people claim on their car insurance. If only the rest realised they had "crap insurance" and were paying for nothing, they could save so much money! This is obviously sarcasm, but I think it's important to remember that much of the data is…

I have ~2.5TB of photos in iCloud, via Apple Photos. Excluding the various sized previews, I doubt many originals have been accessed in quite a while. I also have about 1TB Lightroom library archived to a different service; representing countless hours of photo processing work spanning over a decade. Haven’t touched that one in years. Neither are crap. (Yes, both have other backups and, yes, I’ve probably forgotten t…

I have about the same in iCloud and then dozens of drives of video footage. Beyond the "keep it in case you need it" aspect, another angle is that it's often cheaper to just keep everything than spend all the time it would take to do a thorough cull. Video is much slower to review, but also chews up far more space, so maybe that balances out.

Re: Data centers contain 90% crap data

#66

Post by maigret »

cyberjerkXX wrote: post #41

I wonder what this guy thinks about Internet Archive?

This is the least of problems. The Wayback Machine is 100 petabytes. https://en.wikipedia.org/wiki/Wayback_Machine . "Every day, we create more than 350 million terabytes of data." https://www.rankred.com/largest-data-centers-in-the-world/ , so 3500x the Wayback Machine every day.

Re: Data centers contain 90% crap data

#67

Post by palata »

zanecodes wrote: post #55
AStonesThrow wrote: post #36

Earlier quoted context omitted.

I was just pondering this today, in terms of how much data and objects I create in a Google or Microsoft account, for example, and then they create a burden of cost and maintenance for me down the road. Especially deleting old emails and photos. That's arduous and sometimes poignant as I flush away my personal life story. Cloud services make difficult and sometimes even byzantine processes for deleting stuff, and it'…

I would love a feature like retention policies at the filesystem level. If it could somehow take data provenance into account, that would be even better, i.e. data that I've created directly is irreplaceable and should never be deleted unless I say so and should always be backed up (photographs, writing, project source code, art, game save files, my list of installed applications, etc.); data that I've created indire…

Not at the filesystem level, but Android does something that feels like that: an app can write files in its own space, and when you delete the app, it deletes its files.

Except when the app writes files outside of its own space, which is meant for stuff that should stay (like pictures).

Of course, some apps store the pictures in their private space, and you lose the pictures when you remove the app. And some apps write crap in the shared space. But that seems like a fundamental limitation (even if it was done at the filesystem): how do you make sure that the apps/software you use is doing it right?

Re: Data centers contain 90% crap data

#68

Post by bocytron »

It seems you're all missing the point here: it's not about storing useless data, it's about destroying the environment in the process. I understand you all want to keep everything, just in case, because it's cheap and you don't see the externalities. But there are externalities, and they are big.

Re: Data centers contain 90% crap data

#69

Post by hannob »

I have seen takes about the environmental footprint of unused data before, and I have severe doubts this is a relevant issue.

I mean, sure, there is some impact. Storage media has to be produced. But there's a reason storage is cheap, it's not a whole lot of resources going into it. And hard drives that are idle in some data center without being accessed don't consume a lot of electricity.

There are very real and concerning problems with the environmental impact of IT. But they are primarily found in other areas. Energy consumption is mostly a function of "how much you compute with data", not "how much data you have".

In other words: be concerned about so-called "AI", be concerned about Bitcoin. Don't worry about unused data too much.

Re: Data centers contain 90% crap data

#70

Post by palata »

HPsquared wrote: post #7
boznz wrote: post #2

Don't forget emails.. I have everything I ever sent or received, and I have it backed up. I expect 90% of my inbox is the jpg signature logo they attach to the bottom of my clients email rather than hyperlink.

It's probably deduplicated on the server though, so the millions and millions of messages with that logo likely share the same piece of disk space. Probably one reason why free providers don't tend to offer End-to-End encryption. It prevents deduplication (and otherwise compressing redundant information).

> Probably one reason why free providers don't tend to offer End-to-End encryption.

I would think that email providers tend to not offer E2EE just because it fundamentally isn't practical with email. Providers like Proton try to do it, but it works only if you talk to someone else on Proton (at which point you may as well do it on Signal, which has better encryption).

Post reply on HN