Instagram and Facebook, at least, will compress a 4K video down to 360p if it doesn't get any views for a couple of months. So, there are some mechanisms to degrade the crap.
Data centers contain 90% crap data
121–130 of 146 posts
Re: Data centers contain 90% crap data
#122> One organization I knew of had 1,500 terabytes of data, with less than 2% ever having been accessed after it was first stored. On a related note, probably a similar percentage of people claim on their car insurance. If only the rest realised they had "crap insurance" and were paying for nothing, they could save so much money! This is obviously sarcasm, but I think it's important to remember that much of the data is…
> …I think it's important to remember that much of the data is stored because we don't know what we will need later.
This is the basic argument that every hoarder uses to justify their hoarding.
Re: Data centers contain 90% crap data
#123In this book 'World Wide Waste' he states 'Every time I download an email I
contribute to global warming.' Is this true? Aren't some data centers, Google for example carbon neutral?
Re: Data centers contain 90% crap data
#124One man's crap is another man's crap. My data, which may be 100% crap to you, is obviously not (all) crap to me. I often unerase USB drives, HDDs, and SD cards find pictures, movies, etc. Obviously, they don't have much value to me so lots of it is crap but sometimes there's gold hidden in the crap if you dig.
Re: Data centers contain 90% crap data
#125Make the cost of operating data centers reflect the damage to the environment and see how quickly people optimize. I don’t know how damaging storage is, but that’s the only way to make a difference.
so charge more? I'm sure they are already charging as much as the market will bear. Governments can impose a tax which they already have.
Re: Data centers contain 90% crap data
#126> One organization I knew of had 1,500 terabytes of data, with less than 2% ever having been accessed after it was first stored. On a related note, probably a similar percentage of people claim on their car insurance. If only the rest realised they had "crap insurance" and were paying for nothing, they could save so much money! This is obviously sarcasm, but I think it's important to remember that much of the data is…
This is exactly the problem. Consider an order processing system that processes a million orders a day, and retains them for ninety days. What percentage of those 89-day-old orders are actually needed at 89 days old? It could be quite low, maybe a couple thousand out of a million. But if those orders aren't there, shit hits the fan. PCI compliance audits fail. The ability for customers to reconcile their charges with…
[deleted]
Re: Data centers contain 90% crap data
#127Earlier quoted context omitted.
Sometimes I wish there was a feasible system to audit and reduce redundant photos in my iCloud. I have an embarassing ton of pictures that I never look at just like everyone else. I wouldn’t mind a tool that said “here’s 10^x photos that are the most similar to all your other photos would you like to put them in the trash?”
Thing is, the tool needs you to pick the one instance that survives the purge. Which means looking through 10^x of mostly similar photos to pick one. Removing duplicates isn't that complex already, and these tools exist so anyone can try it. It's just a truely grueling process.
Can you recommend any particular tools to do this?
Re: Data centers contain 90% crap data
#128> One organization I knew of had 1,500 terabytes of data, with less than 2% ever having been accessed after it was first stored. On a related note, probably a similar percentage of people claim on their car insurance. If only the rest realised they had "crap insurance" and were paying for nothing, they could save so much money! This is obviously sarcasm, but I think it's important to remember that much of the data is…
"Photos of kids? Maybe that one will be The One that we end up framing? Miscellaneous business records? Maybe those will be the ones we have to dig out for a tax audit? Web pages on government sites? Maybe there will suddenly be an interest in obscure pages on public health policy if a global pandemic happens."
Perhaps I am missing something, but these examples all sound like candidates for _offline_ storage, for which no third party custodian or data center is required.
The price of large capacity NVMe SSDs continues to fall.
The amount of energy, the resource requirements, not to mention the environmental and community impact, of running a car insurance company are miniscule in comparison to that of running a data center.
Re: Data centers contain 90% crap data
#129> One organization I knew of had 1,500 terabytes of data, with less than 2% ever having been accessed after it was first stored. On a related note, probably a similar percentage of people claim on their car insurance. If only the rest realised they had "crap insurance" and were paying for nothing, they could save so much money! This is obviously sarcasm, but I think it's important to remember that much of the data is…
"Photos of kids? Maybe that one will be The One that we end up framing? Miscellaneous business records? Maybe those will be the ones we have to dig out for a tax audit? Web pages on government sites? Maybe there will suddenly be an interest in obscure pages on public health policy if a global pandemic happens." Perhaps I am missing something, but these examples all sound like candidates for _offline_ storage, for whi…
Offline storage is only easy on the very small and the very large scale. On the very small scale you can get archival-grade CD/DVD/Blurays. On the very large scale you dedicate climate-controlled rooms to an automatic tape library.
In between is a vast gulf where your only good options are disks that you have to occasionally spin up to check integrity, have the disk trigger a data refresh if it's an SSD, and replace any disks that failed and rebuild their data from redundancy (raid or whatever you prefer). HDDs die even when off [1] and flash storage in SSDs is only rated to hold data for three years without power.
Sure, you can roll this yourself, but it can easily go wrong. Easier to either keep the disks spinning or pay someone with a tape archive (e.g. AWS deep glacier)
1 https://www.tomshardware.com/pc-components/storage/twenty-pe...
Re: Data centers contain 90% crap data
#130Thats why human brains forget