[dead]
Data centers contain 90% crap data
71–80 of 146 posts
Re: Data centers contain 90% crap data
#72Don't forget emails.. I have everything I ever sent or received, and I have it backed up. I expect 90% of my inbox is the jpg signature logo they attach to the bottom of my clients email rather than hyperlink.
I ended up working on some software and I was deemed the email guy (it's a very small % of my job but it is the biggest pain). "I need an email when this happens.. and when this happens." The requests are endless and I'm convinced there are people who if they could would do their entire job from their inbox and get everything and anything an application can do via email. The insidious problem is that it never solves…
I can see how that would work (email centric workflow), not unlike how some people now try to have a chat / Slack centric workflow.
Re: Data centers contain 90% crap data
#73> One organization I knew of had 1,500 terabytes of data, with less than 2% ever having been accessed after it was first stored. On a related note, probably a similar percentage of people claim on their car insurance. If only the rest realised they had "crap insurance" and were paying for nothing, they could save so much money! This is obviously sarcasm, but I think it's important to remember that much of the data is…
I think you miss the point of the article, which precisely says that we store a ton of data we don't need to store in the first place.
Photos of kids are obviously not part of that "crap" (even if we have too many of them and it would be worth triaging for our own sake).
The question is: what makes for most of that data? Is that all business records, or is that storing stuff because we can? I've worked in multiple startups, all were tracking users as much as they could. Adding tools that collect data is easy, and storing that data is cheap. "We may need it later". Never needed any of that crap, and it was invading our users' privacy.
In any case, I think the article raises a good point: it's so cheap to store crap that we don't even think about it. And it's bad for the environment. Just like it's so cheap to take a plane that we don't think about it, even when taking the train takes almost the same time.
Re: Data centers contain 90% crap data
#74> the cost of storing data is less than the cost of figuring out what to do with the crap.
Or the cost of figuring out that it's not worth saving...
Re: Data centers contain 90% crap data
#7590% of libraries consist of books that are never opened. These books were all produced by destroying and processing trees, sometimes with toxic chemicals, and their information density is orders of magnitude lower than that of a hard disk or SSD. Same with photo processing, where 90% of photos taken are discarded, and the toxicity of the chemicals is even higher.
So the question isn't simply whether storage is wasted; it's how much waste there is relative to the environmental impact. Granted, books and photographs don't need to be continuously fed energy to make the information available. However, the cost of storage is now so cheap that even with 90% waste, it's economically viable to keep it online. So the problem, if you can call it one, is that energy is too cheap, and externalities are not accounted for in the cost.
Re: Data centers contain 90% crap data
#76It seems you're all missing the point here: it's not about storing useless data, it's about destroying the environment in the process. I understand you all want to keep everything, just in case, because it's cheap and you don't see the externalities. But there are externalities, and they are big.
In which case we should be reading an article about how important it is to correctly price externalities. And that is not this article.
Re: Data centers contain 90% crap data
#77Earlier quoted context omitted.
I have ~2.5TB of photos in iCloud, via Apple Photos. Excluding the various sized previews, I doubt many originals have been accessed in quite a while. I also have about 1TB Lightroom library archived to a different service; representing countless hours of photo processing work spanning over a decade. Haven’t touched that one in years. Neither are crap. (Yes, both have other backups and, yes, I’ve probably forgotten t…
Sometimes I wish there was a feasible system to audit and reduce redundant photos in my iCloud. I have an embarassing ton of pictures that I never look at just like everyone else. I wouldn’t mind a tool that said “here’s 10^x photos that are the most similar to all your other photos would you like to put them in the trash?”
If you’re using Apple Photos, the feature is there. It will detect both exact copies as well as (nearly) identical visual duplicates. Look under utilities.
Re: Data centers contain 90% crap data
#78About a decade and a half ago I worked on a large data migration project at a FAANG. Multi-exabyte scale, many clusters across many countries. Once everyone was moved the old storage platform wasn't completely empty, because the number of migrations was large and users were (naturally) more focused on ensuring their data was in place and available on the target platform rather than ensuring every last thing was delet…
That does sound very interesting. Any insights on what would you do differently if you had to do it again? Any way to accelerate things now that you know the pain or do you think it's quite unavoidable and "legal Time"?
Re: Data centers contain 90% crap data
#79It seems you're all missing the point here: it's not about storing useless data, it's about destroying the environment in the process. I understand you all want to keep everything, just in case, because it's cheap and you don't see the externalities. But there are externalities, and they are big.
This.
We need to think about the data we need to store before we store it, only store the data that we need to store, and only store it for as long as needed.
It reminds me of CIs. It's now so easy to throw 40 jobs on GitHub actions that people don't think about them. I have been in a startup where people would debug in CI: they wouldn't have e.g. Windows on their machine (maybe they should have, given that their product was supposed to run there) and were fixing compilation issues by sending patches and patches to the CI. Every single time it would trigger the 40 jobs. Sometimes you could see a patch sent every 5 min for 3 days (where reproducing the issue locally would actually take 3s and not 5min). They did not even bother disabling the 39 uninteresting jobs.
For open source projects, it's just wasted energy, for private repos it was costing the company a lot. This was just malpractice. But nobody cared. The finance person would say "GitHub is expensive", the CEO that "well we need it" and the engineers that "I don't want that Windows crap on my computer", I suppose.
Re: Data centers contain 90% crap data
#80It seems you're all missing the point here: it's not about storing useless data, it's about destroying the environment in the process. I understand you all want to keep everything, just in case, because it's cheap and you don't see the externalities. But there are externalities, and they are big.
In which case we should be reading an article about how important it is to correctly price externalities. And that is not this article.
Well that article clearly says "it's hurting the environment, and in my experience the vast majority of that data is useless".
Which I believe is not uninteresting, given the amount of answers here where people say "Is that data useless? I don't know, I could imagine that it's not, I think it's a hard problem". Well here we have one person saying "I have experience in that, and I can tell you that most of it is useless". Just a data point, but that's still interesting.