Happened to Pixar Toy Story 2 too.
https://thenextweb.com/news/how-pixars-toy-story-2-was-delet...
I Accidentally Deleted 7TB of Videos Before Going to Production
331–340 of 362 posts
Re: I Accidentally Deleted 7TB of Videos Before Going to Production
#332> but at the time the code seemed completely correct to me It always does. > Well, it teaches me to do more diverse tests when doing destructive operations. Or add some logging and do a dry run and check the results, literally simple prints statements: print("-----") print("Downloading videos ids from url: {url}") print(list of ids) ... ... ... # delete() dangerous action commented out until I'm sure it's right print…
100% on the logging and dry run.
Re: I Accidentally Deleted 7TB of Videos Before Going to Production
#333Earlier quoted context omitted.
I had this experience when, years ago on my first day as group lead at $JOB, I was being shown a RAID 5 production server that held years of valuable, irreplaceable data (because there were no backups. Let me repeat that there were no backups). For some bizarre reason, I thought "oh cool, hot-swappable drives" and pulled one out of the rack. This naturally resulted in loud, persistent beeping from the machine, which…
Isn’t RAID 5 supposed to survive a single disk being taken out?
To add to the comments of cascading failure: if a drive goes bad, another drive from the same manufacturing batch is disproportionately likely to go bad. RAID arrays are often built with drives from the same batch, since they were bought at the same time from the same vendor. This means array failures include multiple drives more often than you'd expect.
Re: I Accidentally Deleted 7TB of Videos Before Going to Production
#334First, I want to say that this is a great post. You always grow stronger when you make mistakes. Writing it up solidify understanding in the learning process. This story resonates with many people here because many experienced engineers had done something similar before. For me, destructive batch operations like this would be two distinct steps: 1. Identify files that need to be deleted; 2. Loop through the list and…
I've never regretted treating tasks like this as a pipeline of discrete steps with explicit outputs and inputs. Sending output to a file, viewing it, then having something process the file is such a great safety net.
Re: I Accidentally Deleted 7TB of Videos Before Going to Production
#335Earlier quoted context omitted.
And most importantly, don't let yourself get into the habit of copy pasting the value
I wonder if your could print some non visible characters in there to taint the copied value in some detectable way.
yeah, that would possibly stop the copy and paste problem. to make it robust they would need to use a string of a few non-visible characters but that would fail if the browser's clipboard system doesn't copy them over for some kind of privacy initiative. might be another way it fails that I can't think of right now.
Re: I Accidentally Deleted 7TB of Videos Before Going to Production
#336Aaaahhh, the feeling you get when you notice that you fucked up. Everything gets quiet, body motion stops, cheeks get hot, heart starts to beat and sinks really low, "fuck, fuck, fuck, fuck, fuck, fuck, fuck, fuck, fuck, fucking shit". Pause. Wait. Think. "Backups, what do I have, how hard will it be to recover? What is lost?". Later you get up and walk in circles, fingers rolling the beard, building the plan in the…
If it is for real, body motion does not exactly stop, it manifests itself in other ways.
Re: I Accidentally Deleted 7TB of Videos Before Going to Production
#337Aaaahhh, the feeling you get when you notice that you fucked up. Everything gets quiet, body motion stops, cheeks get hot, heart starts to beat and sinks really low, "fuck, fuck, fuck, fuck, fuck, fuck, fuck, fuck, fuck, fucking shit". Pause. Wait. Think. "Backups, what do I have, how hard will it be to recover? What is lost?". Later you get up and walk in circles, fingers rolling the beard, building the plan in the…
Don't forget that out-of-body experience where you just kinda float outside yourself.
Re: I Accidentally Deleted 7TB of Videos Before Going to Production
#338Junior Dev: "I'm under an NDA" Also Junior Dev: "Here's my source code"
Unless you're Oracle that code is hardly critical to the business.
Even as a Sr Dev I'd share stuff like that, it's code that'd appear on a stack overflow post anyway.
Re: I Accidentally Deleted 7TB of Videos Before Going to Production
#339Earlier quoted context omitted.
While this is a huge issue, a solution (well, a partial mitigation) I've seen and used is the "Pointing and Calling" technique. The basic idea is that you incorporate more actions beyond reading and typing or pressing a button—generally by having people point at something and say aloud what it is they're doing and what they expect to happen. It's used rather extensively in safety-critical public transportation in Jap…
Hell, GitHub does that to an extent, with the "type the name of this repository to delete it" prompts. Typing the name of the repository isn't exactly perfect, but it's an interesting direction.
Azure has the same when deleting a database just a verify this is the correct one by typing the db name
Re: I Accidentally Deleted 7TB of Videos Before Going to Production
#340Earlier quoted context omitted.
mv then rm is another idiom. So long as you have the space. For database entries, flag for deletion, then delete. In the files case, the move or rename also accomplishes the result of breaking any functionality which still relies on those file ... whilst you can still recover. Way back in the day I was doing filesystem surgery on a Linux system, shuffling partitions around. I meant to issue the 'rf -rm .' in a specif…
You mean cp then rm? And yes, copy, verify, delete. And make sure by the code structure that you either do the three on the same files, or their fail. Also, do it slowly, with just a bit of data on each iteration. That will make the verification step more reliable. Anyway, for a huge majority of cases, only having backups is enough already. Just make sure to test them.
No, mv.
Example:
cd datadir
mkdir delete
mv ./delete
# test to see if anything looks broken.
# This might take a few seconds, or months, though it's usually reasonably brief.
rm -rf ./delete
- It's atomic (on a single filesystem). There's no risk of ending up with a partial operation or an incomplete operation.
- It doesn't copy the data, it renames the file. (mv and rename are largely synonyms.)
- There's no duplication of space usage. Where you're dealing with large files, this is helpful.
The process is similar to the staged deletion most desktop OS users are familiar with, of "drag to trash, then empty trash". Used in the manner I'm deploying it, it's a bit more like a staged warehouse purge or ordering a dumpster bin --- more structured / controlled staged deletion than a household or small office might use.