I Accidentally Deleted 7TB of Videos Before Going to Production • Hacker News Board

Re: I Accidentally Deleted 7TB of Videos Before Going to Production

#201

Post by oneepic » Thu, May 05, 2022, 2:28 PM UTC

davbryn1 wrote: post #21

"What does this teach us? Well, it teaches me to do more diverse tests when doing destructive operations. It also should probably teach something to Vimeo and to my contractor but I doubt it will (and yes, the upload for some reason is still manual to this day. Go figure!)" So you wrote bad code, didn't test it properly, ran it on production on the Friday before a release and are blaming Vimeo and [name redacted]? An…

Earlier in the article, the author does call out that it's bad code, so he's not entirely blaming these companies. Anyway: You should not be afraid of thinking about what each party could have done better. Not just yourself, but other people too. When I look back on times where I only blamed myself for prod issues, it was less of a learning experience, and more focused on beating myself up for no good reason. That approach shows that I'm afraid of the consequences, and it's an effective way to feel isolated from the team instead of improving.

Re: I Accidentally Deleted 7TB of Videos Before Going to Production

#202

Post by notyourday » Thu, May 05, 2022, 2:28 PM UTC

tomkwong wrote: post #200

First, I want to say that this is a great post. You always grow stronger when you make mistakes. Writing it up solidify understanding in the learning process. This story resonates with many people here because many experienced engineers had done something similar before. For me, destructive batch operations like this would be two distinct steps: 1. Identify files that need to be deleted; 2. Loop through the list and…

> 1. Identify files that need to be deleted; 2. Loop through the list and delete them one by one.

> These steps are decoupled so that the list can be validated. Each step can be tested independently. And the scripts are idempotent and can be reused.

This is the most underrated comment.

I'm saying it as someone who had the ultimate oversight of deleting hundreds of TBs per day spread of billions of files on different clouds and local storage.

Re: I Accidentally Deleted 7TB of Videos Before Going to Production

#203

Post by [deleted] » Thu, May 05, 2022, 2:28 PM UTC

shantnutiwari wrote: post #146

What negativity and arrogance in the comments here. Jeez, it's like no one HN ever made a mistake, a bunch of 10xers ninja programmers here. Please read this: >I also want to preface this whole post by saying that I'm a Junior Developer with less than one year of actual experience. Some of the things that might seem obvious to some might not be so for me, thanks! It's just some kid sharing a mistake they made and own…

[deleted]

Re: I Accidentally Deleted 7TB of Videos Before Going to Production

#204

Post by cntrl » Thu, May 05, 2022, 2:29 PM UTC

qwertox wrote: post #162

Aaaahhh, the feeling you get when you notice that you fucked up. Everything gets quiet, body motion stops, cheeks get hot, heart starts to beat and sinks really low, "fuck, fuck, fuck, fuck, fuck, fuck, fuck, fuck, fuck, fucking shit". Pause. Wait. Think. "Backups, what do I have, how hard will it be to recover? What is lost?". Later you get up and walk in circles, fingers rolling the beard, building the plan in the…

damn, your description is spot on and reading this triggered PTSD in me... Last time I had this feeling was two years ago when I destroyed one of our development servers because of a failed application update. I know exactly how I wished Ctrl + Z to exist in real life... We had backups of the machine, but it was still kind of a humiliating feeling to tell everybody and ask for restore from backup (everybody was cool though in the end)

Re: I Accidentally Deleted 7TB of Videos Before Going to Production

#205

Post by notyourday » Thu, May 05, 2022, 2:30 PM UTC

macspoofing wrote: post #173

muglug wrote: post #138

The root of this particular issue was Vimeo's failure to do this migration for their customers. Vimeo OTT has a codebase written in Rails, whereas the main PHP application is written in PHP. At the time Vimeo acquired Vimeo OTT's codebase, the Vimeo OTT codebase was small — around 10,000 lines of Ruby. Rewriting that codebase inside the Vimeo PHP application would have been a tough technical challenge for the all-Rub…

>The root of this particular issue was Vimeo's failure to do this migration for their customers. Yes and No. At the end of the day, you as a business have to insulate yourself from your infrastructure provider.

Vimeo is the only infrastructure provider providing that service. It is impossible to insulate a business from it.

Re: I Accidentally Deleted 7TB of Videos Before Going to Production

#206

Post by Forgeties79 » Thu, May 05, 2022, 2:30 PM UTC

comprev wrote: post #188

DoubleDerper wrote: post #175

Earlier quoted context omitted.

Don't fire for the mistake. Fire for the inability of someone to own it, cover it up, or point fingers at others.

His honesty of admitting to being off his nut while on-call led to his firing, not the action of deleting things.

>His honesty of admitting to being off his nut

This now my favorite euphemism for being high

Re: I Accidentally Deleted 7TB of Videos Before Going to Production

#207

Post by mh- » Thu, May 05, 2022, 2:31 PM UTC

afiori wrote: post #106

koolba wrote: post #97

Earlier quoted context omitted.

> This stops accidentally copying a carriage return and executing the command. For a one-liner sure, but a multi line command can still be catastrophic. Showing the contents of the clipboard in the terminal itself (eg via xclip) or opening an editor and saving the contents to a file are usually better approaches. The latter let’s you craft the entire command in the editor and then run it as a script.

From [0]: [For Bash] Ctrl + x + Ctrl + e : launch editor defined by $EDITOR to input your command. Useful for multi-line commands. I have tested this on windows with a MINGW64 bash, it works similarly to how `git commit` works; by creating a new temporary file and detecting* when you close the editor. [0] https://github.com/onceupon/Bash-Oneliner * Actually I have no idea how this works; does bash wait for the child…

It does create and give a temporary file path to the editor, but then simply waits for the process to exit with a healthy status.

Once that happens, it reads from the temporary file that it created.

Re: I Accidentally Deleted 7TB of Videos Before Going to Production

#208

Post by orange_puff » Thu, May 05, 2022, 2:32 PM UTC

As everyone else has already pointed out, better testing would have been very useful here. For instance, print(len(our_ids)) would have been a dead giveaway that that something was up

I am also a junior dev and completely empathize with being given a lot of responsibility and potentially messing up. I think for someone with < 1 year of experience, to solve the problems you created as fast as you did is really impressive. Thankfully your story ends well :)

Re: I Accidentally Deleted 7TB of Videos Before Going to Production

#209

Post by hbn » Thu, May 05, 2022, 2:33 PM UTC

aristus wrote: post #161

Hey, everyone, ease up. I have: 1) dropped a production database because I thought it was the test database. 2) screwed up a print job costing $100,000 in today’s money and had to do it again 3) crashed all of Facebook with a C++ bug. 4) crashed Facebook photo uploads, with a JavaScript bug, in my first month. 5) literally killed a startup’s cash flow and caused them to lose their merchant account because I over focu…

At my first development job (paid internship at a moderately-sized, though fast-growing business - maybe 300 people at the time?) I introduced a bug that didn't appear until a certain microservice stopped working (my code defaulted in the wrong direction when the ms failed) and as far as I can tell they may have lost or almost lost a pretty big account from it. In an after-hours meeting regarding the issue, one of the higher ups ended up storming out and never showing up again.

In my defence, we had to get 2 PR approvals before anything was merged! But I definitely learned a thing or two from that experience

Re: I Accidentally Deleted 7TB of Videos Before Going to Production

#210

Post by aasasd » Thu, May 05, 2022, 2:33 PM UTC

After having read about plenty of such cases over the years, I have a persistent dread of pulling something like that myself, to the point of being nervous with ‘*’ in the terminal, and generally checking everything twice. (And also have some kind of mild horror-high from corporate snafu stories, weirdly reminiscent of Ballard's ‘Crash’).

So: I never feed the data straight from the gathering script into the modifying script, at least not in the first runs. Instead, I dump the whole list of items into a file, count them in there, gawk at them to see that they're right, and compare with the source data by hand until I begin to annoy myself. Then I feed that file to the second script.