I Accidentally Deleted 7TB of Videos Before Going to Production • Hacker News Board

Re: I Accidentally Deleted 7TB of Videos Before Going to Production

#181

Post by aeroplanetext » Thu, May 05, 2022, 2:05 PM UTC

KrishnaShripad wrote: post #170

shantnutiwari wrote: post #146

What negativity and arrogance in the comments here. Jeez, it's like no one HN ever made a mistake, a bunch of 10xers ninja programmers here. Please read this: >I also want to preface this whole post by saying that I'm a Junior Developer with less than one year of actual experience. Some of the things that might seem obvious to some might not be so for me, thanks! It's just some kid sharing a mistake they made and own…

I have done a lot of such blunders myself. Accidentally deleted my unchecked code and had to re-write everything from memory. I envy those who claim to do no mistakes at all.

I’ve been there! At least when you write it the second time it goes more quickly.

Re: I Accidentally Deleted 7TB of Videos Before Going to Production

#182

Post by ElCapitanMarkla » Thu, May 05, 2022, 2:06 PM UTC

Nice work :D I tend to always add a `--dryrun` flag to any scripts like this these days so that when we move it to production we can run an extra test there just to be sure.

Re: I Accidentally Deleted 7TB of Videos Before Going to Production

#183

Post by JasonFruit » Thu, May 05, 2022, 2:07 PM UTC

I believe if we're honest, we've all done stupid things we should have avoided. I remember a group of about 3000 emails that went out to insurance agents saying that policy #123456789 for Someone Funky was going to be cancelled by underwriting. I also remember very quickly figuring out how to automate Outlook's email recall feature.

We've all made big dumb mistakes. Recover and learn.

Re: I Accidentally Deleted 7TB of Videos Before Going to Production

#184

Post by nicbou » Thu, May 05, 2022, 2:07 PM UTC

shantnutiwari wrote: post #146

What negativity and arrogance in the comments here. Jeez, it's like no one HN ever made a mistake, a bunch of 10xers ninja programmers here. Please read this: >I also want to preface this whole post by saying that I'm a Junior Developer with less than one year of actual experience. Some of the things that might seem obvious to some might not be so for me, thanks! It's just some kid sharing a mistake they made and own…

More importantly, this person is helping us learn from their mistake. This is something that should be encouraged, not mocked.

Re: I Accidentally Deleted 7TB of Videos Before Going to Production

#185

Post by throwaway744678 » Thu, May 05, 2022, 2:08 PM UTC

dncornholio wrote: post #134

What is the f doing in url = f" https://api.ourservice.com/media?page{page}&step=100 ?

It's a Python f-string [0]. A way of formatting a string by directly including a Python expression between curly braces.

[0] https://docs.python.org/3/tutorial/inputoutput.html#tut-f-st...

Re: I Accidentally Deleted 7TB of Videos Before Going to Production

#186

Post by 0xbadcafebee » Thu, May 05, 2022, 2:08 PM UTC

This is more common than you think. Not just losing data, but not having a good handle on where the important parts of the system are, and how close you are to catastrophe. I find diagrams really help. I can recall a visual map of the system when I work on some component, and think, "OH, I remember seeing this component connected to a really critical thing, I need to check something first."

Start by creating one empty page for every component of your system. You won't remember them all, but over time you can add missing ones. Each page is the authoritative source of info on that component. If you need more pages for one component, put them in a directory of the same name as the page and add ".d" to the directory name, and link to them from the first page. Finally, create a diagram (however you want) that includes every component you have a page for. Add the count of components to the top of the diagram. If the count on the diagram doesn't match the number of documents, time to update the diagram. If you ever add, remove or rename a page, time to update the diagram. If you do this the same way for every different system you have, you can link them all together and get both small and large scale diagrams. (p.s. don't waste time automating this unless you find the system changing constantly or you have a very big system)

Re: I Accidentally Deleted 7TB of Videos Before Going to Production

#187

Post by Moru » Thu, May 05, 2022, 2:09 PM UTC

FunnyLookinHat wrote: post #156

shantnutiwari wrote: post #146

What negativity and arrogance in the comments here. Jeez, it's like no one HN ever made a mistake, a bunch of 10xers ninja programmers here. Please read this: >I also want to preface this whole post by saying that I'm a Junior Developer with less than one year of actual experience. Some of the things that might seem obvious to some might not be so for me, thanks! It's just some kid sharing a mistake they made and own…

I was actually really impressed with this individual! For someone who has less than a year of experience, they're showing quite a bit of initiative, drive, and curiosity - which really are what make or break engineers as they develop. Taking the time to do a blog post (effectively a post-mortem) and share it is even better! And yes - I've literally done this exact same error (with TB of video data!). Spending the fol…

My mistake was on floppy disc with source code, other text files and images. Was hand editing (in hex disc editor) the floppy to get back the data, sector by sector. Fun times. Not going back there though :-)

Re: I Accidentally Deleted 7TB of Videos Before Going to Production

#188

Post by comprev » Thu, May 05, 2022, 2:12 PM UTC

DoubleDerper wrote: post #175

comprev wrote: post #168

Earlier quoted context omitted.

When interviewing candidates I always enquire about their professional mistakes. Their reply often is the decider between hiring/rejecting. I want to have colleagues who admit fault, be truthful about actions which lead to the issue, and learn from it. The learning includes organisations perhaps putting additional measures in place to prevent future issues. One candidate told of a story how he was On-Call early in hi…

Don't fire for the mistake. Fire for the inability of someone to own it, cover it up, or point fingers at others.

His honesty of admitting to being off his nut while on-call led to his firing, not the action of deleting things.

Re: I Accidentally Deleted 7TB of Videos Before Going to Production

#189

Post by Closi » Thu, May 05, 2022, 2:13 PM UTC

ceejayoz wrote: post #26

dewey wrote: post #22

Earlier quoted context omitted.

Where do you see the clients name? I only see Vimeo being mentioned.

It has been edited. https://news.ycombinator.com/item?id=31271836

It still breaks the NDA:

* Firstly, you don't have to name the company to break the NDA anyway (you are still disclosing information you aren't supposed to disclose regardless of if it can be linked back to the company).

* Secondly, the client is still named on the front page of the website.

* Thirdly, OP posted this with his real name that trivially links back to the dev shop he is working for. The site also has his CV which lists the client again, with a description of the project to link it to the post.

* Finally, The client can trivially be identified by googling the description in the second paragraph (i.e. just search the named countries in operation plus the word Gym).

Re: I Accidentally Deleted 7TB of Videos Before Going to Production

#190

Post by conductr » Thu, May 05, 2022, 2:13 PM UTC

muglug wrote: post #138

The root of this particular issue was Vimeo's failure to do this migration for their customers. Vimeo OTT has a codebase written in Rails, whereas the main PHP application is written in PHP. At the time Vimeo acquired Vimeo OTT's codebase, the Vimeo OTT codebase was small — around 10,000 lines of Ruby. Rewriting that codebase inside the Vimeo PHP application would have been a tough technical challenge for the all-Rub…

He wasn’t asking them to refactor their internal code bases. But they should be able to whip up the 20 lines of code needed to do this between APIs (or just directly on their servers). Essentially what author was trying to do when he screwed up. For the author this was disposable code, for Vimeo this would have been a reusable utility.

I know how these things happen. Support ticket queues and all. And while I don’t fully know the difference in cost, I would assume a customer upgrading to an Enterprise plan would get a better support experience.

Whoever within authors company negotiated the upgrade to Enterprise (or didn’t) and failed to embed some agreement around OTT to Enterprise transition assistance was the one who made the first mistake.