Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

I Accidentally Deleted 7TB of Videos Before Going to Production

blog.thevinter.com

Post reply on HN
Print view View original

191–200 of 362 posts

Re: I Accidentally Deleted 7TB of Videos Before Going to Production

#191

Post by nso »

Moru wrote: post #187
FunnyLookinHat wrote: post #156

Earlier quoted context omitted.

I was actually really impressed with this individual! For someone who has less than a year of experience, they're showing quite a bit of initiative, drive, and curiosity - which really are what make or break engineers as they develop. Taking the time to do a blog post (effectively a post-mortem) and share it is even better! And yes - I've literally done this exact same error (with TB of video data!). Spending the fol…

My mistake was on floppy disc with source code, other text files and images. Was hand editing (in hex disc editor) the floppy to get back the data, sector by sector. Fun times. Not going back there though :-)

Mine was a DELETE FROM Users; WHERE... Fun was had.

Re: I Accidentally Deleted 7TB of Videos Before Going to Production

#193

Post by SparkyMcUnicorn »

honksillet wrote: post #115
chopin wrote: post #50

Earlier quoted context omitted.

I'd hire this guy if only being for this frank about his mistake. He owned it and that is what I would look for. After deletion, what should he have done? Postpone the go-live? That's often not a a cost-effective option. As for a risk-analysis the worst what could happen was deletion of the remaining videos. I don't think that that makes big difference in this situation. And to do the right thing, you have to have th…

Agreed. But I’d also fire him from this job.

"Recently, I was asked if I was going to fire an employee who made a mistake that cost the company $600,000. No, I replied, I just spent $600,000 training him. Why would I want somebody to hire his experience?"

-- Thomas J. Watson

Re: I Accidentally Deleted 7TB of Videos Before Going to Production

#194

Post by jasonlotito »

davbryn1 wrote: post #21

"What does this teach us? Well, it teaches me to do more diverse tests when doing destructive operations. It also should probably teach something to Vimeo and to my contractor but I doubt it will (and yes, the upload for some reason is still manual to this day. Go figure!)" So you wrote bad code, didn't test it properly, ran it on production on the Friday before a release and are blaming Vimeo and [name redacted]? An…

> This isn't a great article to have attached your name to

A million times better than your comment.

Re: I Accidentally Deleted 7TB of Videos Before Going to Production

#195

Post by samhw »

HowardStark wrote: post #116
tauwauwau wrote: post #88

Earlier quoted context omitted.

Once we get used to doing same thing multiple times a day, it doesn't matter if the log shows that we're about to take a destructive action, we'll still do it. Only thing that is foolproof is to not take the destructive action because people make mistake, it's human nature. I don't know how this can be implemented, may be encrypt the files, take a backup in some other location (which may not be allowed). Multiple rev…

While this is a huge issue, a solution (well, a partial mitigation) I've seen and used is the "Pointing and Calling" technique. The basic idea is that you incorporate more actions beyond reading and typing or pressing a button—generally by having people point at something and say aloud what it is they're doing and what they expect to happen. It's used rather extensively in safety-critical public transportation in Jap…

Hell, GitHub does that to an extent, with the "type the name of this repository to delete it" prompts. Typing the name of the repository isn't exactly perfect, but it's an interesting direction.

Re: I Accidentally Deleted 7TB of Videos Before Going to Production

#196

Post by tasuki »

FunnyLookinHat wrote: post #156
shantnutiwari wrote: post #146

What negativity and arrogance in the comments here. Jeez, it's like no one HN ever made a mistake, a bunch of 10xers ninja programmers here. Please read this: >I also want to preface this whole post by saying that I'm a Junior Developer with less than one year of actual experience. Some of the things that might seem obvious to some might not be so for me, thanks! It's just some kid sharing a mistake they made and own…

I was actually really impressed with this individual! For someone who has less than a year of experience, they're showing quite a bit of initiative, drive, and curiosity - which really are what make or break engineers as they develop. Taking the time to do a blog post (effectively a post-mortem) and share it is even better! And yes - I've literally done this exact same error (with TB of video data!). Spending the fol…

Wrt "less than one year of experience", looking at Nikita's CV and GitHub, despite the title, they aren't really a junior developer :)

Re: I Accidentally Deleted 7TB of Videos Before Going to Production

#197

Post by franciscop »

This is a great technical write up, I'd love to hear the human side of this story as well! When did you tell the higher ups that you deleted production? Was no one more senior on call to try to fix it? Did they want you to learn how to fix it? Or were you the most senior responsible for this whole area? Or did they don't know?

Re: I Accidentally Deleted 7TB of Videos Before Going to Production

#198

Post by codegeek »

nso wrote: post #191
Moru wrote: post #187

Earlier quoted context omitted.

My mistake was on floppy disc with source code, other text files and images. Was hand editing (in hex disc editor) the floppy to get back the data, sector by sector. Fun times. Not going back there though :-)

Mine was a DELETE FROM Users; WHERE... Fun was had.

Usually the recommendation is to not start writing the DELETE query first. Write the SELECT query first and see the results. If you miss the WHERE clause, you will see that immediately. Then change SELECT * to DELETE. But I assume you have learned that lesson already :)

Re: I Accidentally Deleted 7TB of Videos Before Going to Production

#199

Post by franciscop »

tasuki wrote: post #196
FunnyLookinHat wrote: post #156

Earlier quoted context omitted.

I was actually really impressed with this individual! For someone who has less than a year of experience, they're showing quite a bit of initiative, drive, and curiosity - which really are what make or break engineers as they develop. Taking the time to do a blog post (effectively a post-mortem) and share it is even better! And yes - I've literally done this exact same error (with TB of video data!). Spending the fol…

Wrt "less than one year of experience", looking at Nikita's CV and GitHub, despite the title, they aren't really a junior developer :)

True, he's been teaching programming since at least 2018, I was in a similar boat where I'd been programming for almost 5-7 years for fun and profit before my first official fulltime job.

Re: I Accidentally Deleted 7TB of Videos Before Going to Production

#200

Post by tomkwong »

First, I want to say that this is a great post. You always grow stronger when you make mistakes. Writing it up solidify understanding in the learning process.

This story resonates with many people here because many experienced engineers had done something similar before. For me, destructive batch operations like this would be two distinct steps:

1. Identify files that need to be deleted; 2. Loop through the list and delete them one by one.

These steps are decoupled so that the list can be validated. Each step can be tested independently. And the scripts are idempotent and can be reused.

Production operations are always risky. A good practice is to always prepare an execution plan with detailed steps, a validation plan, and a rollback plan. And, review the plan with peers before the operation.

Post reply on HN