Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

I Accidentally Deleted 7TB of Videos Before Going to Production

blog.thevinter.com

Post reply on HN
Print view View original

141–150 of 362 posts

Re: I Accidentally Deleted 7TB of Videos Before Going to Production

#141

Post by desarun »

Oh dude, we've all been there.

9 years ago I was working for a major broadcasting company in the arse end of London as a junior dev, building one of their Android apps.

We'd roll features out months before & enable them with feature flags via a json file we'd manually push to a prod server at a later date.

We'd just built a huge new feature letting you request content to be downloaded to your set top box remotely & it had a 250k marketing campaign to go along with the launch.

Senior dev trusted me with prod deployment rights.

I pushed the wrong json config to prod, launching the feature weeks before the marketing campaign.

Thank god I was a junior perm, that was definitely a firing offence.

Re: I Accidentally Deleted 7TB of Videos Before Going to Production

#142

Post by yohannparis »

honksillet wrote: post #115
chopin wrote: post #50

Earlier quoted context omitted.

I'd hire this guy if only being for this frank about his mistake. He owned it and that is what I would look for. After deletion, what should he have done? Postpone the go-live? That's often not a a cost-effective option. As for a risk-analysis the worst what could happen was deletion of the remaining videos. I don't think that that makes big difference in this situation. And to do the right thing, you have to have th…

Agreed. But I’d also fire him from this job.

Doesn't make sense. Their employer literally paid them to learn from their mistake.

Now, you think they should be fired? So that another employer rips the benefits of that learning experience.

Re: I Accidentally Deleted 7TB of Videos Before Going to Production

#143

Post by emerged »

HowardStark wrote: post #116
tauwauwau wrote: post #88

Earlier quoted context omitted.

Once we get used to doing same thing multiple times a day, it doesn't matter if the log shows that we're about to take a destructive action, we'll still do it. Only thing that is foolproof is to not take the destructive action because people make mistake, it's human nature. I don't know how this can be implemented, may be encrypt the files, take a backup in some other location (which may not be allowed). Multiple rev…

While this is a huge issue, a solution (well, a partial mitigation) I've seen and used is the "Pointing and Calling" technique. The basic idea is that you incorporate more actions beyond reading and typing or pressing a button—generally by having people point at something and say aloud what it is they're doing and what they expect to happen. It's used rather extensively in safety-critical public transportation in Jap…

“I’m removing that semicolon!” (Pointing)

Re: I Accidentally Deleted 7TB of Videos Before Going to Production

#144

Post by desarun »

davbryn1 wrote: post #131
thevinter wrote: post #122

Earlier quoted context omitted.

I'm sorry if it came off like that. The mistake in this case was completely mine (bad code and bad testing). The detour on the other two companies was mostly because this way of deleting/recovering stuff should've probably been avoided in the first place, other than that I'm absolutely not blaming anyone else!

Don't worry about all that - there isn't a developer worth their salt that hasn't made a mistake. But I'd consider having this blog post and HN post retracted purely for future internet checks. It isn't a reflection on you, and your honesty is fantastic. But there is a lot to be said about using a pseudonym when it comes this close to your employers

I'd probably make your github profile private for a while as well. Or at least removing your real name from it.

Re: I Accidentally Deleted 7TB of Videos Before Going to Production

#145

Post by yowlingcat »

thevinter wrote: post #18
legalcorrection wrote: post #3

[deleted]

You're right and I edited the company's name (might be too late but better this way). That said I'm not very happy with the experience of working for TheCompanyTM anyways so I'm in the process of switching jobs. Thanks for the comment :)

As sibling comments indicate, I would advise emailing HN mods to take this post down and remove it from your blog and post it on an anonymous one. Here are the problems you will face:

1) Your current blog has your current employer + client linked to it. 2) Your github has your real name. 3) All of these have been crawled/archived.

None of this bodes well for your career in the future. While I think your blog post is a great war story, it's really not a good idea to post it on your main account which can be traced back to your real name and CV because it will come up the next time you apply for a job.

Unfortunately, even if it illustrates a great deal of ingenuity and creativity on your part in fixing a mess you made, many folks will take one look at it and be judgmental. You have to manage your reputation online and be careful.

Re: I Accidentally Deleted 7TB of Videos Before Going to Production

#146

Post by shantnutiwari »

What negativity and arrogance in the comments here. Jeez, it's like no one HN ever made a mistake, a bunch of 10xers ninja programmers here. Please read this:

>I also want to preface this whole post by saying that I'm a Junior Developer with less than one year of actual experience. Some of the things that might seem obvious to some might not be so for me, thanks!

It's just some kid sharing a mistake they made and owning up. Ease up on the "LOL what an idiot" attitude

Re: I Accidentally Deleted 7TB of Videos Before Going to Production

#148

Post by qayxc »

johnklos wrote: post #82

We can all poke at this person for doing things incorrectly, but one has to wonder what mindset could lead to any programmer ever thinking that: 1) parsing a web page shouldn't be considered incredibly fraught with problems 2) that reloading web pages should be part of (1) 3) that this should ever possibly be run without validating the list of files that would be deleted So forget the specifics. Where are people lear…

> Where are people learning these things, and what do we do to teach them better things?

Learn to learn and learn to work carefully. It starts in school and should be part of a proper college/university education or vocational training.

There's several ways of learning the specifics: by experience on-the-job, which can be hard if mistakes can get you fired; or by putting in the work in your free time.

If your job is to work with certain web frameworks and you're not very experienced, either ask senior devs to assist/review before going live with critical changes. Alternatively, practice at home. Unpopular, but you need to get experience from somewhere. OSS projects are a great way to do that - be that by creating your own or by contributing to an existing one.

Re: I Accidentally Deleted 7TB of Videos Before Going to Production

#149

Post by hayd »

dsego wrote: post #4

> but at the time the code seemed completely correct to me It always does. > Well, it teaches me to do more diverse tests when doing destructive operations. Or add some logging and do a dry run and check the results, literally simple prints statements: print("-----") print("Downloading videos ids from url: {url}") print(list of ids) ... ... ... # delete() dangerous action commented out until I'm sure it's right print…

I'd make sure those include WARN or ERROR (I'd use logging to do that), that way you can grep for those. Spot checking might be difficult if the logs get long.

Re: I Accidentally Deleted 7TB of Videos Before Going to Production

#150

Post by thisNeeds2BeSad »

The only thing that I can remember helping against such actions, is the exponential need for confirmation by intent.

Means, if you delete one small file you need one confirmation, if you delete thousands, you need a intent stating i expect thousand files to be deleted. Same goes for size. So not a okay button, but instead a form allowing you to enter the dimension of the intented outcome. 100 files max, 1 gb max deleted.

If the request goves over the intent, the system aborts.

Post reply on HN