Hey, everyone, ease up. I have: 1) dropped a production database because I thought it was the test database. 2) screwed up a print job costing $100,000 in today’s money and had to do it again 3) crashed all of Facebook with a C++ bug. 4) crashed Facebook photo uploads, with a JavaScript bug, in my first month. 5) literally killed a startup’s cash flow and caused them to lose their merchant account because I over focused on the wrong bugs.
I Accidentally Deleted 7TB of Videos Before Going to Production
161–170 of 362 posts
Re: I Accidentally Deleted 7TB of Videos Before Going to Production
#162Aaaahhh, the feeling you get when you notice that you fucked up. Everything gets quiet, body motion stops, cheeks get hot, heart starts to beat and sinks really low, "fuck, fuck, fuck, fuck, fuck, fuck, fuck, fuck, fuck, fucking shit". Pause. Wait. Think. "Backups, what do I have, how hard will it be to recover? What is lost?". Later you get up and walk in circles, fingers rolling the beard, building the plan in the head. Coffee gets made.
Re: I Accidentally Deleted 7TB of Videos Before Going to Production
#163What is the f doing in url = f" https://api.ourservice.com/media?page{page}&step=100 ?
"f-strings", a (new) way to format strings.
Re: I Accidentally Deleted 7TB of Videos Before Going to Production
#164Earlier quoted context omitted.
I'd hire this guy if only being for this frank about his mistake. He owned it and that is what I would look for. After deletion, what should he have done? Postpone the go-live? That's often not a a cost-effective option. As for a risk-analysis the worst what could happen was deletion of the remaining videos. I don't think that that makes big difference in this situation. And to do the right thing, you have to have th…
Agreed. But I’d also fire him from this job.
[deleted]
Re: I Accidentally Deleted 7TB of Videos Before Going to Production
#165What negativity and arrogance in the comments here. Jeez, it's like no one HN ever made a mistake, a bunch of 10xers ninja programmers here. Please read this: >I also want to preface this whole post by saying that I'm a Junior Developer with less than one year of actual experience. Some of the things that might seem obvious to some might not be so for me, thanks! It's just some kid sharing a mistake they made and own…
I think it was a great post. Reveals a knack for clarity in explanations.
The mistake is simple enough and natural for a junior. If it were just one video or something, it would probably not even be noteworthy.
I think the developer learned from the incident too. So all good.
I do think Vimeo was irresponsible in the whine affair though.
Re: I Accidentally Deleted 7TB of Videos Before Going to Production
#166> but at the time the code seemed completely correct to me It always does. > Well, it teaches me to do more diverse tests when doing destructive operations. Or add some logging and do a dry run and check the results, literally simple prints statements: print("-----") print("Downloading videos ids from url: {url}") print(list of ids) ... ... ... # delete() dangerous action commented out until I'm sure it's right print…
Another technique that I've used with good success is to write a script that dumps out bash commands to delete files individually. I can visually inspect the file, analyze it with other tools, etc and then when I'm happy it's correct just "bash file_full_of_rms.sh" and be confident that it did the right thing.
Ah, I’m glad I’m not the only one who did this. It also means that you can fix things when they break halfway. Say you get an error when the script is processing entry 101 (perhaps it’s running files through ffmpeg). Just fix the error and delete the first 100 lines.
Re: I Accidentally Deleted 7TB of Videos Before Going to Production
#167> but at the time the code seemed completely correct to me It always does. > Well, it teaches me to do more diverse tests when doing destructive operations. Or add some logging and do a dry run and check the results, literally simple prints statements: print("-----") print("Downloading videos ids from url: {url}") print(list of ids) ... ... ... # delete() dangerous action commented out until I'm sure it's right print…
That is called experience.
Good decisions come from experience. Experience comes from making bad decisions.
Re: I Accidentally Deleted 7TB of Videos Before Going to Production
#168What negativity and arrogance in the comments here. Jeez, it's like no one HN ever made a mistake, a bunch of 10xers ninja programmers here. Please read this: >I also want to preface this whole post by saying that I'm a Junior Developer with less than one year of actual experience. Some of the things that might seem obvious to some might not be so for me, thanks! It's just some kid sharing a mistake they made and own…
there's an argument that the best people around are the people who have already (or almost) made some big mistakes. I have made a couple of huge ones - luckily I kept my job
When interviewing candidates I always enquire about their professional mistakes. Their reply often is the decider between hiring/rejecting.
I want to have colleagues who admit fault, be truthful about actions which lead to the issue, and learn from it. The learning includes organisations perhaps putting additional measures in place to prevent future issues.
One candidate told of a story how he was On-Call early in his career and was told situations happened so rarely, just to continue living life as normal.
Unfortunately for him, his pager went off at 02:00am while he was high as a kite on drugs - but felt he had to take action (mostly due to arrogance!).
He promptly deleted production data and things only got worse when he tried to rectify the situation.
Of course he was fired for his actions but ever since he's been stone cold sober when on-call.... just in case.
He learned a valuable lesson about professional responsibilities.
Re: I Accidentally Deleted 7TB of Videos Before Going to Production
#169> but at the time the code seemed completely correct to me
I venture this kind of (misplaced) over-confidence is not atypical of many junior developers. As someone with a few years under my belt, I don't care how sure I was of the code I wrote that deletes important data, I would have gone through the code over and over again, and at least ran a simulation (by maybe logging the generated delete urls for manual verification).
It's a rite of passage and we all went through something like this. It's how you learn and grow.
>It also should probably teach something to Vimeo
No. Even if Vimeo could have made things better, it's still your fault. You have to take responsibility for your business. At the end of the day, if this causes the closure of your company, Vimeo is still fine.
Re: I Accidentally Deleted 7TB of Videos Before Going to Production
#170What negativity and arrogance in the comments here. Jeez, it's like no one HN ever made a mistake, a bunch of 10xers ninja programmers here. Please read this: >I also want to preface this whole post by saying that I'm a Junior Developer with less than one year of actual experience. Some of the things that might seem obvious to some might not be so for me, thanks! It's just some kid sharing a mistake they made and own…
I have done a lot of such blunders myself. Accidentally deleted my unchecked code and had to re-write everything from memory.
I envy those who claim to do no mistakes at all.