Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

I Accidentally Deleted 7TB of Videos Before Going to Production

blog.thevinter.com

Post reply on HN
Print view View original

171–180 of 362 posts

Re: I Accidentally Deleted 7TB of Videos Before Going to Production

#171

Post by smokey_circles »

davbryn1 wrote: post #21

"What does this teach us? Well, it teaches me to do more diverse tests when doing destructive operations. It also should probably teach something to Vimeo and to my contractor but I doubt it will (and yes, the upload for some reason is still manual to this day. Go figure!)" So you wrote bad code, didn't test it properly, ran it on production on the Friday before a release and are blaming Vimeo and [name redacted]? An…

Oof, we wouldn't work well together. Very rarely is someone good enough to be this obnoxious.

Re: I Accidentally Deleted 7TB of Videos Before Going to Production

#172

Post by tomhallett »

RcouF1uZ4gsC wrote: post #36

This is one of those times that even if you don’t use a fully functional language, trying to make as much of your program logic pure functions would be helpful. It also makes it more testable. Instead of putting the delete call right in the loop, split it into four functions. function getAllVimeoVideos() function getAllDbVideos() function getVideosToDelete(vimeo_videos, db_videos) function deleteVideos(videos_to_dele…

This was going to be my exact recommendation. By “separating the concerns”, you make it easier on my pretty much every dimension: testing in unit tests, doing a dry run in production, ability to read the code (you and code reviews), and in some cases your code will be written in a more functional way reducing variable scoping issues.

Re: I Accidentally Deleted 7TB of Videos Before Going to Production

#173

Post by macspoofing »

muglug wrote: post #138

The root of this particular issue was Vimeo's failure to do this migration for their customers. Vimeo OTT has a codebase written in Rails, whereas the main PHP application is written in PHP. At the time Vimeo acquired Vimeo OTT's codebase, the Vimeo OTT codebase was small — around 10,000 lines of Ruby. Rewriting that codebase inside the Vimeo PHP application would have been a tough technical challenge for the all-Rub…

>The root of this particular issue was Vimeo's failure to do this migration for their customers.

Yes and No. At the end of the day, you as a business have to insulate yourself from your infrastructure provider.

Re: I Accidentally Deleted 7TB of Videos Before Going to Production

#174

Post by hansel_der »

maweki wrote: post #69
maccard wrote: post #61

Earlier quoted context omitted.

Deleting * may not be though. Your selection needs to be idempotent.

idempotency means that f(X) = f(f(X)). Modifying the X inbetween is not allowed. Is there really an initial environment where rm * ; rm * ; does something different than rm * once?

> Is there really an initial environment where rm * ; rm * ; does something different than rm * once?

if * expands to the rm binary itself, maybe.

Re: I Accidentally Deleted 7TB of Videos Before Going to Production

#175

Post by DoubleDerper »

comprev wrote: post #168
692 wrote: post #157

Earlier quoted context omitted.

there's an argument that the best people around are the people who have already (or almost) made some big mistakes. I have made a couple of huge ones - luckily I kept my job

When interviewing candidates I always enquire about their professional mistakes. Their reply often is the decider between hiring/rejecting. I want to have colleagues who admit fault, be truthful about actions which lead to the issue, and learn from it. The learning includes organisations perhaps putting additional measures in place to prevent future issues. One candidate told of a story how he was On-Call early in hi…

Don't fire for the mistake. Fire for the inability of someone to own it, cover it up, or point fingers at others.

Re: I Accidentally Deleted 7TB of Videos Before Going to Production

#176

Post by nspattak »

shantnutiwari wrote: post #146

What negativity and arrogance in the comments here. Jeez, it's like no one HN ever made a mistake, a bunch of 10xers ninja programmers here. Please read this: >I also want to preface this whole post by saying that I'm a Junior Developer with less than one year of actual experience. Some of the things that might seem obvious to some might not be so for me, thanks! It's just some kid sharing a mistake they made and own…

LOL!

I have multiple years of experience than this man and still I could *very* *too* *easily* make a 7Tb mistake (or likely more :P )

Re: I Accidentally Deleted 7TB of Videos Before Going to Production

#177

Post by grumple »

shantnutiwari wrote: post #146

What negativity and arrogance in the comments here. Jeez, it's like no one HN ever made a mistake, a bunch of 10xers ninja programmers here. Please read this: >I also want to preface this whole post by saying that I'm a Junior Developer with less than one year of actual experience. Some of the things that might seem obvious to some might not be so for me, thanks! It's just some kid sharing a mistake they made and own…

This sort of mistake happens all the time when you write in multiple languages. A key solution is code review, a standard practice which doesn't seem to have happened here (and certainly isn't the fault of a junior).

Re: I Accidentally Deleted 7TB of Videos Before Going to Production

#178

Post by boygobbo »

KrishnaShripad wrote: post #170
shantnutiwari wrote: post #146

What negativity and arrogance in the comments here. Jeez, it's like no one HN ever made a mistake, a bunch of 10xers ninja programmers here. Please read this: >I also want to preface this whole post by saying that I'm a Junior Developer with less than one year of actual experience. Some of the things that might seem obvious to some might not be so for me, thanks! It's just some kid sharing a mistake they made and own…

I have done a lot of such blunders myself. Accidentally deleted my unchecked code and had to re-write everything from memory. I envy those who claim to do no mistakes at all.

Don't envy them - they are deluding themselves.

Re: I Accidentally Deleted 7TB of Videos Before Going to Production

#179

Post by mbiondi »

dsego wrote: post #4

> but at the time the code seemed completely correct to me It always does. > Well, it teaches me to do more diverse tests when doing destructive operations. Or add some logging and do a dry run and check the results, literally simple prints statements: print("-----") print("Downloading videos ids from url: {url}") print(list of ids) ... ... ... # delete() dangerous action commented out until I'm sure it's right print…

Agreed, I've also been burned doing stupid things like this and always print out the commands and check them before actually doing the commit.

As they say, measure twice, cut once.

Don't feel bad, I think every professional in IT goes through something similar at one time or another.

Re: I Accidentally Deleted 7TB of Videos Before Going to Production

#180

Post by esquivalience »

davbryn1 wrote: post #104
chopin wrote: post #50

Earlier quoted context omitted.

I'd hire this guy if only being for this frank about his mistake. He owned it and that is what I would look for. After deletion, what should he have done? Postpone the go-live? That's often not a a cost-effective option. As for a risk-analysis the worst what could happen was deletion of the remaining videos. I don't think that that makes big difference in this situation. And to do the right thing, you have to have th…

Owning the mistake would be fine if he did that - he did'nt. He blamed the company he was contracting for. That's a big no from me

It's as if we read different articles. He literally writes that he made "A series of mistakes that could've probably been easily prevented."
Post reply on HN