I Accidentally Deleted 7TB of Videos Before Going to Production

Post by oauea » Thu, May 05, 2022, 5:22 PM UTC

swid wrote: post #235

Blackcatmaxy wrote: post #227

Earlier quoted context omitted.

There was a thread recently about a repo that accidentally went private and lost all of its stars because of confusion with GH teams vs GH profile readme repo naming. I think this type of prompt is very useful for explicitly preventing the rare worst case scenarios but the problem is making any type of prompt "routine" so that our brains fail to process it.

The suggestion in that post about how to fix it is good, and mirrors one I read in the Rachael by the Bay blog - type the number of machines to continue: https://rachelbythebay.com/w/2020/10/26/num/ The take away by both is there is actually something to do which can wake people up when the stakes are high, and they might not be doing what they expect.

Post by [deleted] » Thu, May 05, 2022, 5:23 PM UTC

Post by hnlmorg » Thu, May 05, 2022, 5:27 PM UTC

nix23 wrote: post #154

hnlmorg wrote: post #20

Earlier quoted context omitted.

I love ZFS too but that's not really relevant to this discussion because the deleted items were on a video hosting platform and the company did already have local copies.

Yes and? Make a snapshot on live. Again, never touch data before snapshot.

Post by chernevik » Thu, May 05, 2022, 5:35 PM UTC

muglug wrote: post #138

The root of this particular issue was Vimeo's failure to do this migration for their customers. Vimeo OTT has a codebase written in Rails, whereas the main PHP application is written in PHP. At the time Vimeo acquired Vimeo OTT's codebase, the Vimeo OTT codebase was small — around 10,000 lines of Ruby. Rewriting that codebase inside the Vimeo PHP application would have been a tough technical challenge for the all-Rub…

Post by beardedetim » Thu, May 05, 2022, 5:40 PM UTC

deltarholamda wrote: post #263

qwertox wrote: post #162

Aaaahhh, the feeling you get when you notice that you fucked up. Everything gets quiet, body motion stops, cheeks get hot, heart starts to beat and sinks really low, "fuck, fuck, fuck, fuck, fuck, fuck, fuck, fuck, fuck, fucking shit". Pause. Wait. Think. "Backups, what do I have, how hard will it be to recover? What is lost?". Later you get up and walk in circles, fingers rolling the beard, building the plan in the…

Pffft, it's not a real panic until you weigh the pros and cons of leaving the country with nothing but the clothes on your back and becoming a illegal immigrant shepherd in a nation with too many consonants in its name. (Your description is so, so, spot on.)