Live data from Hacker News

I Accidentally Deleted 7TB of Videos Before Going to Production

blog.thevinter.com

251–260 of 362 posts

Re: I Accidentally Deleted 7TB of Videos Before Going to Production

#251
post #158

Earlier quoted context omitted.

What do you recommend, to not get intro trouble if there are spaces or newlines in the file names?

Don't use a shell script.

Do you mean, always pass the list directly to the next script via function calls, without writing it to an intermediate file / pipeline?

Re: I Accidentally Deleted 7TB of Videos Before Going to Production

#252
This is why you have backups. Good on you to have them!

When I just started as a junior dev at a small company I made the classic mistake of emptying the prod db instead of my local dev db. This was a small and in hindsight insignificant project. But Google was our customer, so it didn't feel insignificant at the time.

In this case my inexperience was partly my savior. All the data was inputted by people via a web form. Normally you're supposed to use POST to submit a form. But I was quite clueless at the time, so I had used GET. This meant all requests were still in the Apache logs. I could simply replay all requests.

I still feel my hard pounding when I think about the moment I realized what had happened. I was really relieved when everything was back!

What I learned from this incident:

- make automated backups

- no access to prod db from anywhere but prod

Re: I Accidentally Deleted 7TB of Videos Before Going to Production

#253
A great success story as far as I'm concerned, even if it doesn't reflect well on Vimeo support. But a good reminder to have someone doublecheck your logic if you aim to delete massive amounts of data from production. And to check if the backups are working (producing restorable data) on a regular basis. Sometimes they just seem to be working, as I have learned the hard way...

Re: I Accidentally Deleted 7TB of Videos Before Going to Production

#254

This is why you have backups. Good on you to have them! When I just started as a junior dev at a small company I made the classic mistake of emptying the prod db instead of my local dev db. This was a small and in hindsight insignificant project. But Google was our customer, so it didn't feel insignificant at the time. In this case my inexperience was partly my savior. All the data was inputted by people via a web fo…

Yea, I’ve wiped out an entire government’s form library once. Backups are a career saver.

Re: I Accidentally Deleted 7TB of Videos Before Going to Production

#256

Earlier quoted context omitted.

I was involved with archiving of data that was legally required to be retained for PSD2 compliance. So it was pretty important that the data was correctly archived, but it was just as important that it was properly removed from other places due to data protection. This is basically the approach that was taken: log before and after every action exactly what data or files is being acted on and how. Don't actually do it…

Once we get used to doing same thing multiple times a day, it doesn't matter if the log shows that we're about to take a destructive action, we'll still do it. Only thing that is foolproof is to not take the destructive action because people make mistake, it's human nature. I don't know how this can be implemented, may be encrypt the files, take a backup in some other location (which may not be allowed). Multiple rev…

> Multiple reviewers here didn't catch the mistake

Sure, but we can only do so much. I find its good bang for buck and alternatives that might prevent that are not always available, so we do the best we can. You gotta make a call on whether its enough or not.

Re: I Accidentally Deleted 7TB of Videos Before Going to Production

#257
The conclusion should include that backup at separate locations is key. Also, that the backups are tested and work. I worked with clients that had everything from lightning strikes destroying servers to ransomware to people making mistakes. No problem with solid backups. There is a difference between a good process and skill.

Re: I Accidentally Deleted 7TB of Videos Before Going to Production

#258

Aaaahhh, the feeling you get when you notice that you fucked up. Everything gets quiet, body motion stops, cheeks get hot, heart starts to beat and sinks really low, "fuck, fuck, fuck, fuck, fuck, fuck, fuck, fuck, fuck, fucking shit". Pause. Wait. Think. "Backups, what do I have, how hard will it be to recover? What is lost?". Later you get up and walk in circles, fingers rolling the beard, building the plan in the…

Poetic! Love it

Re: I Accidentally Deleted 7TB of Videos Before Going to Production

#260
post #99

Earlier quoted context omitted.

Yes, I find command line tools that have a "--dry-run" flag to be very helpful. If the tool (or script or whatever) is performing some destructive or expensive change, then having the ability to ask "what do you think I want to do?" is great. It's like the difference between "do what I say" and "do what I mean"...

That's what I like about powershell. Every script can include a "SupportsShouldProcess" [1] attribute. What this means is that you can pass two new arguments to you script, which have standardized names across the whole platform: - -WhatIf to see what would happen if you run the script; - -Confirm, which asks for confirmation before any potentially destructive action. Moreover these arguments get passed down to any c…

I’m a bit confused (because I didnt read the docs)… does calling it with “—whatif” exercise the same code path as calling without, only the “do destructive stuff” automagically doesn’t do anything? Or is it a separate routine that you have to write?

Cause if it is an entirely separate code path, doesn’t that introduce a case where what you say you’ll isn’t exactly what actually happens?

Post reply on HN