Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

I Accidentally Deleted 7TB of Videos Before Going to Production

blog.thevinter.com

Post reply on HN
Print view View original

111–120 of 362 posts

Re: I Accidentally Deleted 7TB of Videos Before Going to Production

#111

Post by Danieru »

tauwauwau wrote: post #88
dkersten wrote: post #15

Earlier quoted context omitted.

I was involved with archiving of data that was legally required to be retained for PSD2 compliance. So it was pretty important that the data was correctly archived, but it was just as important that it was properly removed from other places due to data protection. This is basically the approach that was taken: log before and after every action exactly what data or files is being acted on and how. Don't actually do it…

Once we get used to doing same thing multiple times a day, it doesn't matter if the log shows that we're about to take a destructive action, we'll still do it. Only thing that is foolproof is to not take the destructive action because people make mistake, it's human nature. I don't know how this can be implemented, may be encrypt the files, take a backup in some other location (which may not be allowed). Multiple rev…

That form had a couple weird checkboxes with odd wording. It is a famous mistake, but also rather understandable just because the form was cryptic.

Re: I Accidentally Deleted 7TB of Videos Before Going to Production

#112

Post by JadeNB »

dkersten wrote: post #15
dsego wrote: post #4

> but at the time the code seemed completely correct to me It always does. > Well, it teaches me to do more diverse tests when doing destructive operations. Or add some logging and do a dry run and check the results, literally simple prints statements: print("-----") print("Downloading videos ids from url: {url}") print(list of ids) ... ... ... # delete() dangerous action commented out until I'm sure it's right print…

I was involved with archiving of data that was legally required to be retained for PSD2 compliance. So it was pretty important that the data was correctly archived, but it was just as important that it was properly removed from other places due to data protection. This is basically the approach that was taken: log before and after every action exactly what data or files is being acted on and how. Don't actually do it…

> Then have multiple people inspect the logs.

I think that this is the most important part of any check. Your parent refers to checking the log five times, but, at least in my experience, I won't catch any more errors on the fifth time than the first—if I once saw what I expected rather than what was there, I'll keep doing so. Of course everyone has their blind spots, but, as in the famous Swiss-cheese approach, we just hope that they don't line up!

Re: I Accidentally Deleted 7TB of Videos Before Going to Production

#113

Post by irrational »

tauwauwau wrote: post #88
dkersten wrote: post #15

Earlier quoted context omitted.

I was involved with archiving of data that was legally required to be retained for PSD2 compliance. So it was pretty important that the data was correctly archived, but it was just as important that it was properly removed from other places due to data protection. This is basically the approach that was taken: log before and after every action exactly what data or files is being acted on and how. Don't actually do it…

Once we get used to doing same thing multiple times a day, it doesn't matter if the log shows that we're about to take a destructive action, we'll still do it. Only thing that is foolproof is to not take the destructive action because people make mistake, it's human nature. I don't know how this can be implemented, may be encrypt the files, take a backup in some other location (which may not be allowed). Multiple rev…

Because everyone assumes that everyone else is looking at it more closely than they are. “I’ll just do a cursory look since I’m sure everyone else is doing a in-depth look.” Narrator: nobody did an in-depth search.

Re: I Accidentally Deleted 7TB of Videos Before Going to Production

#114

Post by Reason077 »

> "What does this teach us? Well, it teaches me to do more diverse tests when doing destructive operations."

I think it also teaches us that adversity sometimes leads to better solutions. I love that the OP made a hacky script that did in 4 hours what a guy was paid to do manually over several months!

Re: I Accidentally Deleted 7TB of Videos Before Going to Production

#115

Post by honksillet »

chopin wrote: post #50
davbryn1 wrote: post #21

"What does this teach us? Well, it teaches me to do more diverse tests when doing destructive operations. It also should probably teach something to Vimeo and to my contractor but I doubt it will (and yes, the upload for some reason is still manual to this day. Go figure!)" So you wrote bad code, didn't test it properly, ran it on production on the Friday before a release and are blaming Vimeo and [name redacted]? An…

I'd hire this guy if only being for this frank about his mistake. He owned it and that is what I would look for. After deletion, what should he have done? Postpone the go-live? That's often not a a cost-effective option. As for a risk-analysis the worst what could happen was deletion of the remaining videos. I don't think that that makes big difference in this situation. And to do the right thing, you have to have th…

Agreed. But I’d also fire him from this job.

Re: I Accidentally Deleted 7TB of Videos Before Going to Production

#116

Post by HowardStark »

tauwauwau wrote: post #88
dkersten wrote: post #15

Earlier quoted context omitted.

I was involved with archiving of data that was legally required to be retained for PSD2 compliance. So it was pretty important that the data was correctly archived, but it was just as important that it was properly removed from other places due to data protection. This is basically the approach that was taken: log before and after every action exactly what data or files is being acted on and how. Don't actually do it…

Once we get used to doing same thing multiple times a day, it doesn't matter if the log shows that we're about to take a destructive action, we'll still do it. Only thing that is foolproof is to not take the destructive action because people make mistake, it's human nature. I don't know how this can be implemented, may be encrypt the files, take a backup in some other location (which may not be allowed). Multiple rev…

While this is a huge issue, a solution (well, a partial mitigation) I've seen and used is the "Pointing and Calling" technique. The basic idea is that you incorporate more actions beyond reading and typing or pressing a button—generally by having people point at something and say aloud what it is they're doing and what they expect to happen.

It's used rather extensively in safety-critical public transportation in Japan [1] and to a lesser extent in New York (along with many other countries) [2]. This can easily extend to software without overcomplicating by just setting the expectation that engineers, Q&A, etc. do this even when alone.

[1] https://www.atlasobscura.com/articles/pointing-and-calling-j...

[2] https://en.wikipedia.org/wiki/Pointing_and_calling

Re: I Accidentally Deleted 7TB of Videos Before Going to Production

#117

Post by JadeNB »

ineedasername wrote: post #103
cruano wrote: post #70

Earlier quoted context omitted.

That was our SOP for running DELETE SQL commands on production too, a script that generates a .sql that's run manually. It saved out asses a fair amount of times

Yeah, wish I'd learned that the easy way. Fresh into one of my first jobs I was working with a vendor's custom interface to merge/purge duplicate records. It didn't have a good method of record matching on inserts from the customer web interface so a large % of records had duplicates. Anyway, I selected what I though was a "merge all duplicates" option without previewing results. What I had actually done was "merge a…

> Yeah, wish I'd learned that the easy way.

I think that, if you've only learned something like that the easy way, then you haven't learned it yet. As long as everything's only ever gone right, it's easy to think, I'm in a rush this one time, and I've never really needed those safety procedures before, ….

Re: I Accidentally Deleted 7TB of Videos Before Going to Production

#118

Post by mkr-hn »

KingOfCoders wrote: post #95
thevinter wrote: post #90

Earlier quoted context omitted.

Just to clarify, my company is under an NDA and not personally me. It also encompasses only the actual project details so a post like this is legally compliant. (Not a lawyer, might be wrong)

So you're not under an NDA as you wrote. I don't know your position but I would assume a NDA is part of your freelancer or employee contract.

OP might at least want to consult with a contract lawyer in Italy to make sure.

Re: I Accidentally Deleted 7TB of Videos Before Going to Production

#119

Post by mikeryan »

einsty wrote: post #73
maweki wrote: post #69

Earlier quoted context omitted.

idempotency means that f(X) = f(f(X)). Modifying the X inbetween is not allowed. Is there really an initial environment where rm * ; rm * ; does something different than rm * once?

In the case of any live system, i would say yes. Additional, and different, files could have appeared on the file system in between the times of each rm *.

* is just short hand for a list of files. Calling rm with the same list of files will have the same results if you call it multiple times. That’s idempotent.

Your example is changing the list of files, or arguments to rm between runs. Same as pc85’s example where the timestamp argument changes.

Re: I Accidentally Deleted 7TB of Videos Before Going to Production

#120

Post by dboreham »

johnklos wrote: post #82

We can all poke at this person for doing things incorrectly, but one has to wonder what mindset could lead to any programmer ever thinking that: 1) parsing a web page shouldn't be considered incredibly fraught with problems 2) that reloading web pages should be part of (1) 3) that this should ever possibly be run without validating the list of files that would be deleted So forget the specifics. Where are people lear…

College? Parents? In my experience it runs pretty deep so not sure it can be easily trained out. This mindset is probably quite useful in evolutionary terms: rush at the attacking bear without thinking, for example.
Post reply on HN