Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

I Accidentally Deleted 7TB of Videos Before Going to Production

blog.thevinter.com

Post reply on HN
Print view View original

81–90 of 362 posts

Re: I Accidentally Deleted 7TB of Videos Before Going to Production

#81

Post by malexbone »

chopin wrote: post #50
davbryn1 wrote: post #21

"What does this teach us? Well, it teaches me to do more diverse tests when doing destructive operations. It also should probably teach something to Vimeo and to my contractor but I doubt it will (and yes, the upload for some reason is still manual to this day. Go figure!)" So you wrote bad code, didn't test it properly, ran it on production on the Friday before a release and are blaming Vimeo and [name redacted]? An…

I'd hire this guy if only being for this frank about his mistake. He owned it and that is what I would look for. After deletion, what should he have done? Postpone the go-live? That's often not a a cost-effective option. As for a risk-analysis the worst what could happen was deletion of the remaining videos. I don't think that that makes big difference in this situation. And to do the right thing, you have to have th…

Agree 100%. Acknowledged mistake, moved forward to find a solution. Reflected on lessons learned. Shared valuable lesson.

To me this indicates intelligence, competence, integrity, grit and generosity. TechnicL proficiency is much easier to come by than integrity, grit and generosity. I would trust the author to deliver on commitments.

Re: I Accidentally Deleted 7TB of Videos Before Going to Production

#82

Post by johnklos »

We can all poke at this person for doing things incorrectly, but one has to wonder what mindset could lead to any programmer ever thinking that:

  1) parsing a web page shouldn't be considered incredibly fraught with problems
  2) that reloading web pages should be part of (1)
  3) that this should ever possibly be run without validating the list of files that would be deleted
So forget the specifics. Where are people learning these things, and what do we do to teach them better things?

Re: I Accidentally Deleted 7TB of Videos Before Going to Production

#84

Post by masswerk »

krit_dms wrote: post #58
masswerk wrote: post #19

Controversial opinion: And this is why block syntax by white space is not for production.

This is hardly a whitespace issue

Ah, yes, I just noticed the difference in indentation. In actuality, the error about the mental model of variable states.

Re: I Accidentally Deleted 7TB of Videos Before Going to Production

#85

Post by switch007 »

SnowHill9902 wrote: post #77
batch12 wrote: post #67

It's like the first time you run rm -rf /path/to/delete/ * And realize it is taking too long...

Can you explain? I feel like it removes / but not sure why.

The error is the space before the asterisk. The original intention was to delete the contents of the folder /path/to/delete/. Instead, the asterisk enumerates files in the current directory and they get deleted

Re: I Accidentally Deleted 7TB of Videos Before Going to Production

#86

Post by francis-io »

zrail wrote: post #66
dsego wrote: post #4

> but at the time the code seemed completely correct to me It always does. > Well, it teaches me to do more diverse tests when doing destructive operations. Or add some logging and do a dry run and check the results, literally simple prints statements: print("-----") print("Downloading videos ids from url: {url}") print(list of ids) ... ... ... # delete() dangerous action commented out until I'm sure it's right print…

Another technique that I've used with good success is to write a script that dumps out bash commands to delete files individually. I can visually inspect the file, analyze it with other tools, etc and then when I'm happy it's correct just "bash file_full_of_rms.sh" and be confident that it did the right thing.

This was taught to me in my first linux admin job.

I was running commands manually to interact with files and databases, but was quickly shown that even just writing all the commands out, one by one gives room personally review and get a peer review, and also helps with typos. I could ask a colleague "I'm about to run all these commands on the DB, do you see any problem with this?". It also reduces the blame if things go wrong if it managed to pass approval by two engineers.

While I'm thinking back, another little tip I was told was to always put a "#" in front of any command I paste into a terminal. This stops accidentally copying a carriage return and executing the command.

Re: I Accidentally Deleted 7TB of Videos Before Going to Production

#87

Post by KarlKode »

SnowHill9902 wrote: post #77
batch12 wrote: post #67

It's like the first time you run rm -rf /path/to/delete/ * And realize it is taking too long...

Can you explain? I feel like it removes / but not sure why.

Besides recursively deleting /path/to/delete/ the command also deletes all (non hidden) content of the current directory (note the * at the end of the line). I assume the correct command would be /path/to/delete/*.

Re: I Accidentally Deleted 7TB of Videos Before Going to Production

#88

Post by tauwauwau »

dkersten wrote: post #15
dsego wrote: post #4

> but at the time the code seemed completely correct to me It always does. > Well, it teaches me to do more diverse tests when doing destructive operations. Or add some logging and do a dry run and check the results, literally simple prints statements: print("-----") print("Downloading videos ids from url: {url}") print(list of ids) ... ... ... # delete() dangerous action commented out until I'm sure it's right print…

I was involved with archiving of data that was legally required to be retained for PSD2 compliance. So it was pretty important that the data was correctly archived, but it was just as important that it was properly removed from other places due to data protection. This is basically the approach that was taken: log before and after every action exactly what data or files is being acted on and how. Don't actually do it…

Once we get used to doing same thing multiple times a day, it doesn't matter if the log shows that we're about to take a destructive action, we'll still do it. Only thing that is foolproof is to not take the destructive action because people make mistake, it's human nature. I don't know how this can be implemented, may be encrypt the files, take a backup in some other location (which may not be allowed).

Multiple reviewers here didn't catch the mistake

https://www.bloombergquint.com/markets/citi-s-900-million-mi...

Re: I Accidentally Deleted 7TB of Videos Before Going to Production

#90

Post by thevinter »

KingOfCoders wrote: post #33
searchableguy wrote: post #24

Earlier quoted context omitted.

They said they are a junior developer with not much experience. I'm afraid they may not know what is and isn't covered under NDA.

My tip would be: read what you sign.

Just to clarify, my company is under an NDA and not personally me. It also encompasses only the actual project details so a post like this is legally compliant. (Not a lawyer, might be wrong)
Post reply on HN