Live data from Hacker News

I deleted data from production

blef.fr

61–68 of 68 posts

Re: I deleted data from production

#61
post #55

I have sympathy for the author and find a lot of the criticism a bit harsh. On my teams, I tend to be the person put in the role of doing DevOps-ish work because it's something I'm familiar with. I honestly find it incredibly stress inducing. It seems to me there is an asymmetry in the risk of work between team members that never gets captured by management. There are degrees of difference with security and data conc…

Thanks.

Re: I deleted data from production

#62
post #41

My database distributes data in real-time globally: http://root.rupy.se That way it's almost impossible to have anything catastrophic happen unless the whole planet goes boom!

That's an interesting project, however you might want to consider slightly altering the CSS for the homepage: http://rupy.se/

Currently the main font:

  font-size: 0.6em;
ends up being around 9.6px on Firefox with default settings in regards to font size, which makes it pretty much unreadable on a 1920x1080 monitor (21.5").

Example screenshot: https://imgur.com/IsPvHqE

Re: I deleted data from production

#63
It reminds me that back in July 2019. I was using the Windows file explorer to investigate an issue. When clicking on a folder to open it, my wrist slightly twitched while clicking on the folder, which led to the accidental move of a 650 GB operational data folder into a code repository folder.

For some reason the move into the repository folder was instantaneous but the opposite wasn't. All data import attempts errored during the 14 hours required for the data to be moved back to its original place. I emailed and voiced my apologies to the team present both in India and in France.

While I was not fired on the spot, it might have indirectly led to the disappearance of our devops (devoops?) team in Paris. Fortunately enough I guess I was the last team member to be sent away.

For my defense, our dev team had a intervention scope so big that they were essentially super-men once they have been granted access. One the rights granted to us was full write-access to all incoming data folders.

Re: I deleted data from production

#64
post #44
post #21

Earlier quoted context omitted.

In jest but personally I don't think you are a real programmer unless you have done something like this. It's kind of "welcome to the club" right of passage

Making mistakes like these in dev/test/sandbox/playground environments is one thing, but making them in prod over and over again is different. Yes, everyone should experience making catastrophic mistakes but please spare some effort to only do it in non-prod...

Many places don't have those separated environments and discipline

Re: I deleted data from production

#65
post #54
post #19

Earlier quoted context omitted.

Mistakes happen. People forget which environment they are on. people forget where statement in their DELETE query. People misremember their cwd prior to running rm. Sometimes it's lack of experience, sometimes lack of sleep, sometimes just shit luck. Very rarely it's lack of responsibility. And regarding your team members: if you don't trust them, don't rant about it on public forum, man, move them away from importan…

Hmmm, what you’ve described is lack of responsibility of recognising one’s own failings. There’s no excuse for that.

I do not see where I said that. I am sorry, but you're projecting. Safety rules are written in blood; production safety rules are written in critical incidents and cold sweat. To err is human.

Re: I deleted data from production

#66
post #51

There are three examples in the article: removing /usr (!!), running "hdfs dfs -rm /data" and "terraform destroy". Even with basic knowledge, these are clearly destructive operations - and to top it all, being run in Production. What strikes me as odd is that in all the cases, there is no change control. And by that I don't necessarily mean a full-on ServiceNow-type change process; I mean there was no "review". Even…

I never said it was in the same company.

Neither did I assume so - just pointing out this pattern :)

Re: I deleted data from production

#67
post #9

This is why he is Data Engineering Coach and not actually responsible for production systems now. (apparently) Everyone loves reading tech horror stories and peeling off the take-away lessons. My lesson would be, don't hire this guy and don't use him as a coach. He's careless! "But this kind of thing could happen to anyone" - sure, anyone who is irresponsible. Am I ranting? Let me continue. I work on a team now with…

[deleted]

Re: I deleted data from production

#68
post #9

This is why he is Data Engineering Coach and not actually responsible for production systems now. (apparently) Everyone loves reading tech horror stories and peeling off the take-away lessons. My lesson would be, don't hire this guy and don't use him as a coach. He's careless! "But this kind of thing could happen to anyone" - sure, anyone who is irresponsible. Am I ranting? Let me continue. I work on a team now with…

Oh. Well why don’t you use your frustration into writing a message to the careless team member and firing him from the team? If you are working in a non professional team it doesn’t mean that others don’t learn from theirs mistakes. Also, how did end up in the company with such low quality team members?
Post reply on HN