I have sympathy for the author and find a lot of the criticism a bit harsh. On my teams, I tend to be the person put in the role of doing DevOps-ish work because it's something I'm familiar with. I honestly find it incredibly stress inducing. It seems to me there is an asymmetry in the risk of work between team members that never gets captured by management. There are degrees of difference with security and data conc…
I deleted data from production
61–68 of 68 posts
Re: I deleted data from production
#62My database distributes data in real-time globally: http://root.rupy.se That way it's almost impossible to have anything catastrophic happen unless the whole planet goes boom!
Currently the main font:
font-size: 0.6em;
ends up being around 9.6px on Firefox with default settings in regards to font size, which makes it pretty much unreadable on a 1920x1080 monitor (21.5").Example screenshot: https://imgur.com/IsPvHqE
Re: I deleted data from production
#63For some reason the move into the repository folder was instantaneous but the opposite wasn't. All data import attempts errored during the 14 hours required for the data to be moved back to its original place. I emailed and voiced my apologies to the team present both in India and in France.
While I was not fired on the spot, it might have indirectly led to the disappearance of our devops (devoops?) team in Paris. Fortunately enough I guess I was the last team member to be sent away.
For my defense, our dev team had a intervention scope so big that they were essentially super-men once they have been granted access. One the rights granted to us was full write-access to all incoming data folders.
Re: I deleted data from production
#64Earlier quoted context omitted.
In jest but personally I don't think you are a real programmer unless you have done something like this. It's kind of "welcome to the club" right of passage
Making mistakes like these in dev/test/sandbox/playground environments is one thing, but making them in prod over and over again is different. Yes, everyone should experience making catastrophic mistakes but please spare some effort to only do it in non-prod...
Re: I deleted data from production
#65Earlier quoted context omitted.
Mistakes happen. People forget which environment they are on. people forget where statement in their DELETE query. People misremember their cwd prior to running rm. Sometimes it's lack of experience, sometimes lack of sleep, sometimes just shit luck. Very rarely it's lack of responsibility. And regarding your team members: if you don't trust them, don't rant about it on public forum, man, move them away from importan…
Hmmm, what you’ve described is lack of responsibility of recognising one’s own failings. There’s no excuse for that.
Re: I deleted data from production
#66There are three examples in the article: removing /usr (!!), running "hdfs dfs -rm /data" and "terraform destroy". Even with basic knowledge, these are clearly destructive operations - and to top it all, being run in Production. What strikes me as odd is that in all the cases, there is no change control. And by that I don't necessarily mean a full-on ServiceNow-type change process; I mean there was no "review". Even…
I never said it was in the same company.
Re: I deleted data from production
#67This is why he is Data Engineering Coach and not actually responsible for production systems now. (apparently) Everyone loves reading tech horror stories and peeling off the take-away lessons. My lesson would be, don't hire this guy and don't use him as a coach. He's careless! "But this kind of thing could happen to anyone" - sure, anyone who is irresponsible. Am I ranting? Let me continue. I work on a team now with…
Re: I deleted data from production
#68This is why he is Data Engineering Coach and not actually responsible for production systems now. (apparently) Everyone loves reading tech horror stories and peeling off the take-away lessons. My lesson would be, don't hire this guy and don't use him as a coach. He's careless! "But this kind of thing could happen to anyone" - sure, anyone who is irresponsible. Am I ranting? Let me continue. I work on a team now with…