I deleted data from production
1–10 of 68 posts
Re: I deleted data from production
#2I share 3 stories about my previous experience when I deleted not on purpose data in my Hadoop clusters or when we ran terraform destroy over our GCP projects.
This is a personal post about what I learnt and some takeaways.
Did you ever feel the same way?
Re: I deleted data from production
#3I share 3 stories about my previous experience when I deleted not on purpose data in my Hadoop clusters or when we ran terraform destroy over our GCP projects. This is a personal post about what I learnt and some takeaways. Did you ever feel the same way?
Thank you for sharing your experience !
Re: I deleted data from production
#4> Create a good wheel environment, ask for help and *do hide stuff from colleagues*
Quoi? Do *not* hide stuff from colleagues.
That's probably what ended up causing `terraform destroy`.
Do not hide stuff from your coworkers. It's toxic in a workplace. Hopefully it's a typo.
Re: I deleted data from production
#5IAC promise was ease of re-creation. [My anecdotal] Reality shows that's not really the case. IAC automation helps to create 'easy parts' and to delete _everything_. Creation of hard pars, the ones which involve state is still a problem. After many ears of using cfengine/chef/puppet/ansible/terraform I can say those tools don't help much with solving infra complexity. Early arguments were that 'nobody understand those shell scrips' or 'know manually edited configs'. Now the same shell script live inside container 'sidecars' and do 'pip install' from them and instead of manually altered configs we have k8s annotations with unpredictable 'far reaching' inter-references.
Re: I deleted data from production
#6IAC promise was ease of re-creation. [My anecdotal] Reality shows that's not really the case. IAC automation helps to create 'easy parts' and to delete _everything_. Creation of hard pars, the ones which involve state is still a problem. After many ears of using cfengine/chef/puppet/ansible/terraform I can say those tools don't help much with solving infra complexity. Early arguments were that 'nobody understand thos…
The hard stateful parts should be your data layer (replicated db, etc) and everything else should be cached stateless data. IAC for the stateful layer is tough but contained. IAC for the rest should be fairly recreatable quite quickly.
Re: I deleted data from production
#7> Create a good wheel environment, ask for help and *do hide stuff from colleagues* Quoi? Do *not* hide stuff from colleagues. That's probably what ended up causing `terraform destroy`. Do not hide stuff from your coworkers. It's toxic in a workplace. Hopefully it's a typo.
I'm pretty sure it's meant to include "not" with regard to the ethos of the entire article. (Ought to be fixed though.)
Re: I deleted data from production
#8A friend of mine once ran the Ansible equivalent of terraform destroy in our aws prod env in the middle of the day (of a fintech company). I could see how several if our prod machines got terminated. Fortunately the ones that mattered the most, had termination protection. But it was a scary moment.
Re: I deleted data from production
#9This is why he is Data Engineering Coach and not actually responsible for production systems now. (apparently)
Everyone loves reading tech horror stories and peeling off the take-away lessons. My lesson would be, don't hire this guy and don't use him as a coach. He's careless!
"But this kind of thing could happen to anyone" - sure, anyone who is irresponsible.
Am I ranting? Let me continue. I work on a team now with someone who seems to consistently forget to save router configurations and another person who yawns through meetings because he doesn't sleep. These guys make careless mistakes and we "learn" from them. Except we don't learn the most important lesson: Irresponsible people need to be off the team. They make more work for everyone else and make the team look bad.
But I guess the tech horror stories that amuse everyone on the forum or at the pub are better than saying, "I am responsible and do good work."
Re: I deleted data from production
#10The stories scream incompetent people being incompetent.