My database distributes data in real-time globally: http://root.rupy.se
That way it's almost impossible to have anything catastrophic happen unless the whole planet goes boom!
I deleted data from production
41–50 of 68 posts
Re: I deleted data from production
#42Talkinf of personal stories, I'd be very happy to have a tool that enforces some verification process over SQL operation ran on production database (you know, to fix that horrible data quality issue on 2 rows that screws the whole business logic in the upper layer that would take a month to fix). Basically a tool that allow someone to run some SQL on production but, before doing it, request his credentials, then run…
As a first step toward your wish, I suggest using mycli or pgcli on servers instead of the default mysql or pgsql client. By default, it asks for confirmation whenever you type a destructive command. It does have a log, just like the official CLI clients. It also provides completion and syntax highlighting.
https://github.com/dbcli/
Re: I deleted data from production
#43> Create a good wheel environment, ask for help and do hide stuff from colleagues
(emphasis mine) Was this a typo? If it is, it's kind of funny that an article about making mistakes in production also made this tiny but critical mistake.
Re: I deleted data from production
#44This is why he is Data Engineering Coach and not actually responsible for production systems now. (apparently) Everyone loves reading tech horror stories and peeling off the take-away lessons. My lesson would be, don't hire this guy and don't use him as a coach. He's careless! "But this kind of thing could happen to anyone" - sure, anyone who is irresponsible. Am I ranting? Let me continue. I work on a team now with…
In jest but personally I don't think you are a real programmer unless you have done something like this. It's kind of "welcome to the club" right of passage
Making mistakes like these in dev/test/sandbox/playground environments is one thing, but making them in prod over and over again is different.
Yes, everyone should experience making catastrophic mistakes but please spare some effort to only do it in non-prod...
Re: I deleted data from production
#45This is why he is Data Engineering Coach and not actually responsible for production systems now. (apparently) Everyone loves reading tech horror stories and peeling off the take-away lessons. My lesson would be, don't hire this guy and don't use him as a coach. He's careless! "But this kind of thing could happen to anyone" - sure, anyone who is irresponsible. Am I ranting? Let me continue. I work on a team now with…
That's why you have quality assurance. People make mistakes. Your process shall be able to detect and correct them.
Re: I deleted data from production
#46Earlier quoted context omitted.
None of the code for consumer production is verified formally. So please spare us the bs that you make no mistakes. You are just lucky. Be humble because the complexity of modern systems is insane, there is no way you have all cases covered, if you had you would had a formal proof. We all are just doing our best to cover most of the edge cases. That is why we need to keep learning from other peoples mistakes.
Read the article. We're not talking about formal verification. >It's Sunday morning and I just discovered that I've lost 3To of data and that all data pipelines have stop working because on Friday I ran for no reason hdfs dfs -rm /data This is profound incompetence.
That's why you have a backup. I also noticed that i'm incompetent on friday evenings so i avoid doing sysadmin work then.
Re: I deleted data from production
#47This is why he is Data Engineering Coach and not actually responsible for production systems now. (apparently) Everyone loves reading tech horror stories and peeling off the take-away lessons. My lesson would be, don't hire this guy and don't use him as a coach. He's careless! "But this kind of thing could happen to anyone" - sure, anyone who is irresponsible. Am I ranting? Let me continue. I work on a team now with…
I really like the effort you made in following more than 2 links to see what I'm doing. The issue is that you also don't have the full context on everything that happened.
I don't know on which kind of teams you work in, but I personally worked a lot in teams where I was the only person responsible to do data related tech stuff for other people. I never said that I wasn't responsible but I was alone, so setuping backup was often the last of my priorities. I was and I did all my best to fix things as fast as possible and to remove any issues it could lead in the future.
When I was setuping Hadoop cluster in 2014, almost no-one knows what it was in my local market. So let us try, then fail and learn.
If you also think from 3 stories I'm writing on the internet that I'm careless so be it. But I also recommend you to fix your "router configuration" problem rather than ranting on posts on internet.
Re: I deleted data from production
#48> Create a good wheel environment, ask for help and do hide stuff from colleagues (emphasis mine) Was this a typo? If it is, it's kind of funny that an article about making mistakes in production also made this tiny but critical mistake.
Yes, oops.
Re: I deleted data from production
#49I've had someone do, in essence, a `terraform destroy`. I don't think we ever learned if it was that, or some other command … just that it was indeed a terraform removal. The plan was not read, of course. Instant incident. More recently, had a coworker want to run terraform plan, couldn't, because the state file is access controlled, and so he just reset the configuration locally to a blank local state, plan (which w…
> I can't tell why the copy in the article didn't work? You should be able to move /usr to another partition, no? The way it reads, they didn't know how POSIX permissions work, and maybe still don't. The article says that sudo needs to be owned by uid 0, but does not mention the critical suid bit. Maybe they just recursively copied /usr without preserving permissions? Also, the mention of sudo makes it pretty clear t…
Yeah I made some shortcuts about it in the post for the sake of trying to keep it short. But the biggest issue with sudo was the suid bit you're right.
You're totally right in your message. I did not had access to root password.
I was junior and while I was trying to fix stuff I was adding more problem to the initial one that was a disk space issue.
Re: I deleted data from production
#50> Create a good wheel environment, ask for help and *do hide stuff from colleagues* Quoi? Do *not* hide stuff from colleagues. That's probably what ended up causing `terraform destroy`. Do not hide stuff from your coworkers. It's toxic in a workplace. Hopefully it's a typo.
I'm pretty sure it's meant to include "not" with regard to the ethos of the entire article. (Ought to be fixed though.)
Yeah, fixed and oops too late because everyone read it.