The maturity of the article is laughable. I'm sure my age is the same as the people who wrote it, but this is unacceptable: dropping databases in prod is a serious issue, not a joke. I think the culture of the company is toxic and not professional at any level. #change-my-mind
We deleted the production database by accident
351–360 of 456 posts
Re: We deleted the production database by accident
#352Earlier quoted context omitted.
"fear required when working out what decision to make" People like you keep making the same mistake, creating companies/organisations/industries/societies that run on fear of failure. We've tried it a thousand times, and it never works. You can't solve crime by making all punishments hearsh death, we've tried that in 1700 in Britain and crimerate was sky high. This culture gave us disasters in USSR and famine in Chin…
I think my point is probably being misunderstood and that is my fault for explaining it poorly. See I fucked up :) The fear I speak of is a personal barrier which is lacking in a lot of people. They can sleep quite happily at night knowing they did a shitty job and it's going to explode down the line. It's not their problem. They don't care. I can't do that. Even if there are no direct consequences for me. This is no…
Yes, medical professionals use checklists. They also have a harsh and very unforgiving culture that fosters craftsmanship and values professionalism above all else. You see this in other high-stakes professions too.
You cannot just take the checklist and ignore the relentless focus on quality, the feelings of personal failure and countless hours honing and caring for the craft.
Developers are notorious for being lazy AF, so it's not hard to explain our obsession with "just fix the system". It's a required but not sufficient condition.
Re: We deleted the production database by accident
#353Re: We deleted the production database by accident
#354I'm appalled at the way some people here receive an honest postmortem of a human fuck-up. The top 3 comments, as I write this, can be summarized as "no, it's your fault and you're stupid for making the fault". This is not good! We don't want to scare people into writing less of these. We want to encourage people to write more of them. An MBA style "due to a human error, we lost a day of your data, we're tremendously…
Get some sleep, do a thorough investigation, and the results of that are the post mortem that we would like published and where you learn from.
Publishing some premature thoughts without actual insight is not helping anybody. It will just invite the hate that you are seeing in this thread.
Re: We deleted the production database by accident
#355Earlier quoted context omitted.
I think we're hitting peak tech. All this "technical" knowledge just dates itself in a year's time anyway. Eventually, you come to realise that the more tech you've got, the more problems you have. . Now developers spend more time googling errors and plugging in libraries and webservices together than writing any actual code. Sometimes I wish for a techless cloudless revolution when we just go back to the foundations…
> Sometimes I wish for a techless cloudless revolution when we just go back to the foundations of computers and is use plain text wherever possible. ... because software in the 60s/70s/80s was so reliable and bug-free?!
Re: We deleted the production database by accident
#356Earlier quoted context omitted.
Yeah, but "pilot was drinking alcohol" would be considerate issue, would lead to fired pilot and would lead to more alcohol testing. I understand what you are taking about, but aviation has also strong expectations on pilots.
Believe it or not, even "pilot is an alcoholic" is still part of the no blame culture in aviation. As long as the pilot reports himself he'll not be fired for that. Look up the HIMS program to read more details.
Re: We deleted the production database by accident
#357Earlier quoted context omitted.
Do use CSV (and other similar formats) for read-only data which fits entirely in the memory. It is great for data safety -- chown/chmod the file, and you can be sure your scripts won't touch this. And if you are accessing live instance, you can be pretty sure that you won't accidentally break it by obtaining a database lock. Now "csv" in particular is kinda bad because it never got standardized, so if you you have co…
I agree with the JSON suggestion, but what advantage is there to TSV versus CSV? I have experienced pain with both characters (tab and comma), particularly when I am not the one creating the output file.
You use this tsv with Unix tools like "cut", "paste", and ad-hoc scripts.
There is also "tsv" as defined by excel, which has quoting and stuff. It is basically a dialect of CSV (Python even uses the same module to read it), and has all the disadvantages of CSV. Avoid it.
Re: We deleted the production database by accident
#358Shit happens. You learn and try to never repeat it. And share with others so hopefully they learn.
Ps. Don't do knee-jerk late at night quick patches. For example don't stop a database that has run out of disk space, try to migrate the data in memory first... And also do proper backup monitoring, and restores. Having 30 days of 0 byte backups is not that helpful. :)
Re: We deleted the production database by accident
#359At one point someone called it "good enough" and they basically had to honor the customer word if they had purchased something and it wasn't there.
It was a mess.
It was on all major news, and it was really bad press. In the end, they actually had a massive bump in their sales afterwards. Everyone went to checkout their own purchases and ended up buying something else, and the news was like free ads.
https://www.washingtonpost.com/business/capitalbusiness/the-...