Github did something similar. https://github.blog/2010-11-15-today-s-outage/
We deleted the production database by accident
391–400 of 456 posts
Re: We deleted the production database by accident
#392Earlier quoted context omitted.
> Even when it was a suicidal pilot flying the plane into a mountain on purpose. Someone had to supervise him (there are two crew members in the cockpit for a reason), someone gave him a medical, there is automation in the cockpit that could have at least caused an alarm, etc. There was indeed a suicidal pilot that flew into a mountain, I'm not sure if you were deliberately referencing that specific time. In that cas…
I was referring specifically to the Germanwings incident. That pilot shouldn't have been in the cockpit to begin with - his eyesight was failing, he had mental problems (has been medically treated for suicidal tendencies), etc. This was not discovered nor identified, due to deficiencies in the system (doctors didn't have the duty to report this, he withheld the information from his employer, etc.) The issue with the…
The change to require a cabin crew member in the cockpit is a good one.
Not really sure what you can about the suicidal tendencies. If you make pilots report medical treatment for suicidal tendencies, they aren't going to seek treatment for suicidal tendencies.
Re: We deleted the production database by accident
#393Earlier quoted context omitted.
Sometimes ambulances are occupied and taking a taxi goes faster. Especially if it's something which isn't immediately life threatening. I once dislocated my shoulder while on a large trampoline and was unable to get up from my hands and knees due to the intense pain whenever the trampoline wobbled. The ambulance was redirected to more serious injuries three times. I was stuck in that position waiting for two hours be…
Note that ambulance ride (depending on insurance) may cost an order or 2 of magnitude more than the taxi. Well worth it in some circumstances - but not always the best option.
This didn't spring to my mind as I'm Swedish and here it's less than a taxi and any medical costs beyond the first USD $130 per year is covered by the free health insurance.
Re: We deleted the production database by accident
#394This postmortem is incomplete: it fails to address the main three roots of the problem:
1. This business is too flippant with their write-able production access.
2. No user should have DROP DATABASE grants on production.
3. Clearly one of their employees was using a port forward to access production.
Re: We deleted the production database by accident
#395While several other users have posted takeaways for how to prevent this from happening, I'd be interested in if anybody has an idea of how this happened given the code that was posted?
Presumably, a managed DB service should essentially never be available on `localhost`. Additionally, it would be very weird for `config.DevelopmentConfig` to return the production database credentials.
Re: We deleted the production database by accident
#396This post is embarrassing. "yeah we were drinking and accidentally nuked the prod DB. Not sure why. Shit happens!" Who would read this and think they should trust this company? Any number of protections could have been taken to prevent this and production access in any state other than fully alert and attentive shouldn't happen unless it is absolutely necessary for emergency reasons
This reply is embarrassing. It's a person working on their side project. Have a glass of wine mate.
A previous post said it was "almost free" and I feel like a less cavalier attitude is called for if people are paying for the service. Otherwise sure it doesn't matter
Re: We deleted the production database by accident
#397Earlier quoted context omitted.
If you can do this, then yes by all means do it, but that has significant impact on disk usage.
It doesn’t have to be local. In fact it shouldn’t be local anyway as backups accessible to be deleted from the source aren’t real backups any way. You can the restore from a recent base backup and roll forward the WAL to just before the snafu.
Yeah, but that can still create a situation where you can create WALs faster than they're uploaded. (And if there's a good solution to that, I'm interested, because we ran a server out of disk this way last month at work)
Re: We deleted the production database by accident
#398Earlier quoted context omitted.
I disagree. Culturally speaking we like to pat people on their back when they do something stupid and comfort them. But most of the time this isn’t productive because it doesn’t instil the requisite fear required when working out what decision to make. What happens is we have growing complacency and disassociation from consequences. Do you press the button on something potentially destructive because your are confide…
"fear required when working out what decision to make" People like you keep making the same mistake, creating companies/organisations/industries/societies that run on fear of failure. We've tried it a thousand times, and it never works. You can't solve crime by making all punishments hearsh death, we've tried that in 1700 in Britain and crimerate was sky high. This culture gave us disasters in USSR and famine in Chin…
> The only thing that can solve this problem is structural change.
Well, care to elaborate on this? What do we have to change, and to what end?
Re: We deleted the production database by accident
#399Earlier quoted context omitted.
I have a little metadata table in production that has a field that says “this is a production database”. The delete-everything script reads that flag via a SQL query that will error out of it’s set in the same transaction as the deletion. To prevent the flag from getting cleared in production, the production software stack will refuse to run if the “production” flag is not set.
I would flip the logic. If database does not have flag that says it is non-production assume it is production.
It's a BOOLEAN NOT NULL. I don't recall off the top of my head whether TRUE means production or TRUE means testing.
Re: We deleted the production database by accident
#400Earlier quoted context omitted.
This is why I am against the current trend of over-complicating stacks for political or marketing reasons. Every startup nowadays wants microservices and/or serverless and a mashup of dozens of different SaaS (some that can't easily be simulated locally) from day 1 while a "boring" monolithic app will get them running just fine.
Yup, 100% agree. It may be that you will eventually need an auto-scalable message queue and api gateway, but for most people a web server and csv will serve the first thousand customers
"for most people a web server and csv will serve the first thousand customers"
I hear this kind of thought-terminating cliche a lot on here and it makes absolutely no sense.
If # of users is a rough approximate of a company's success and more successful companies tend to hire more engineers ... then actually the majority of engineers would not have the luxury of not needing to think about scalability.
With engineering salaries being what they are, why would you think that "most people" are employed working on systems that only have 1000 users?