Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

We deleted the production database by accident

keepthescore.co

Post reply on HN
Print view View original

431–440 of 456 posts

Re: We deleted the production database by accident

#431

Post by mr_toad »

oppositelock wrote: post #151

You have to put a lot of thought into protecting and backing up production databases, and backups are not good enough without regular testing of recovery. I have been running Postgres in production supporting $millions in business for years. Here's how it's set up. These days I use RDS in AWS, but the same is doable anywhere. First, the primary server is configured to send write ahead logs (WAL) to a secondary server…

> I have all these wonderful DB backups lying around, so I bring up a new DB from the backups

It’s nice to have that capability, but some databases are just too big to have multiple copies lying around, or to able to create a sandbox for everyone.

Re: We deleted the production database by accident

#432

Post by mr_toad »

aszen wrote: post #303
heavenlyblue wrote: post #237

Earlier quoted context omitted.

And you didn’t even bother to do a query of the actual maximum length value of the columns you were mutating? Or at least query and see the text in there? Basically you just blindly ran the migration on the data and checked if it didn’t fail? The lesson here is not about cleverness unfortunately.

I did see some values and found them reasonable, problem of the whole thing was there were atleast 200 or so tables with dozens of columns each and only two or so tables were excluded from being dumped locally. So yes I could have noticed their length 0 if I had looked carefully amidst hundreds of rows but since my faulty logic of prod db = local db didn't even consider this possible I didn't bother. If it had been j…

> my faulty logic of prod db = local db didn't

It happens. “It worked in dev” is the database equivalent of “worked on my machine”.

Re: We deleted the production database by accident

#433

Post by necovek »

Agentlien wrote: post #194
necovek wrote: post #176

Earlier quoted context omitted.

Or a blog on being unable to drive your kid to an emergency room because you just finished a glass of wine over dinner. A problem with devices of that type is that they only test for a potential source of inability to drive safely. What we want is to test for an inability to drive safely. And while one is easy and might give some quick wins, the drawbacks scare me too much.

Being unable to take your car when your child needs to go to the ER would be terrible. Actually getting in the car while under the influence such of stress and alcohol sounds worse. I know someone who had a glass of wine just before her daughter needed to be brought to the hospital. This was just two days ago. She simply concluded she could not drive. Luckily, she was able to get a taxi.

You are still simply going with one glass of wine will affect everyone equally, and that is proven to be untrue. (Or stress, for that matter)

While your friend made a call judging their own abilities and the level of emergency, that's exactly how it should be: cars should not stop us humans for making that decision.

(Fwiw, if you were just having an alcoholic drink, a breathalizer would show a much higher concentration even though alcohol might not have even kicked in or there wasn't enough for it to kivk in at all)

Re: We deleted the production database by accident

#434

Post by cutemonster »

fogihujy wrote: post #422
cutemonster wrote: post #411

Earlier quoted context omitted.

How do you prevent forwarding ports? Then one needs to disable ssh access? Nice that you were a person he felt ok with sharing the mistake with, I suppose that's an important part of being head of eng.

`AllowTcpForwarding No` There are ways around it, of course, but it prevents the scenario described above.

Thanks

Re: We deleted the production database by accident

#435

Post by Talinx »

jonplackett wrote: post #425
janoc wrote: post #269

Earlier quoted context omitted.

It is not only that but also realizing that there is never a single cause to an accident or incident. Even when it was a suicidal pilot flying the plane into a mountain on purpose. Someone had to supervise him (there are two crew members in the cockpit for a reason), someone gave him a medical, there is automation in the cockpit that could have at least caused an alarm, etc. So even when the accident is ultimately ca…

It’s interesting that this is the exact opposite of how we think about crime and punishment. All criminals are like the pilot, just the person who did the action. But the reasons for them becoming criminals is a seldom taken into account. The emphasis is on blaming and punishing them rather than figuring out the cause and stopping it happening again.

To stop the cause from happening is not always feasible. It might also be against human rights.

Re: We deleted the production database by accident

#436

Post by janoc »

confidantlake wrote: post #392
janoc wrote: post #361

Earlier quoted context omitted.

I was referring specifically to the Germanwings incident. That pilot shouldn't have been in the cockpit to begin with - his eyesight was failing, he had mental problems (has been medically treated for suicidal tendencies), etc. This was not discovered nor identified, due to deficiencies in the system (doctors didn't have the duty to report this, he withheld the information from his employer, etc.) The issue with the…

The change to require a cabin crew member in the cockpit is a good one. Not really sure what you can about the suicidal tendencies. If you make pilots report medical treatment for suicidal tendencies, they aren't going to seek treatment for suicidal tendencies.

That should have been reported by the doctor. Lubitz (the pilot) was denied an American license for this before - and somehow it wasn't caught/discovered when he got the Lufthansa/Germanwings job. Or nobody has followed up on it.

On the day of the crash he was not supposed to be on the plane at all - a paper from the doctors was found at his place after the crash declaring him unfit for duty. He kept it from his employer and it wasn't reported by the doctors neither (they didn't have the duty to do so), so the airline had no idea. Making a few of the holes in the cheese align nicely.

Pilots have the obligation to report when they are unfit for duty already, (no matter what the reason, being treated for a psychiatric problem certainly applies, though).

What was/is missing is the obligation of doctors to report such important issue to the employer when the crewman is unfit. It could be argued that it would be an invasion of privacy but there are precedents for this - e.g. failed medicals are routinely being reported to the authorities (not just for pilots - also for car drivers, gun holders, etc. where the corresponding licenses are then suspended), as are discoveries of e.g. child abuse.

Re: We deleted the production database by accident

#437

Post by janoc »

odyssey7 wrote: post #368
janoc wrote: post #269

Earlier quoted context omitted.

It is not only that but also realizing that there is never a single cause to an accident or incident. Even when it was a suicidal pilot flying the plane into a mountain on purpose. Someone had to supervise him (there are two crew members in the cockpit for a reason), someone gave him a medical, there is automation in the cockpit that could have at least caused an alarm, etc. So even when the accident is ultimately ca…

My impression of the Swiss cheese model is that it's used to take liability from the software vendor and (optionally) put it back on the software purchaser. Sure, there was a software error, but really, Mr. Customer, if this was so important, then you really should have been paying more attention and noticed the data issues sooner.

Nonsense.

Software vendor cannot be held responsible for errors committed by the user.

That would be blaming a parachute maker for the death of the guy who jumped out of a plane without a parachute or with one rigged wrong despite the explicit instructions (or industrial best practices) telling him not to do so.

Certainly vendors need to make sure that their product is fit for the purpose and doesn't contain glaring design problems (e.g. the infamous Therac-25 scandal) but that alone is not enough to prevent a disaster.

For example, in the cited article there was no "software error". The data haven't been lost because of a bug in some 3rd party code.

Data security and safety is always a process, there is no magic bullet you can buy and be done with it, with no effort of your own.

The swiss cheese model shows this - some of the cheese layers are safeguards put in place by the vendor, the others are there for you to put in place (e.g. the various best practices, safe work procedures, backups, etc.) If you don't, well, you are making the holes easier to align because there are now fewer safety layers between you and the disaster. By your own choice.

Re: We deleted the production database by accident

#438

Post by Aeolun »

tcbasche wrote: post #428
Aeolun wrote: post #291

Earlier quoted context omitted.

> taking responsibility for their own incompetence. I think they’re doing that with this post? At least I find it hard to imagine myself writing down that I’d drunk a few glasses of wine and dropped the production database. You cannot expect all engineers to be fully versed in the vagarities of database administration. Especially if they’re the only ones working on something.

Not really, they blamed 'complexity' and 'computer gremlins' rather than admitting that, perhaps, they made a shitty mistake. > It’s a function that deletes the local database and creates all the required tables from scratch Why would anyone have this? It's just dumb and embarrassing

> Why would anyone have this? It's just dumb and embarrassing

Anyone doing some form of decent integration testing?

Generally it’s a different database than the one used to develop locally, but the concept is the same.

Re: We deleted the production database by accident

#439

Post by Aeolun »

Agentlien wrote: post #298
Aeolun wrote: post #264

Earlier quoted context omitted.

That’s a problem with the ambulance service. Not with people being able to drive while drunk.

Yes. I was answering a comment suggesting the use of an ambulance (instead of a regular taxi). Simply pointing out that, in practice, there are times when a taxi can get you there faster.

Fair enough :)

Re: We deleted the production database by accident

#440

Post by amingilani »

nullsense wrote: post #224
amingilani wrote: post #99

Earlier quoted context omitted.

It does but that was an exceptional role. The company needed emergency patches to a running product while they hired a whole engineering team. As such, I was the only one around doing things, and there wasn't any documentation for me to work off of. I actually waited until nightfall just incase I bumped the server offline because we had low traffic during those hours.

What's the story behind this company/job? Was it some sort of total dumpster fire?

I wouldn't classify it as that but they had had trouble in the past which lead to a lot of their team leaving, and were now looking to recover from it.

I was only there for a short time though. Hopefully they figured things out.

Post reply on HN