Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

We deleted the production database by accident

keepthescore.co

Post reply on HN
Print view View original

361–370 of 456 posts

Re: We deleted the production database by accident

#361

Post by janoc »

quietbritishjim wrote: post #324
janoc wrote: post #269

Earlier quoted context omitted.

It is not only that but also realizing that there is never a single cause to an accident or incident. Even when it was a suicidal pilot flying the plane into a mountain on purpose. Someone had to supervise him (there are two crew members in the cockpit for a reason), someone gave him a medical, there is automation in the cockpit that could have at least caused an alarm, etc. So even when the accident is ultimately ca…

> Even when it was a suicidal pilot flying the plane into a mountain on purpose. Someone had to supervise him (there are two crew members in the cockpit for a reason), someone gave him a medical, there is automation in the cockpit that could have at least caused an alarm, etc. There was indeed a suicidal pilot that flew into a mountain, I'm not sure if you were deliberately referencing that specific time. In that cas…

I was referring specifically to the Germanwings incident.

That pilot shouldn't have been in the cockpit to begin with - his eyesight was failing, he had mental problems (has been medically treated for suicidal tendencies), etc. This was not discovered nor identified, due to deficiencies in the system (doctors didn't have the duty to report this, he withheld the information from his employer, etc.)

The issue with the door was only the last element of the chain.

There were changes as the result of this incident - the cabin crew member has to be in the cockpit whenever one of the pilots steps out, there were changes to how the doors operate, etc.

Re: We deleted the production database by accident

#363

Post by user5994461 »

t0mas88 wrote: post #338
watwut wrote: post #262

Earlier quoted context omitted.

Yeah, but "pilot was drinking alcohol" would be considerate issue, would lead to fired pilot and would lead to more alcohol testing. I understand what you are taking about, but aviation has also strong expectations on pilots.

Believe it or not, even "pilot is an alcoholic" is still part of the no blame culture in aviation. As long as the pilot reports himself he'll not be fired for that. Look up the HIMS program to read more details.

You can google and find cases of US pilots getting fired and sentenced to a year in prison for flying intoxicated.

Maybe they don't get fired if they report themselves unable to fly beforehand but I wouldn't quite call that a no blame culture.

Re: We deleted the production database by accident

#366

Post by janoc »

globular-toast wrote: post #277
janoc wrote: post #269

Earlier quoted context omitted.

It is not only that but also realizing that there is never a single cause to an accident or incident. Even when it was a suicidal pilot flying the plane into a mountain on purpose. Someone had to supervise him (there are two crew members in the cockpit for a reason), someone gave him a medical, there is automation in the cockpit that could have at least caused an alarm, etc. So even when the accident is ultimately ca…

There is sometimes a single cause, but as the parent comment pointed out, that should never be the case and is a flaw in the system. We are gradually working towards single errors being correctable, but we're not there yet. On the railways in Britain the failures were extensively documented. Years ago it was possible for a single failure to cause a loss. But over the years the systems have been patched and if you loo…

There is almost never a single cause. If a single cause can trigger a disaster, then there is another cause by definition - poor system design.

E.g. in the article's case it is clear that there is some sort of procedural deficiency there that allows the configuration variables to be set wrong and thus cause a connection to the wrong database.

Another one is that the function that has directly caused the data loss DOES NOT CHECK for this.

Yet another WTF is that if that code is meant to ever run on a development system, why is it in a production codebase in the first place?

And the worst bit? They throw arms up in the air, unable to identify the reason why this has happened. So they are leaving the possibility open to another similar mistake happening in the future, even though they have removed the offending code.

Oh and the fact that they don't have backups except for those of the hosting provider (which really shouldn't be relied on except as the last hail Mary solution!) is telling.

That's not a robust system design, especially if they are hosting customers' data.

Re: We deleted the production database by accident

#367

Post by ystad »

> Thankfully our database is a managed database from DigitalOcean, which means that DigitalOcean automatically do backups once a day. Do cloud providers provide a smaller window for backup. Are there better ways to reduce the backup window here for DBs? Would love to understand any techniques folks use to minimize the backup window?

Re: We deleted the production database by accident

#368

Post by odyssey7 »

janoc wrote: post #269
t0mas88 wrote: post #243

Earlier quoted context omitted.

The software sector needs a bit of aviation safety culture: 50 years ago the conclusion "pilot error" as the main cause was virtually banned from accident investigation. The new mindset is that any system or procedure where a single human error can cause an incident is a broken system. So the blame isn't on the human pressing the button, the problem is the button or procedure design being unsuitable. The result was a…

It is not only that but also realizing that there is never a single cause to an accident or incident. Even when it was a suicidal pilot flying the plane into a mountain on purpose. Someone had to supervise him (there are two crew members in the cockpit for a reason), someone gave him a medical, there is automation in the cockpit that could have at least caused an alarm, etc. So even when the accident is ultimately ca…

My impression of the Swiss cheese model is that it's used to take liability from the software vendor and (optionally) put it back on the software purchaser. Sure, there was a software error, but really, Mr. Customer, if this was so important, then you really should have been paying more attention and noticed the data issues sooner.

Re: We deleted the production database by accident

#369

Post by wizzwizz4 »

qz2 wrote: post #255
suzakus wrote: post #246

Earlier quoted context omitted.

It's a piece of software for scoreboards. Not the Therac-25, nor an airplane.

This time. Some days it’s just an on line community that gets burned to the ground. Other days it’s just a service tied into hundred of small businesses that gets burned to the ground. Other says it’s massive financial platform getting burned to the ground. I’m responsible for the latter but the former two have had a much larger impact for many people when they occur. Trivialising the lax administrative discipline be…

Prismo is an example of the first: https://fediverse.blog/~/Prismo/on-prismo-data-loss

The project never recovered.

Re: We deleted the production database by accident

#370

Post by rwbhn »

Agentlien wrote: post #228
kungtotte wrote: post #206

Earlier quoted context omitted.

Maybe we should, as a society, invest in taxis equipped with medical facilities and trained personnel so that they can provide first response medical treatment while on the way to the ER. I'm sure that would save a lot of lives. An ambulatory medical service, if you will.

Sometimes ambulances are occupied and taking a taxi goes faster. Especially if it's something which isn't immediately life threatening. I once dislocated my shoulder while on a large trampoline and was unable to get up from my hands and knees due to the intense pain whenever the trampoline wobbled. The ambulance was redirected to more serious injuries three times. I was stuck in that position waiting for two hours be…

Note that ambulance ride (depending on insurance) may cost an order or 2 of magnitude more than the taxi. Well worth it in some circumstances - but not always the best option.
Post reply on HN