Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

We deleted the production database by accident

keepthescore.co

Post reply on HN
Print view View original

371–380 of 456 posts

Re: We deleted the production database by accident

#371

Post by rwbhn »

robryan wrote: post #251
dingdingdang wrote: post #231

Earlier quoted context omitted.

Yeah, my guess would be that the script got executed on prod server by, ups was I in that terminal window, accident! Localhost is after all the-local-host, no matter what server it's on. Better to also have clear convention regarding name of prod db name versus test db (i.e. "test_mysaas" versus "mysaas"). Plus of course using git with a hook specifically for preview versus production (i.e. "git push production") tha…

Yeah this is how I once deleted my production database. One thing I did to mitigate this was colour code the prompts for local/staging/production.

Yeah, if one has to regularly interact directly with prod, some sort of visual indicator is super helpful.

Re: We deleted the production database by accident

#372

Post by m463 »

t0mas88 wrote: post #243
skrebbel wrote: post #240

I'm appalled at the way some people here receive an honest postmortem of a human fuck-up. The top 3 comments, as I write this, can be summarized as "no, it's your fault and you're stupid for making the fault". This is not good! We don't want to scare people into writing less of these. We want to encourage people to write more of them. An MBA style "due to a human error, we lost a day of your data, we're tremendously…

The software sector needs a bit of aviation safety culture: 50 years ago the conclusion "pilot error" as the main cause was virtually banned from accident investigation. The new mindset is that any system or procedure where a single human error can cause an incident is a broken system. So the blame isn't on the human pressing the button, the problem is the button or procedure design being unsuitable. The result was a…

That reminds me of the time during the rise of the PC when windows would do something wrong, from a confusing interface all the way up to a blue screen of death.

What happened is that users started blaming themselves for what was going wrong, or start thinking they needed a new PC because problems would become more frequent.

From the perspective of a software guy, it was obvious that windows was the culprit but people would assign blame elsewhere and frequently point the finger at themselves.

so yes - an FAA investigation would end up unraveling the nonsense and point to windows.

That said, aviation level of safety is reliable and dependable and few single points of failure and... there are no private kit jets darnit!

There is a continuum from nothing changes & everything works to everything changes & nothing works. You have to choose the appropriate place on the dial for the task. Sounds like this is a one-man band.

Re: We deleted the production database by accident

#373

Post by AdrianB1 »

neillyons wrote: post #265
t0mas88 wrote: post #243

Earlier quoted context omitted.

The software sector needs a bit of aviation safety culture: 50 years ago the conclusion "pilot error" as the main cause was virtually banned from accident investigation. The new mindset is that any system or procedure where a single human error can cause an incident is a broken system. So the blame isn't on the human pressing the button, the problem is the button or procedure design being unsuitable. The result was a…

This sounds quite interesting. Any books you could recommend on the "pilot error" topic.

As a GA pilot I know people that had accidents with planes and I know that in most cases what is the the official report and what really happened are not the same, so any book would have to rely on inaccurate or unreal data. For airliners it is easy because there are flight recorders, for GA it is still a bit of Wild West.

Re: We deleted the production database by accident

#374

Post by alistairSH »

caspii wrote: post #286
cranekam wrote: post #177

Earlier quoted context omitted.

The whole piece has a slightly annoying flippant tone to it. We were drunk! Computers just.. do this stuff sometimes! Better to sound contrite and boring in such a situation IMO. Also I agree with other comments: doing some work after a glass or two should be fine because you should have other defences in place. “Not being drunk” shouldn’t be the only protection you have against disaster.

Yeah, I agree I'm being slightly flippant. But it's just a side-project and I will continue late night coding with a glass of wine. I find it hugely enjoyable. I would have a different mind-set if I was writing software for power stations as a professional.

But it's just a side-project and I will continue late night coding with a glass of wine.

Normally, this would be fine. But, it appears the site has paying members. Presumably, it's not "just a side-project" to them. You owe them better than tinkering with prod while tipsy.

Re: We deleted the production database by accident

#375

Post by AdrianB1 »

ivan_gammel wrote: post #272
danmaz74 wrote: post #210

Earlier quoted context omitted.

If you're talking about a web application or API back end with a smallish startup team, time to market is definitely going to be much longer for a microservices architecture compared to developing a "monolith" in a batteries included framework like eg rails, using a single database.

If you think it’s much longer, then you haven’t done it with modern frameworks. I’m CTO of a German startup, which went from a two-founder team to over 250 employees in 70 locations in 3 years. For us microservice architecture was crucial to deliver plenty of things in time. Low coupling, micro-teams of 2 ppl max working on several projects at once... we did not have luxury of coordinating monolith releases. Adding o…

It is probably not what you intended, but this is how it sounds like: we have a hundred micro-teams of 2 working in silo on low-coupling microservices and we don't have the luxury of coordinating an end to end design.

Edit: 2 questions were asked, too deep to reply. 1. You said 250 people, nothing about IT. Based on the info provided, this was the image reflected. 2. "the luxury of coordinating a monolith". Done well, it is not much more complicated that coordinating the design of microservices, some can argue it is the same effort.

Re: We deleted the production database by accident

#376

Post by gonzo41 »

odyssey7 wrote: post #368
janoc wrote: post #269

Earlier quoted context omitted.

It is not only that but also realizing that there is never a single cause to an accident or incident. Even when it was a suicidal pilot flying the plane into a mountain on purpose. Someone had to supervise him (there are two crew members in the cockpit for a reason), someone gave him a medical, there is automation in the cockpit that could have at least caused an alarm, etc. So even when the accident is ultimately ca…

My impression of the Swiss cheese model is that it's used to take liability from the software vendor and (optionally) put it back on the software purchaser. Sure, there was a software error, but really, Mr. Customer, if this was so important, then you really should have been paying more attention and noticed the data issues sooner.

You can't outsource risk.

Re: We deleted the production database by accident

#378

Post by ivan_gammel »

AdrianB1 wrote: post #375
ivan_gammel wrote: post #272

Earlier quoted context omitted.

If you think it’s much longer, then you haven’t done it with modern frameworks. I’m CTO of a German startup, which went from a two-founder team to over 250 employees in 70 locations in 3 years. For us microservice architecture was crucial to deliver plenty of things in time. Low coupling, micro-teams of 2 ppl max working on several projects at once... we did not have luxury of coordinating monolith releases. Adding o…

It is probably not what you intended, but this is how it sounds like: we have a hundred micro-teams of 2 working in silo on low-coupling microservices and we don't have the luxury of coordinating an end to end design. Edit: 2 questions were asked, too deep to reply. 1. You said 250 people, nothing about IT. Based on the info provided, this was the image reflected. 2. "the luxury of coordinating a monolith". Done well…

That’s an interesting interpretation, but... 1. our whole IT team is 15 people, covering every aspect of automation of a business with significant complexity of the domain and big physical component (medical practices). 2. can you elaborate more on end to end design? I struggle to find the way of thinking which could lead to that conclusion.

Re: We deleted the production database by accident

#379

Post by AdrianB1 »

emodendroket wrote: post #124
bigbubba wrote: post #95

Earlier quoted context omitted.

You know it's totally feasible to make a car that won't turn on for drunk people. Should those systems be installed on all cars, in pursuit of creating systems that don't permit stupid actions? Maybe such a breathalyzer interlock could be installed on your workstation too. After all, your systems and processes should prevent you from stupid things.

Honestly not a bad idea to install the interlocks on all cars.

It will be a great idea when the reliability of the system will have a large number of nines, so chances that you are stranded in the middle of nowhere and the car does not want to start because of a fault will be less than being hit by an asteroid. Other than that, people would consider it an unsafe product and refuse to use it and people vote for what finally becomes a law.

I heard the same argument for electronic gun safety measures, except that no government agency even consider using it for their own guns. Why? They are not reliable enough, yet.

Re: We deleted the production database by accident

#380

Post by AdrianB1 »

info781 wrote: post #360

Why did they not have archive log mode on in production? Losing a db is one thing, but should have only been one hour of data lost.

Focus, time, expertise and cost. For a side project with no revenue, cost is a very important factor. The others come with having side projects.
Post reply on HN