Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

We deleted the production database by accident

keepthescore.co

Post reply on HN
Print view View original

421–430 of 456 posts

Re: We deleted the production database by accident

#421

Post by hnlmorg »

alistairSH wrote: post #374
caspii wrote: post #286

Earlier quoted context omitted.

Yeah, I agree I'm being slightly flippant. But it's just a side-project and I will continue late night coding with a glass of wine. I find it hugely enjoyable. I would have a different mind-set if I was writing software for power stations as a professional.

But it's just a side-project and I will continue late night coding with a glass of wine. Normally, this would be fine. But, it appears the site has paying members. Presumably, it's not "just a side-project" to them. You owe them better than tinkering with prod while tipsy.

I don't think that's fair. We've all had occasions from time to time when we've had a drink at lunch time or even had to do emergency work in the evening after having a drink.

The bigger issue is the lack of guardrails to protect against accidental damage. This is also a common trait for hobby projects (after all, it's more fun to hack stuff together) but hopefully the maintainer will use this experience as a sobering reminder (pun intended) to put some guardrails in place now.

Re: We deleted the production database by accident

#422

Post by fogihujy »

cutemonster wrote: post #411
xtracto wrote: post #403

Earlier quoted context omitted.

This. The person that erased the database in my case came forward to me as soon as we realized what had happened. At that moment I was very happy it was an "inside job", it meant I could discard hacking. As its said before: he made a mistake. The error was allowing the prod database to to be port forwarded from a non prod environment. As head of eng that was MY error. So I owned to it and we changed policies.

How do you prevent forwarding ports? Then one needs to disable ssh access? Nice that you were a person he felt ok with sharing the mistake with, I suppose that's an important part of being head of eng.

`AllowTcpForwarding No`

There are ways around it, of course, but it prevents the scenario described above.

Re: We deleted the production database by accident

#423

Post by sushshshsh »

vp8989 wrote: post #400
sushshshsh wrote: post #105

Earlier quoted context omitted.

Yup, 100% agree. It may be that you will eventually need an auto-scalable message queue and api gateway, but for most people a web server and csv will serve the first thousand customers

"for most people a web server and csv will serve the first thousand customers" I hear this kind of thought-terminating cliche a lot on here and it makes absolutely no sense. If # of users is a rough approximate of a company's success and more successful companies tend to hire more engineers ... then actually the majority of engineers would not have the luxury of not needing to think about scalability. With engineerin…

"what is an MVP even", the post!

Re: We deleted the production database by accident

#424

Post by q6fqa »

KronisLV wrote: post #323
bigbubba wrote: post #95

Earlier quoted context omitted.

You know it's totally feasible to make a car that won't turn on for drunk people. Should those systems be installed on all cars, in pursuit of creating systems that don't permit stupid actions? Maybe such a breathalyzer interlock could be installed on your workstation too. After all, your systems and processes should prevent you from stupid things.

Replace a breathalyzer with something that's less intruisive (like a camera with AI that would observe the person, AI with thermal imaging or air quality sensors, or another possibly-fictional-yet-believable piece of technology) and suddenly, in my eyes, the technology in this thought experiment becomes a no brainer. If there was more of a cultural pressure against drunk driving and actual mechanisms to prevent it th…

Mercedes[1] has drowsiness detector, it observes the driver through sensors. It's freakishly accurate.

[1] https://media.daimler.com/marsMediaSite/en/instance/ko/ATTEN...

Re: We deleted the production database by accident

#425

Post by jonplackett »

janoc wrote: post #269
t0mas88 wrote: post #243

Earlier quoted context omitted.

The software sector needs a bit of aviation safety culture: 50 years ago the conclusion "pilot error" as the main cause was virtually banned from accident investigation. The new mindset is that any system or procedure where a single human error can cause an incident is a broken system. So the blame isn't on the human pressing the button, the problem is the button or procedure design being unsuitable. The result was a…

It is not only that but also realizing that there is never a single cause to an accident or incident. Even when it was a suicidal pilot flying the plane into a mountain on purpose. Someone had to supervise him (there are two crew members in the cockpit for a reason), someone gave him a medical, there is automation in the cockpit that could have at least caused an alarm, etc. So even when the accident is ultimately ca…

It’s interesting that this is the exact opposite of how we think about crime and punishment. All criminals are like the pilot, just the person who did the action. But the reasons for them becoming criminals is a seldom taken into account. The emphasis is on blaming and punishing them rather than figuring out the cause and stopping it happening again.

Re: We deleted the production database by accident

#426

Post by ikiris »

thunderrabbit wrote: post #213
ikiris wrote: post #193

Earlier quoted context omitted.

excuse me, but no. this is harmful bullshit. Yes, backups are vitally important, but no it is not possible to accidentally rm -rf with proper design. It's possible to have the most dangerous credentials possible and still make it difficult to do catastrophic global changes. Hell it's my job to make sure this is the case.

> not possible to accidentally rm -rf with proper design. Can you say more about this? I understand rm -rf, but not sure how I could design that to be impossible for the most dangerous credentials.

You can make the most dangerous credentials involve getting a keycard from a safe, and multi party sign off, not possible to deploy to more than X machines at a time with a sliding window of application, independent systems with proper redundant and failback design, canary analysis, etc etc etc.

I didn't even mean you can only make it difficult, I meant you can make it almost impossible to harm a real production environment in such a nuclear way without herculean effort and quite frankly likely collusion from multiple parties.

Re: We deleted the production database by accident

#427

Post by xtracto »

cutemonster wrote: post #411
xtracto wrote: post #403

Earlier quoted context omitted.

This. The person that erased the database in my case came forward to me as soon as we realized what had happened. At that moment I was very happy it was an "inside job", it meant I could discard hacking. As its said before: he made a mistake. The error was allowing the prod database to to be port forwarded from a non prod environment. As head of eng that was MY error. So I owned to it and we changed policies.

How do you prevent forwarding ports? Then one needs to disable ssh access? Nice that you were a person he felt ok with sharing the mistake with, I suppose that's an important part of being head of eng.

Nope . The solution is to password protect and not give the pass to developers. Or only give read only access.

Re: We deleted the production database by accident

#428

Post by tcbasche »

Aeolun wrote: post #291
tcbasche wrote: post #196

Earlier quoted context omitted.

Yeah why should I treat anything this company does with any level of seriousness? Why should anyone? It's lucky it's just some online scoreboard because I'm sure as shit this stuff has happened before with more critical systems and it scares the hell out of me that engineers are fine blaming "gremlins" instead of taking responsibility for their own incompetence.

> taking responsibility for their own incompetence. I think they’re doing that with this post? At least I find it hard to imagine myself writing down that I’d drunk a few glasses of wine and dropped the production database. You cannot expect all engineers to be fully versed in the vagarities of database administration. Especially if they’re the only ones working on something.

Not really, they blamed 'complexity' and 'computer gremlins' rather than admitting that, perhaps, they made a shitty mistake.

> It’s a function that deletes the local database and creates all the required tables from scratch

Why would anyone have this? It's just dumb and embarrassing

Re: We deleted the production database by accident

#429

Post by bromuro »

skrebbel wrote: post #240

I'm appalled at the way some people here receive an honest postmortem of a human fuck-up. The top 3 comments, as I write this, can be summarized as "no, it's your fault and you're stupid for making the fault". This is not good! We don't want to scare people into writing less of these. We want to encourage people to write more of them. An MBA style "due to a human error, we lost a day of your data, we're tremendously…

I think you should wait 10+ hours to read different kind of comments on HN.

For example, if i open the comments about a “14 hours ago” post, I usually see a top comment about other comments (like yours).

I then feel so out of the loop because i don’t see the “commenters” your are referring too - so the thread that follows seem off topic to me.

Re: We deleted the production database by accident

#430

Post by mr_toad »

mrmonkeyman wrote: post #352
qz2 wrote: post #341

Earlier quoted context omitted.

I think my point is probably being misunderstood and that is my fault for explaining it poorly. See I fucked up :) The fear I speak of is a personal barrier which is lacking in a lot of people. They can sleep quite happily at night knowing they did a shitty job and it's going to explode down the line. It's not their problem. They don't care. I can't do that. Even if there are no direct consequences for me. This is no…

HN likes to downplay this, apparantly, but not everything can be boiled down to bureaucracy. Yes, medical professionals use checklists. They also have a harsh and very unforgiving culture that fosters craftsmanship and values professionalism above all else. You see this in other high-stakes professions too. You cannot just take the checklist and ignore the relentless focus on quality, the feelings of personal failure…

> medical professionals

operate in an environment where they don’t fully understand the systems they’re working with (human biology still has many unknowns), and many mistakes are irreversible.

If you look at the worst performing IT departments, they suffer from the same problems: they don’t fully understand how their systems work, and they lack easy ways to reverse changes.

Post reply on HN