Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

We deleted the production database by accident

keepthescore.co

Post reply on HN
Print view View original

401–410 of 456 posts

Re: We deleted the production database by accident

#401

Post by thamer »

skytreader wrote: post #235
unnouinceput wrote: post #199

Quote: "Note that host is hardcoded to localhost. This means it should never connect to any machine other than the developer machine. Also: of course we use different passwords and users for development and production. We’re too tired to figure it out right now. The gremlins won this time." No they didn't. Instead one of your gremlins ran this function directly on the production machine. This isn't rocket science, ju…

> Instead one of your gremlins ran this function directly on the production machine. Exactly my first hypothesis too. But then keepthescore claims, > of course we use different passwords and users for development and production. How would this hypothesis explain that? --- Metadialogue: John Watson: "I deduce that someone changed the dev config source so that it uses the production config values." Sherlock Holmes: "My…

Agreed, it seems like most people making suggestions above are missing the point about credentials. The code they present explicitly references `config.DevelopmentConfig`:

    database = config.DevelopmentConfig.DB_DATABASE
    user = config.DevelopmentConfig.DB_USERNAME
    password = config.DevelopmentConfig.DB_PASSWORD
One way this could happen is if the objects under `config` were loaded from separate files, and the dev file was changed to a symlink to the prod file. So `config.DevelopmentConfig` always loads /opt/myapp/config/dev.cfg but a developer had dev.cfg -> prod.cfg and the prod credentials and connection details were loaded into `config.DevelopmentConfig`.

Just an idea.

Re: We deleted the production database by accident

#402

Post by macNchz »

rmrfrmrf wrote: post #387
macNchz wrote: post #60

Earlier quoted context omitted.

A likely culprit. Having worked on a bunch of early-stage products where best practices are a distant future dream, I’ve developed a few “seatbelt” habits I use to avoid these kinds of things. One of them is to always use a random high-number local port if I’m tunneling to a production service. Another is to change my terminal theme to a red background before connecting to anything in production...never want to click…

what's a situation where you'd be tunneling to a production service?

I think the most common reason I’ve had in the past was to connect to the RabbitMQ web admin while dealing with some emergent issue with task throughput, a common problem area with hacked together web apps that start to get real traffic. It’s also handy to be able to use a more advanced SQL client that’s not installed on the server (pgcli, emacs, etc) when digging around for something that’s causing errors in production.

Re: We deleted the production database by accident

#403

Post by xtracto »

rocqua wrote: post #257
vanviegen wrote: post #211

Earlier quoted context omitted.

> No one owned up to it, but had a pretty good idea who it was. That sounds like you're putting (some of) the blame on whoever misclicked. As opposed to everyone who has allowed this insanely dangerous situation to exist.

Misclicking is a tiny forgivable mistake. Not immediately calling up your boss to say "I fucked up big" is not a mistake, it is a conscious bad action.

This. The person that erased the database in my case came forward to me as soon as we realized what had happened. At that moment I was very happy it was an "inside job", it meant I could discard hacking.

As its said before: he made a mistake. The error was allowing the prod database to to be port forwarded from a non prod environment. As head of eng that was MY error. So I owned to it and we changed policies.

Re: We deleted the production database by accident

#404

Post by Nextgrid »

adwn wrote: post #347
tamrix wrote: post #225

Earlier quoted context omitted.

I think we're hitting peak tech. All this "technical" knowledge just dates itself in a year's time anyway. Eventually, you come to realise that the more tech you've got, the more problems you have. . Now developers spend more time googling errors and plugging in libraries and webservices together than writing any actual code. Sometimes I wish for a techless cloudless revolution when we just go back to the foundations…

> Sometimes I wish for a techless cloudless revolution when we just go back to the foundations of computers and is use plain text wherever possible. ... because software in the 60s/70s/80s was so reliable and bug-free?!

It most likely had less moving parts & failure modes than a modern microservice mess.

Re: We deleted the production database by accident

#405

Post by Nextgrid »

csomar wrote: post #320
Nextgrid wrote: post #89

Earlier quoted context omitted.

This is why I am against the current trend of over-complicating stacks for political or marketing reasons. Every startup nowadays wants microservices and/or serverless and a mashup of dozens of different SaaS (some that can't easily be simulated locally) from day 1 while a "boring" monolithic app will get them running just fine.

For the most part, we are not complicating stuff. Today's requirements are complicated. We used to operate from the commandline on a single processor. Now things are complicated: People expect a Web UI, High availability, integration with their phone, Email notification, 2FA authentication, and then you have things like SSL/HTTPS, Compliance, and you need to log the whole thing for errors or compliance or whatever. S…

All of these can be done just fine in a monolithic Django/Ruby/PHP/Java/etc app.

Re: We deleted the production database by accident

#406

Post by 0kl »

> Note that host is hardcoded to localhost. This means it should never connect to any machine other than the developer machine.

Just to help with the postmortem:

1) “localhost” is just a loopback to whatever machine you’re on

2) the user and pw are pulled from config

So someone was running this from the production server or had the production DB mapped to localhost and ran it with a production config for some reason (working with prod data maybe). The hard coding to localhost will only ensure that it works for the machine it’s called on - in this case the prod server.

Things you might do to avoid this in the future include a wide spread of things, the main recommendations I’d have are:

1) only put production artifacts on prod

2) limit developer access to prod data

Best of luck

Re: We deleted the production database by accident

#408

Post by vlovich123 »

janoc wrote: post #361
quietbritishjim wrote: post #324

Earlier quoted context omitted.

> Even when it was a suicidal pilot flying the plane into a mountain on purpose. Someone had to supervise him (there are two crew members in the cockpit for a reason), someone gave him a medical, there is automation in the cockpit that could have at least caused an alarm, etc. There was indeed a suicidal pilot that flew into a mountain, I'm not sure if you were deliberately referencing that specific time. In that cas…

I was referring specifically to the Germanwings incident. That pilot shouldn't have been in the cockpit to begin with - his eyesight was failing, he had mental problems (has been medically treated for suicidal tendencies), etc. This was not discovered nor identified, due to deficiencies in the system (doctors didn't have the duty to report this, he withheld the information from his employer, etc.) The issue with the…

Any examinations of whether or not the job itself has properties that might cause the medical issues?

Re: We deleted the production database by accident

#409

Post by ENOTTY »

t0mas88 wrote: post #243
skrebbel wrote: post #240

I'm appalled at the way some people here receive an honest postmortem of a human fuck-up. The top 3 comments, as I write this, can be summarized as "no, it's your fault and you're stupid for making the fault". This is not good! We don't want to scare people into writing less of these. We want to encourage people to write more of them. An MBA style "due to a human error, we lost a day of your data, we're tremendously…

The software sector needs a bit of aviation safety culture: 50 years ago the conclusion "pilot error" as the main cause was virtually banned from accident investigation. The new mindset is that any system or procedure where a single human error can cause an incident is a broken system. So the blame isn't on the human pressing the button, the problem is the button or procedure design being unsuitable. The result was a…

The idea that multiple failures must occur for catastrophic failure is found in certain parts of the computing community. https://en.wikipedia.org/wiki/Defense_in_depth_(computing)

Re: We deleted the production database by accident

#410

Post by cutemonster »

rocqua wrote: post #257
vanviegen wrote: post #211

Earlier quoted context omitted.

> No one owned up to it, but had a pretty good idea who it was. That sounds like you're putting (some of) the blame on whoever misclicked. As opposed to everyone who has allowed this insanely dangerous situation to exist.

Misclicking is a tiny forgivable mistake. Not immediately calling up your boss to say "I fucked up big" is not a mistake, it is a conscious bad action.

Another thought: the company culture and approach to hiring and firing, can cause people to try to hide mistakes, although they don't really want to?
Post reply on HN