We deleted the production database by accident • Hacker News Board

Re: We deleted the production database by accident

#191

Post by hannofcart » Sun, Oct 18, 2020, 6:27 AM UTC

bachmeier wrote: post #144

hannofcart wrote: post #46

It takes a great deal of integrity to admit that you deleted a database because you were mucking around in your infra after red wine. And it bodes well for your firm that that doesn't get you fired either. These things happen to the best of us but having dealt with it responsibly and honestly as a team is something you can be proud of IMO.

> And it bodes well for your firm that that doesn't get you fired either. Maybe there's more than one interpretation of "bodes well" but not knowing how to do the one thing customers were paying you to do isn't consistent with my definition. > having dealt with it responsibly and honestly as a team is something you can be proud of IMO. "We were drinking wine and deleted the database and now your data's gone LOL" is n…

Please re-read what I wrote. I was praising their honesty. Not egging them on to be more sloppy.

Re: We deleted the production database by accident

#192

Post by thih9 » Sun, Oct 18, 2020, 6:35 AM UTC

michelpp wrote: post #83

> Computers are just too complex and there are days when the complexity gremlins win. I'm sorry for your data loss, but this is a false and dangerous conclusion to make. You can avoid this problem. There are good suggestions in this thread, but I suggest you use Postgres's permission system to REVOKE DROP action on production except for a very special user that can only be logged in by a human, never a script. And NE…

> You can avoid this problem.

The article isn’t claiming that the problem is impossible to solve.

On the contrary: “However, we will figure out what went wrong and ensure that this particular error doesn’t happen again.”.

Re: We deleted the production database by accident

#193

Post by ikiris » Sun, Oct 18, 2020, 6:36 AM UTC

bsder wrote: post #174

michelpp wrote: post #83

> Computers are just too complex and there are days when the complexity gremlins win. I'm sorry for your data loss, but this is a false and dangerous conclusion to make. You can avoid this problem. There are good suggestions in this thread, but I suggest you use Postgres's permission system to REVOKE DROP action on production except for a very special user that can only be logged in by a human, never a script. And NE…

> You can avoid this problem. No, you can't. No matter how good you are, you can always "rm -rf" your world. Yes, we can make it harder, but, at the end of the day, some human, somewhere, has to pull the switch on the stuff that pushes to prod. You can clobber prod manually, or you accidentally write an erroneous script that clobbers prod. Either way--prod is toast. The word of the day is "backups".

excuse me, but no. this is harmful bullshit.

Yes, backups are vitally important, but no it is not possible to accidentally rm -rf with proper design.

It's possible to have the most dangerous credentials possible and still make it difficult to do catastrophic global changes. Hell it's my job to make sure this is the case.

Re: We deleted the production database by accident

#194

Post by Agentlien » Sun, Oct 18, 2020, 6:38 AM UTC

necovek wrote: post #176

WarOnPrivacy wrote: post #132

Earlier quoted context omitted.

Yes. I could finally start a failed interlock story blog.

Or a blog on being unable to drive your kid to an emergency room because you just finished a glass of wine over dinner. A problem with devices of that type is that they only test for a potential source of inability to drive safely. What we want is to test for an inability to drive safely. And while one is easy and might give some quick wins, the drawbacks scare me too much.

Being unable to take your car when your child needs to go to the ER would be terrible.

Actually getting in the car while under the influence such of stress and alcohol sounds worse.

I know someone who had a glass of wine just before her daughter needed to be brought to the hospital. This was just two days ago. She simply concluded she could not drive. Luckily, she was able to get a taxi.

Re: We deleted the production database by accident

#195

Post by danielh » Sun, Oct 18, 2020, 6:41 AM UTC

danellis wrote: post #77

> after a couple of glasses of red wine, we deleted the production database by accident > It’s tempting to blame the disaster on the couple of glasses of red wine. However, the function that wiped the database was written whilst sober. It was _written_ then, but you're still admitting to the world that your employees do work on production systems after they've been drinking. Since they were working so late, one might…

According to the about page, "the employees" consist of one guy working on it in his spare time.

Re: We deleted the production database by accident

#196

Post by tcbasche » Sun, Oct 18, 2020, 6:45 AM UTC

emodendroket wrote: post #119

This post is embarrassing. "yeah we were drinking and accidentally nuked the prod DB. Not sure why. Shit happens!" Who would read this and think they should trust this company? Any number of protections could have been taken to prevent this and production access in any state other than fully alert and attentive shouldn't happen unless it is absolutely necessary for emergency reasons

Yeah why should I treat anything this company does with any level of seriousness? Why should anyone?

It's lucky it's just some online scoreboard because I'm sure as shit this stuff has happened before with more critical systems and it scares the hell out of me that engineers are fine blaming "gremlins" instead of taking responsibility for their own incompetence.

Re: We deleted the production database by accident

#197

Post by jamil7 » Sun, Oct 18, 2020, 6:46 AM UTC

greggman3 wrote: post #164

Nextgrid wrote: post #89

Earlier quoted context omitted.

This is why I am against the current trend of over-complicating stacks for political or marketing reasons. Every startup nowadays wants microservices and/or serverless and a mashup of dozens of different SaaS (some that can't easily be simulated locally) from day 1 while a "boring" monolithic app will get them running just fine.

What? I thought that was just the opposite. The advantage of serverless is that I pay AWS to make backups so I don't have to. I mean under time pressure if it do it myself I'll skip making backups, setting permissions perfectly, and making sure I can actually restore those backups. If I go with a microservice, the whole point is they already do those things for me. No?

What does serverless have to do with making backups? Any managed database can do this for you. Microservices attempt to facilitate multiple teams working on a system independently. They’re mostly solving human problems not technical.

The grandparent comments point is a single person or team can deploy a monolith on herkou and avoid a huge amount of complexity. Especially in the beginning.

Re: We deleted the production database by accident

#198

Post by ivan_gammel » Sun, Oct 18, 2020, 6:49 AM UTC

Nextgrid wrote: post #89

sushshshsh wrote: post #84

Earlier quoted context omitted.

As a mid level developer contributing to various large corporate stacks, I would say the systems are too complex and it's too easy to break things in non obvious ways. Gone are the days of me just being able to run a simple script that accesses data read only an exports the result elsewhere as an output.

This is why I am against the current trend of over-complicating stacks for political or marketing reasons. Every startup nowadays wants microservices and/or serverless and a mashup of dozens of different SaaS (some that can't easily be simulated locally) from day 1 while a "boring" monolithic app will get them running just fine.

Microservices are easy to build and throw away these days. In startups time to market is more important than future investment in devops. In terms of speed of delivery they are not worse than monolithic architecture (also not better). For similar reasons SaaS is and must be the default way to build IT infrastructure, because it has much lower costs and time to deploy, compared to custom software development.

Re: We deleted the production database by accident

#199

Post by unnouinceput » Sun, Oct 18, 2020, 6:58 AM UTC

Quote: "Note that host is hardcoded to localhost. This means it should never connect to any machine other than the developer machine. Also: of course we use different passwords and users for development and production. We’re too tired to figure it out right now.

The gremlins won this time."

No they didn't. Instead one of your gremlins ran this function directly on the production machine. This isn't rocket science, just the common sense conclusion. Now it would be a good time to check those auditing logs / access logs you're suppose to have them enabled on said production machine.

Re: We deleted the production database by accident

#200

Post by hibbelig » Sun, Oct 18, 2020, 7:04 AM UTC

dnautics wrote: post #107

sushshshsh wrote: post #105

Earlier quoted context omitted.

Yup, 100% agree. It may be that you will eventually need an auto-scalable message queue and api gateway, but for most people a web server and csv will serve the first thousand customers

Please don't use csv. At the very least use SQLite. But hosted sqls are probably the smart thing to do.

The way I understand sushsh‘s suggestion is that CSV is an alternative to an API gateway and message queue. I don’t think the suggestion was to replace a database with CSV.