Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

We deleted the production database by accident

keepthescore.co

Post reply on HN
Print view View original

121–130 of 456 posts

Re: We deleted the production database by accident

#121

Post by theamk »

sushshshsh wrote: post #84
michelpp wrote: post #83

> Computers are just too complex and there are days when the complexity gremlins win. I'm sorry for your data loss, but this is a false and dangerous conclusion to make. You can avoid this problem. There are good suggestions in this thread, but I suggest you use Postgres's permission system to REVOKE DROP action on production except for a very special user that can only be logged in by a human, never a script. And NE…

As a mid level developer contributing to various large corporate stacks, I would say the systems are too complex and it's too easy to break things in non obvious ways. Gone are the days of me just being able to run a simple script that accesses data read only an exports the result elsewhere as an output.

Why are these days gone?

Just a few weeks ago, I set up a read-only user for myself, and moved all modify permission to role one must explicitly assume. Really helped me with peace of mind while developing the simple scripts that access data read only. This was on our managed AWS RDS database,

Re: We deleted the production database by accident

#122

Post by emodendroket »

sushshshsh wrote: post #105
Nextgrid wrote: post #89

Earlier quoted context omitted.

This is why I am against the current trend of over-complicating stacks for political or marketing reasons. Every startup nowadays wants microservices and/or serverless and a mashup of dozens of different SaaS (some that can't easily be simulated locally) from day 1 while a "boring" monolithic app will get them running just fine.

Yup, 100% agree. It may be that you will eventually need an auto-scalable message queue and api gateway, but for most people a web server and csv will serve the first thousand customers

There is sense in not building more services than you need. But many folks end up finding it hard to break away from their monolith and then it becomes an albatross. Not sure how to account for that.

Re: We deleted the production database by accident

#123

Post by fma »

jzer0cool wrote: post #86

Could someone explain more what caused the prod wipe? The snip here indicates it is using a 'dev' credential (it is a different pass than prod right?) - how does a db connection occur at all?

Good catch. Wouldn't surprise me if there's 1 username/password for all their DB environments.

Re: We deleted the production database by accident

#124

Post by emodendroket »

bigbubba wrote: post #95
cle wrote: post #90

Earlier quoted context omitted.

No. Your systems and processes should protect you from doing something stupid, because we’ve all done stupid things. Most stupid things are done whilst sober. In this case there were like 10 relatively easy things that could have prevented this. Your ability to mentally compile and evaluate your code before you hit enter is not a reliable way to protect your production systems. Coding after drinking is probably not a…

You know it's totally feasible to make a car that won't turn on for drunk people. Should those systems be installed on all cars, in pursuit of creating systems that don't permit stupid actions? Maybe such a breathalyzer interlock could be installed on your workstation too. After all, your systems and processes should prevent you from stupid things.

Honestly not a bad idea to install the interlocks on all cars.

Re: We deleted the production database by accident

#125

Post by fma »

jlgaddis wrote: post #31

The wonderful thing about computers is that they do exactly what they are told to do. The worst thing about computers? They do exactly what they are told to do.

I do a lot of work with middle/high school students. Without fail someone would yell "Why is it doing x!"...to which my standard reply is "Because you told it to".

Re: We deleted the production database by accident

#126

Post by apta »

ethbr0 wrote: post #115
dnautics wrote: post #107

Earlier quoted context omitted.

Please don't use csv. At the very least use SQLite. But hosted sqls are probably the smart thing to do.

> Please don't use csv Could you elaborate? I'm interested in the specific reasons.

Reliability, atomic updates, roll backs, proper backups, history management, proper schema, etc.

Re: We deleted the production database by accident

#127

Post by netheril96 »

amingilani wrote: post #47

Happens to all of us. Once I required logs from the server. The log file was a few gigs and still in use. so I carefully duplicated it, grepped just the lines I needed into another file and downloaded the smaller file. During this operation, the server ran out of memory—presumably because of all the files I'd created—and before I know it I'd managed to crash 3 services and corrupted the database—which was also on thi…

Why does the DB get corrupted? Does ACID mean anything these days?

Re: We deleted the production database by accident

#128

Post by sushshshsh »

emodendroket wrote: post #122
sushshshsh wrote: post #105

Earlier quoted context omitted.

Yup, 100% agree. It may be that you will eventually need an auto-scalable message queue and api gateway, but for most people a web server and csv will serve the first thousand customers

There is sense in not building more services than you need. But many folks end up finding it hard to break away from their monolith and then it becomes an albatross. Not sure how to account for that.

parallel rewrites before the current prod system ever hits performance problems :)

Re: We deleted the production database by accident

#129

Post by candiddevmike »

john_moscow wrote: post #81

Just my 2 cents. I run a small software business that involves a few moderately-sized databases. The day I moved from a fully managed hosting to a Linux VPS, I have crontabbed a script like this to run several times a day: for db in `mysql [...] | grep [...]` do mysqldump [...] > $db.sql done git commit -a -m "Automatic backup" git push [backup server #1] git push [backup server #2] git push [backup server #3] git gc…

Anyone reading the above: please don't do this. Git is not made for database backups, use a real backup solution like WAL archiving or dump it into restic/borg. Your git repo will balloon at an astronomical rate, and I can't imagine why anyone would diff database backups like this.

Re: We deleted the production database by accident

#130

Post by ultrarunner »

TedDoesntTalk wrote: post #93
cle wrote: post #90

Earlier quoted context omitted.

No. Your systems and processes should protect you from doing something stupid, because we’ve all done stupid things. Most stupid things are done whilst sober. In this case there were like 10 relatively easy things that could have prevented this. Your ability to mentally compile and evaluate your code before you hit enter is not a reliable way to protect your production systems. Coding after drinking is probably not a…

> Coding after drinking is probably not a good idea I’ve done some of my most productive work this way. Not on production systems fortunately, and not in a long time.

Riding the Ballmer Peak is a precarious operation, but I simply cannot deny its occasional effectiveness.
Post reply on HN