Companies tend to get really good at backups after events like this.
We deleted the production database by accident
161–170 of 456 posts
Re: We deleted the production database by accident
#162Earlier quoted context omitted.
parallel rewrites before the current prod system ever hits performance problems :)
Sounds expensive
But potentially within the budget of a business with a large base of proven customers.
While annoying technically, for early stage startups, performance problems caused by an overly large number of users are almost always a good problem to have and are a far rarer sight than startups that have over-architected their technical solution without the concomitant proven actual users.
Re: We deleted the production database by accident
#163Just my 2 cents. I run a small software business that involves a few moderately-sized databases. The day I moved from a fully managed hosting to a Linux VPS, I have crontabbed a script like this to run several times a day: for db in `mysql [...] | grep [...]` do mysqldump [...] > $db.sql done git commit -a -m "Automatic backup" git push [backup server #1] git push [backup server #2] git push [backup server #3] git gc…
I don't believe having a massive repo with backups would be the ideal solution. Couldn't you just upload the backup to an s3 bucket instead?
Re: We deleted the production database by accident
#164Earlier quoted context omitted.
As a mid level developer contributing to various large corporate stacks, I would say the systems are too complex and it's too easy to break things in non obvious ways. Gone are the days of me just being able to run a simple script that accesses data read only an exports the result elsewhere as an output.
This is why I am against the current trend of over-complicating stacks for political or marketing reasons. Every startup nowadays wants microservices and/or serverless and a mashup of dozens of different SaaS (some that can't easily be simulated locally) from day 1 while a "boring" monolithic app will get them running just fine.
What? I thought that was just the opposite. The advantage of serverless is that I pay AWS to make backups so I don't have to. I mean under time pressure if it do it myself I'll skip making backups, setting permissions perfectly, and making sure I can actually restore those backups. If I go with a microservice, the whole point is they already do those things for me. No?
Re: We deleted the production database by accident
#165Earlier quoted context omitted.
It really depends on your database size. This works just fine for ~300MB databases. Git gc takes pretty good care of the fluff and once every couple of years I reset the repository to prune the old snapshots. The big plus is that you can reuse your existing git infrastructure, so the marginal setup costs are minimal. You can always switch to a more specialized solution if the repository size starts bugging you, but d…
Git GC won't do anything here unless you're deleting commits or resetting the repo constantly. Every commit will keep piling up, and you will never prune anything like you would a traditional backup tool. The day you do decide to start pruning things, expect your computer to burst into flames as it struggles to rewrite the commit history! Using a real database backup solution isn't a premature optimization, it's basi…
[deleted]
Re: We deleted the production database by accident
#166Earlier quoted context omitted.
> Coding after drinking is probably not a good idea I’ve done some of my most productive work this way. Not on production systems fortunately, and not in a long time.
Riding the Ballmer Peak is a precarious operation, but I simply cannot deny its occasional effectiveness.
Well played, and it's always pleasant to find XKCD in one's search results. (And an on-topic one at that).
For ref: https://xkcd.com/323/
Re: We deleted the production database by accident
#167> Computers are just too complex and there are days when the complexity gremlins win. Wow. But then again it's not like programmers handle dangerous infrastructure like trucks, military rockets or nuclear power plants. Those are reserved for adults
Yep, I hate when I'm dealing with a system literally comprised of logic and a magic gremlin shows up to ruin my day.
Seems like they though a casual "everyman" type of explanation would suffice, but really who would trust them after this?
Re: We deleted the production database by accident
#168Related: Little Bobby Tables (https://xkcd.com/327/)
Re: We deleted the production database by accident
#169Earlier quoted context omitted.
It really depends on your database size. This works just fine for ~300MB databases. Git gc takes pretty good care of the fluff and once every couple of years I reset the repository to prune the old snapshots. The big plus is that you can reuse your existing git infrastructure, so the marginal setup costs are minimal. You can always switch to a more specialized solution if the repository size starts bugging you, but d…
Git GC won't do anything here unless you're deleting commits or resetting the repo constantly. Every commit will keep piling up, and you will never prune anything like you would a traditional backup tool. The day you do decide to start pruning things, expect your computer to burst into flames as it struggles to rewrite the commit history! Using a real database backup solution isn't a premature optimization, it's basi…
I haven't dug into too much detail, but doing git gc did have a noticeable effect on size and subsequent update performance. I assume, some temporary artifacts got consolidated.
Also resetting the repository once every 1-2 years, and keeping the old one for a while is fine for smaller setups.
Depending on your business size and the amount of resources you want to allocate towards "basic system administration", accomplishing the same task with fewer tools could have its advantages.
Re: We deleted the production database by accident
#170Sack whoever is responsible;