181–190 of 456 posts
This happened to me (someone in my team) a while ago but with mongo. The production database was ssh-tunneled to the default port of the guys computer and he ran tests that cleaned the database first. Now... our scenario was such that we could NOT lose those 7 hours because each customer record lost meant $5000 usd penalty. What saved us is that I knew about the oplog (binlog in mysql) so after restoring the backup i…
Earlier quoted context omitted.
I am not a drinker myself (drink 1-3 times a year), but in the past I have coded while slightly buzzed on a few occasions. I could not believe the level of focus I had. I never investigated it further, but I'm pretty sure the effects of alcohol on our coding abilities is not nearly as bad as it affects our motor skills. Imo, fatigue is far worse.
When I was not yet a teetotaler, each time I was hitting my maths textbooks after a few drinks, I could not believe my level of focus, and everything was clear and obvious. Textbook pages were flying at a speed never seen. Of course the next day, when re-reading the same pages, I was always discovering that the previous day I had everything wrong, nothing was obvious, and all my reasoning when with alcohol was false…
Earlier quoted context omitted.
It really depends on your database size. This works just fine for ~300MB databases. Git gc takes pretty good care of the fluff and once every couple of years I reset the repository to prune the old snapshots. The big plus is that you can reuse your existing git infrastructure, so the marginal setup costs are minimal. You can always switch to a more specialized solution if the repository size starts bugging you, but d…
Git GC won't do anything here unless you're deleting commits or resetting the repo constantly. Every commit will keep piling up, and you will never prune anything like you would a traditional backup tool. The day you do decide to start pruning things, expect your computer to burst into flames as it struggles to rewrite the commit history! Using a real database backup solution isn't a premature optimization, it's basi…
We had several MySQL string columns as long text type in our database but they should have been varchar(255) or so. So I was assigned to convert these columns to their appropriate size.
Being the good developer I was, I decided to download a snapshot of the prod database locally and checked the maximum string length we had for each column via a script. Using this script it made a migration query that would alter column types to match their maximum used length keeping the minimum length as varchar (255).
I tested that migration and everything looked good, it passed code review and was run on prod. Soon after we start getting complaints from users that their old email texts have been truncated. I then realize the stupidity of the whole thing, the local dump of production database always wiped out many columns clean for privacy like the email body column. So the script thought it had max length of 0 and decided to convert the column to varchar(255).
I realize the whole thing may look incredibly stupid, that's only because the naming for db columns was in a foreign european language so I didn't know even know the semantics of each column.
Thankfully my seniors managed to restore that column and took the responsibility themselves since they had passed the review.
We still did fix those unusually large columns but this time by simple duplicate alter queries for each of those columns instead of using fancy scripts.
I think a valuable lesson was learned that day to not rely on hacky scripts just to reduce some duplicate code.
I now prefer clarity and explicitness when writing such scripts instead of trying to be too clever and automating everything.
Very unsettled by the flippancy of the entire article
I love this post. This sort of thing happens to everyone, most people just are not willing to be so open about it. I was once sshed to the production server, and was cleaning up some old files that got created by an errant script, one which file was '~'. So, to clean it up, I type `rm -rf ~`.
Didn't have a backup of it unfortunately, though thankfully there wasn't anything too critical in there. Mostly just lost a bunch of utility scripts and dotfiles. I feel like it's beneficial in the long run for everyone to make a mistake like this once early on in their career.
Earlier quoted context omitted.
Why does the DB get corrupted? Does ACID mean anything these days?
Not original poster, but up to 2010, default MySQL table type was MyISAM, which does not support transactions.
It can even fail while in the middle of a transaction commit.
So transactions won't fix this.
Yes, people are not perfect and computer systems are complex. Admit it and don't be so overconfident.
”Errare humanum est”, prepare your backups.