31–40 of 78 posts
OUCH. Doesn't django let you check if models are valid without saving them? In rails I can do: Family.all.select {|f| !f.valid?}
Happy to hear that you had backups and not too much data was lost because you caught it quickly. But besides the obvious takeaway of always having recent, tested backups, I think it's also important to get out of the habit of ever using a REPL (or SSH for that matter) to connect to production servers. Write a one-off script, document when and why you made it, check it into your VCS of choice, and run it against the t…
I usually write my scripts in a temporary file and run them through Django with something like: $ python manage.py shell In this instance, I didn't even consider that what I was doing could be catastrophic. But you're right, you want to avoid doing things manually. In the future, I'll track all the scripts I run. And thanks for the tip about Fabric, I'll check it out.
Or, you know, have a staging server.
Who works with a prod database in an interactive interpreter? My eyes are literally wide in shock.
I do this from time to time. But I also type "BEGIN" before anything else.
I once opened a transaction, did an ALTER TABLE, and the site hung until I either committed or rolled back the transaction.
OUCH. Doesn't django let you check if models are valid without saving them? In rails I can do: Family.all.select {|f| !f.valid?}
Who works with a prod database in an interactive interpreter? My eyes are literally wide in shock.
"We had a backup from earlier in the week" week? That seems kind of off. I'd suggest setting up backups more often than that, especially for production data. Sorry to hear about that. I destroyed a drive once a long time ago (~1994) as well. Happens to the best of us.
Because of this, we're going to back up parts of our database more frequently. A lot of our data isn't mission critical, so it wouldn't make sense to do daily backups for it. Other things, like family accounts, obviously are. We're going to back those up daily from here on out.
Even non-mission critical data loss can be bad.
This allows you to go back in time to an earlier snapshot of your data, without having to do a full database backup.
Unfortunately, it's still a pain to setup, even with tools like https://github.com/greg2ndQuadrant/repmgr. Wish there was more convention over configuration.
(I use Heroku's WAL-E to backup the data to S3. https://github.com/heroku/WAL-E)
Earlier quoted context omitted.
I usually write my scripts in a temporary file and run them through Django with something like: $ python manage.py shell In this instance, I didn't even consider that what I was doing could be catastrophic. But you're right, you want to avoid doing things manually. In the future, I'll track all the scripts I run. And thanks for the tip about Fabric, I'll check it out.
For bonus points, clone your production db locally and test your script against it a few times first. Or, you know, have a staging server.
This is the reason why enterprises always use staging servers with test suites.
It is irrelevant if you use NoSQL, SQL, or magic, if you don't have development+staging+production platforms for a business reliant on an information system, you're doing it wrong and it will bite you hard eventually.