Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

I Accidentally Deleted All Our Data

taylor.fausak.me

Post reply on HN
Print view View original

41–50 of 78 posts

Re: I Accidentally Deleted All Our Data

#41

Post by gojomo »

taylorfausak wrote: post #24
skurry wrote: post #23

I wonder if it's possible that it was a MongoDB glitch and not a user error. From what I've read, it's notorious for randomly losing records. Maybe the loop save triggered it?

I definitely don't attribute this to MongoDB (or MongoEngine or pymongo or Django). I haven't seen it lose any of our documents to date. Silently dropping documents when saving them without making any changes would be a huge problem. It's astronomically unlikely.

Maybe something in the Mongo-ORM layer went wrong, so you were iterating over empty models, and then saving them back. Could that have created the absences seen later? (The way you talk about your client automatically fixing things on the load/save cycle suggests you might have some magic there.)

It just seems hard to believe you confused 'save' and 'delete'...

Re: I Accidentally Deleted All Our Data

#42

Post by mikeryan »

taylorfausak wrote: post #9
Pewpewarrows wrote: post #7

Happy to hear that you had backups and not too much data was lost because you caught it quickly. But besides the obvious takeaway of always having recent, tested backups, I think it's also important to get out of the habit of ever using a REPL (or SSH for that matter) to connect to production servers. Write a one-off script, document when and why you made it, check it into your VCS of choice, and run it against the t…

I usually write my scripts in a temporary file and run them through Django with something like: $ python manage.py shell In this instance, I didn't even consider that what I was doing could be catastrophic. But you're right, you want to avoid doing things manually. In the future, I'll track all the scripts I run. And thanks for the tip about Fabric, I'll check it out.

The advice isn't about how you do the script.

You should never, ever be running test scripts and code against a production db.

Re: I Accidentally Deleted All Our Data

#43

Post by jrockway »

joevandyk wrote: post #35
jrockway wrote: post #14

Earlier quoted context omitted.

I do this from time to time. But I also type "BEGIN" before anything else.

Watch out for database locks if you do this. I once opened a transaction, did an ALTER TABLE, and the site hung until I either committed or rolled back the transaction.

Yeah. Changing the schema is not something I'd do with code running against the database.

Also, my database typically refuses to start a transaction if the entire database is locked, and my application handles failing transactions by waiting a while and retrying.

Re: I Accidentally Deleted All Our Data

#44

Post by nirvdrum »

taylorfausak wrote: post #22
latchkey wrote: post #17

"We had a backup from earlier in the week" week? That seems kind of off. I'd suggest setting up backups more often than that, especially for production data. Sorry to hear about that. I destroyed a drive once a long time ago (~1994) as well. Happens to the best of us.

Because of this, we're going to back up parts of our database more frequently. A lot of our data isn't mission critical, so it wouldn't make sense to do daily backups for it. Other things, like family accounts, obviously are. We're going to back those up daily from here on out.

Is making a backup of everything that costly? My experience is you'll screw something up trying to be selective. With storage being so cheap these days, it's hardly worth the risk.

I don't use mongo so it's not 100% relevant, but I found setting up a postgres slave and running my backups from that is a nice solution. There's no additional load on my master and I just dump the backups into S3.

Re: I Accidentally Deleted All Our Data

#45

Post by taylorfausak »

gojomo wrote: post #41
taylorfausak wrote: post #24

Earlier quoted context omitted.

I definitely don't attribute this to MongoDB (or MongoEngine or pymongo or Django). I haven't seen it lose any of our documents to date. Silently dropping documents when saving them without making any changes would be a huge problem. It's astronomically unlikely.

Maybe something in the Mongo-ORM layer went wrong, so you were iterating over empty models, and then saving them back. Could that have created the absences seen later? (The way you talk about your client automatically fixing things on the load/save cycle suggests you might have some magic there.) It just seems hard to believe you confused 'save' and 'delete'...

I'm pretty sure nothing fishy went on behind the scenes. The client "automatically" fixes things through our API when it tries to log in because it will create an account if it doesn't exist.

I, too, have a hard time believing I confused "save" and "delete", but I'm reasonably sure that's what happened. I think I misused Python's shell history. I'll bet I hit the up arrow, which resurrected "family.delete()" instead of "family.save()".

Re: I Accidentally Deleted All Our Data

#46

Post by jrockway »

joevandyk wrote: post #37
drivebyacct2 wrote: post #13

Who works with a prod database in an interactive interpreter? My eyes are literally wide in shock.

It depends, if you're working with a smaller database with a smaller team with a smaller client base, there's not as much harm. There could possibly be more harm in having more ceremony for doing simple things.

I disagree with this. Making a script and testing it against a copy of the database is a good idea in 100% of cases. The only reason you don't do that is because you're lazy and stupid. (I've done this because I'm lazy and stupid.)

When you defeat a safety interlock system, expect to be injured.

Re: I Accidentally Deleted All Our Data

#47

Post by tripzilch »

Wowwww! Great to hear you had backups and could mostly restore your loss though!

You said you were working from the interactive shell so you didn't have history? Now I always use IPython, which does keep a history (which I set to very long), isn't there some way you can config the default Python interactive shell to keep a history as well? Sounds like that would be useful for all sorts of unexpected things. Otherwise, get IPython, it's really good, but still a plain interactive shell only with extra fancy features.

Glad to hear it turned out mostly all right though. I always feel for these data-loss stories because I can imagine what it'd be like ... that sinking feeling. Brrrr.

Re: I Accidentally Deleted All Our Data

#48

Post by taylorfausak »

tripzilch wrote: post #47

Wowwww! Great to hear you had backups and could mostly restore your loss though! You said you were working from the interactive shell so you didn't have history? Now I always use IPython, which does keep a history (which I set to very long ), isn't there some way you can config the default Python interactive shell to keep a history as well? Sounds like that would be useful for all sorts of unexpected things. Otherwis…

Python's default interactive shell have history, but it only keeps it for the current session. Once you kill it, the history is gone forever. Once I realized what I'd done, I checked to see if Python keeps the history anywhere. It doesn't, but I found several wrapper scripts that do (and IPython).

Re: I Accidentally Deleted All Our Data

#49

Post by adavies42 »

> Unfortunately my screen session doesn’t have enough scrollback to see what I entered earlier in the day.

`defscrollback 100000` is probably the single most important thing in my .screenrc. And if you're working in production, you should really have `deflog on` too.

Re: I Accidentally Deleted All Our Data

#50

Post by sneak »

mootothemax wrote: post #33

Potential lifesaver for anyone using MySQL, you can start the client with --i-am-a-dummy to prevent DELETE or UPDATE statements without a WHERE clause: http://sql-info.de/mysql/notes/I-am-a-dummy.html

Why is this not the default, with an option to DISable?
Post reply on HN