Live data from Hacker News

I Accidentally Deleted All Our Data

taylor.fausak.me

71–78 of 78 posts

Re: I Accidentally Deleted All Our Data

#71
post #58

Earlier quoted context omitted.

Yeah. Changing the schema is not something I'd do with code running against the database. Also, my database typically refuses to start a transaction if the entire database is locked, and my application handles failing transactions by waiting a while and retrying.

Why not?

Timeouts probably.

If you mean why is it a problem at all, it's usually because of database size. On any large scale deployment (ie you have at least a million users) schema modifications will take hours. The only way to do reliable schema modifications is to have extra capacity and do it in stages. Also, your forward changes have to be backwards compatible. (AKA you're not allowed to both add and remove a column at the same time.)

The way to do it is to take some of your slaves out of the request pool and run the alter tables on them. You do this many times depending on your available capacity. (You probably can't just rip out half your slaves, you probably need to do at least 3 batches.) After you've altered all your slaves you can promote one to master and take the master offline to do its own alter. Then you push the code changes to production and add the old master back into the pool as a slave once it's done its alter.

In this scenario you need 3x the time the alter takes. So if the alter takes 6-7 hours (common in mysql if you have a large-ish table) it's going to take you at least 18 hours before you can push your code that depends on a database change.

Doing this manually at scale instead of an automated deployment process is extremely risky and will almost certainly be screwed up often.

This is one of the main reasons people are hoping schemaless databases work out in practice.

Re: I Accidentally Deleted All Our Data

#72
I added a SAN LUN to a volume that turned out to be in use, and was storing our only copy of a 500GB (sic) backup database tracking 60k+ tapes. After laying out the partition on the volume someone noticed that the database disappeared, and we were able to revert the partition and get the data back. Ended up buying some pretty nice Cognac for shortening the life of few folks. No idea why they weren't backing up their database though.

Re: I Accidentally Deleted All Our Data

#73

Earlier quoted context omitted.

I do this from time to time. But I also type "BEGIN" before anything else.

Watch out for database locks if you do this. I once opened a transaction, did an ALTER TABLE, and the site hung until I either committed or rolled back the transaction.

A lot of us are guilty of a quick data-munging hack at the command line from time to time, but the idea of altering the schema from a REPL makes me want to cry.

Re: I Accidentally Deleted All Our Data

#74
post #71
post #58

Earlier quoted context omitted.

Why not?

Timeouts probably. If you mean why is it a problem at all, it's usually because of database size. On any large scale deployment (ie you have at least a million users) schema modifications will take hours . The only way to do reliable schema modifications is to have extra capacity and do it in stages. Also, your forward changes have to be backwards compatible. (AKA you're not allowed to both add and remove a column at…

Agreed on backwards compatible, but for application semantics; not types of schema changes.

There's no reason to not be allowed to both add and remove a column at the same time, or to merge and split whole tables. In your example, these kinds of changes would not be possible.

There's also no reason to not be able to run old and new code at the same time, or to revert a schema change.

With ChronicDB we reduced schema changes to:

  $ chd change -f upgrade_map mydb
Schemaless databases don't solve this, just as an instantaneous ALTER TABLE won't solve this.

Re: I Accidentally Deleted All Our Data

#75
post #52
post #50

Earlier quoted context omitted.

Why is this not the default, with an option to DISable?

Because it's against the SQL standard (not that MySQL cares very much for that anyway). Just like SELECT * FROM table returns all the rows, DELETE FROM table deletes all the rows.

Let me rephrase: Why isn't the default mode of the mysql command line client to diverge from the SQL standard in the name of safety, with a command line option to enable "100% standard" (aka unsafe) mode?

I understand the SQL standard, and I understand the behavior. Why is the behavior of the software defaulting to "fuck up your life" in deference to some book on a shelf? (Especially considering you could revert to "fuck up your life" mode easily with a flag.)

Re: I Accidentally Deleted All Our Data

#77
post #57

Earlier quoted context omitted.

I disagree with this. Making a script and testing it against a copy of the database is a good idea in 100% of cases. The only reason you don't do that is because you're lazy and stupid. (I've done this because I'm lazy and stupid.) When you defeat a safety interlock system, expect to be injured.

There is no efficient safety interlock system when modifying data. Even if confident your test passed, what happens when you later realize it hadn't, and at what cost?

No one said it'll work 100% of the time. But it most likely would have saved the OP a lot of pain. Just because something isn't a perfect solution, it doesn't mean it's not worth doing.

Re: I Accidentally Deleted All Our Data

#78
post #6

Although direct database update isn't recommended in SAP, still for some strange reasons it was needed in one of the projects to modify a wrong entry done by functional consultant. One of my fellow programmer was assigned this task and this is what she did - Update set = instead of Update set = where The code was executed on development server(thankfully) and it created a big mess. Full day work of 5 guys was lost. P…

Wait... how can hosing a development environment cause people to lose work? The entire point of the development environment is so that when (not if: when) you hose it, you don't experience much pain at all.
Post reply on HN