Earlier quoted context omitted.
thats why I never have a DELETE in any query, only UPDATE and a state field (ie. deleted) performance advantage here as well since indexes aren't rebuilt and no table lock
Indexes cab certainly update. Pretty sure innodb does not table lock for delete either. Also from a privacy perspective you can't keep people's data around forever.
Instapaper's backup method
21–30 of 46 posts
Re: Instapaper's backup method
#22Anyone else find this scheme completely atrocious? 1. Relying on a home computer on the critical path for data backup and persistence for a business 2. Relying on a high latency, low quality networking path between the slave db and the 'home mac' rather than a more reliable link between two machines in a datacenter. 3. A poor persistence model for long lived backups 4. No easy way to programatically recover old backu…
Regarding point #1 - Marco's "Home Computer" is a Mac Pro (per other posts he's made) - it has Xeon proceesors, ECC RAM, etc. Much closer to a server than what you can pick up at Best Buy for $399.
Re: Instapaper's backup method
#23- Instantiate the backup (at its binlog position 259) - Replay the binlog from position 260 through 999 - Replay the binlog from position 1001 through 1200 And you’ll have a copy of the complete database if that destructive query had never happened.
This only works if the changes in positions 1001-1200 were unaffected by the undesired changes in position 1000. Seems rather unlikely to me, but maybe in the case of his particular schema it works out.
Re: Instapaper's backup method
#24Re: Instapaper's backup method
#25It seems unnecessarily exposed to an event affecting Marco's home - fire, burglary, natural disaster etc. It would appear more prudent to back up to a cloud location. Either, as he mentions, S3, or a VPS somewhere.
Re: Instapaper's backup method
#26I don't think backing up the entire db to a laptop is a good idea, since laptops can get both lost and stolen. As somebody who uses the service, I am not super-comfortable with knowing that a full copy of my account and everything I save is sitting on a laptop somewhere. It would be much better if these dumps were made to S3, or somewhere else that is actually in a secure datacenter (and a step that includes the word…
It's not explicitly stated in the article, but in the tweet that started it all[1], he mentioned it was a Mac Pro, rather than a laptop. So that's somewhat less likely to be stolen than a laptop that is taken out of the house regularly. That said, I agree with you, and I hope it's at least encrypted. [1] http://twitter.com/#!/marcoarment/status/6035374438621184
there is a reason datacenters were built
thinking about this after I left my comment, having all that data on your local machine is just crazy - you are one browser exploit or break-in away from having it fall into somebody elses hands. It isn't professional for a web service to be doing this - esp one that is now charging some customers.
Re: Instapaper's backup method
#27Earlier quoted context omitted.
thats why I never have a DELETE in any query, only UPDATE and a state field (ie. deleted) performance advantage here as well since indexes aren't rebuilt and no table lock
Indexes cab certainly update. Pretty sure innodb does not table lock for delete either. Also from a privacy perspective you can't keep people's data around forever.
checkout what wordpress does (from wp-content/plugins/akismet/akismet.php):
if ( (mt_rand(1, 10) == 3) ) {
// WP 2.0: run this one time in ten
akismet_delete_old();
}
and then in that function, after the DELETE, is this: $wpdb->query("OPTIMIZE TABLE $wpdb->comments");
I am sure there are plenty of people out there having fun trying to work out why there tables suddenly lock and they see an optimize process running randomly. I am also sure it runs fine in their unit tests when they have 1 post and 2 comments.all this because they DELETE :) They have all comments, those that have been approved, those that are in moderation, and all spam, in the same table - so if they don't delete the table would become unmanagable, so it is the design at fault and the wrong solution. If you search source at github or somewhere similar, you will find projects with OPTIMIZE everywhere - solving a real problem entirely the wrong way
I got used to it because the advantages just far outweigh the disadvantages. Records don't disappear for malicious reasons or because of mistakes - you can purge records marked delete every 30 days with a background process, if you like - but I no longer, ever, type that keyword into an app.
Re: Instapaper's backup method
#28Re: Instapaper's backup method
#29Earlier quoted context omitted.
The cost of storage on Amazon is very low - roughly $3 per month for Marco's 22GB. I imagine most people running a company would have a separate corporate account linked to a credit card, so that personal circumstances have less of a major effect month to month.
It doesn't matter how much it costs. I still don't want to lose my backups due to financial circumstances. I would say a fire destroying multiple safe places to store a backup (i.e., leaving a flash drive or DVD at my partner's home) is a lot less likely than financial mishap.
Re: Instapaper's backup method
#30Anyone else find this scheme completely atrocious? 1. Relying on a home computer on the critical path for data backup and persistence for a business 2. Relying on a high latency, low quality networking path between the slave db and the 'home mac' rather than a more reliable link between two machines in a datacenter. 3. A poor persistence model for long lived backups 4. No easy way to programatically recover old backu…
You mean the story of magnolia or what do you mean?