Live data from Hacker News

Instapaper's backup method

marco.org

21–30 of 46 posts

Re: Instapaper's backup method

#21
post #19
post #13

Earlier quoted context omitted.

thats why I never have a DELETE in any query, only UPDATE and a state field (ie. deleted) performance advantage here as well since indexes aren't rebuilt and no table lock

Indexes cab certainly update. Pretty sure innodb does not table lock for delete either. Also from a privacy perspective you can't keep people's data around forever.

Maybe this is overly pedantic, but you can do whatever you want with people's data, so long as you inform them of your policies and they agree to them.

Re: Instapaper's backup method

#22
post #17
post #14

Anyone else find this scheme completely atrocious? 1. Relying on a home computer on the critical path for data backup and persistence for a business 2. Relying on a high latency, low quality networking path between the slave db and the 'home mac' rather than a more reliable link between two machines in a datacenter. 3. A poor persistence model for long lived backups 4. No easy way to programatically recover old backu…

Regarding point #1 - Marco's "Home Computer" is a Mac Pro (per other posts he's made) - it has Xeon proceesors, ECC RAM, etc. Much closer to a server than what you can pick up at Best Buy for $399.

It's not about performance nor price, but conditions in which the machine operates. Many servers used nowadays are cheaper than high-end desktop machines.

Re: Instapaper's backup method

#23
The scenario he presents of being able to recover from an unintentionally broad delete or update query would seem to only work in the simplest of databases. He says:

- Instantiate the backup (at its binlog position 259) - Replay the binlog from position 260 through 999 - Replay the binlog from position 1001 through 1200 And you’ll have a copy of the complete database if that destructive query had never happened.

This only works if the changes in positions 1001-1200 were unaffected by the undesired changes in position 1000. Seems rather unlikely to me, but maybe in the case of his particular schema it works out.

Re: Instapaper's backup method

#25
post #3

It seems unnecessarily exposed to an event affecting Marco's home - fire, burglary, natural disaster etc. It would appear more prudent to back up to a cloud location. Either, as he mentions, S3, or a VPS somewhere.

I agree with that.. Kinda expected a better setup for a real business.

Re: Instapaper's backup method

#26
post #12

I don't think backing up the entire db to a laptop is a good idea, since laptops can get both lost and stolen. As somebody who uses the service, I am not super-comfortable with knowing that a full copy of my account and everything I save is sitting on a laptop somewhere. It would be much better if these dumps were made to S3, or somewhere else that is actually in a secure datacenter (and a step that includes the word…

It's not explicitly stated in the article, but in the tweet that started it all[1], he mentioned it was a Mac Pro, rather than a laptop. So that's somewhat less likely to be stolen than a laptop that is taken out of the house regularly. That said, I agree with you, and I hope it's at least encrypted. [1] http://twitter.com/#!/marcoarment/status/6035374438621184

ye not much better

there is a reason datacenters were built

thinking about this after I left my comment, having all that data on your local machine is just crazy - you are one browser exploit or break-in away from having it fall into somebody elses hands. It isn't professional for a web service to be doing this - esp one that is now charging some customers.

Re: Instapaper's backup method

#27
post #19
post #13

Earlier quoted context omitted.

thats why I never have a DELETE in any query, only UPDATE and a state field (ie. deleted) performance advantage here as well since indexes aren't rebuilt and no table lock

Indexes cab certainly update. Pretty sure innodb does not table lock for delete either. Also from a privacy perspective you can't keep people's data around forever.

even with innodb you will still find yourself running optimize.

checkout what wordpress does (from wp-content/plugins/akismet/akismet.php):

  if ( (mt_rand(1, 10) == 3) ) {
    // WP 2.0: run this one time in ten
    akismet_delete_old();
  }
and then in that function, after the DELETE, is this:

  $wpdb->query("OPTIMIZE TABLE $wpdb->comments");
I am sure there are plenty of people out there having fun trying to work out why there tables suddenly lock and they see an optimize process running randomly. I am also sure it runs fine in their unit tests when they have 1 post and 2 comments.

all this because they DELETE :) They have all comments, those that have been approved, those that are in moderation, and all spam, in the same table - so if they don't delete the table would become unmanagable, so it is the design at fault and the wrong solution. If you search source at github or somewhere similar, you will find projects with OPTIMIZE everywhere - solving a real problem entirely the wrong way

I got used to it because the advantages just far outweigh the disadvantages. Records don't disappear for malicious reasons or because of mistakes - you can purge records marked delete every 30 days with a background process, if you like - but I no longer, ever, type that keyword into an app.

Re: Instapaper's backup method

#28
Would the people who are upset that Marco is using his 'home' computer feel the same if he instead said it was at his office? Offices get broken into or have equipment stolen too - I'm not sure why people think this is so irresponsible given that he works from home now.

Re: Instapaper's backup method

#29
post #8

Earlier quoted context omitted.

The cost of storage on Amazon is very low - roughly $3 per month for Marco's 22GB. I imagine most people running a company would have a separate corporate account linked to a credit card, so that personal circumstances have less of a major effect month to month.

It doesn't matter how much it costs. I still don't want to lose my backups due to financial circumstances. I would say a fire destroying multiple safe places to store a backup (i.e., leaving a flash drive or DVD at my partner's home) is a lot less likely than financial mishap.

The solution to this then seems to be that Amazon should allow for you to prepay for AWS credit so one does not need to worry about their bank accounts suddenly being frozen, or some other mishap, just that they have X months of runaway in AWS credit for typical S3 charges.

Re: Instapaper's backup method

#30
post #14

Anyone else find this scheme completely atrocious? 1. Relying on a home computer on the critical path for data backup and persistence for a business 2. Relying on a high latency, low quality networking path between the slave db and the 'home mac' rather than a more reliable link between two machines in a datacenter. 3. A poor persistence model for long lived backups 4. No easy way to programatically recover old backu…

> even more disturbing is that this isn't a new problem

You mean the story of magnolia or what do you mean?

Post reply on HN