Instapaper's backup method
marco.org
Instapaper's backup method
1–10 of 46 posts
Re: Instapaper's backup method
#2That's really an amazing system. Super redundant.
A relatively easy boost, which he briefly mentioned, would be to also store the data in S3. That should be easy enough to be automated, which could provide a a somewhat-reliable off-site backup.
However, Instapaper has the benefit of a (relatively) small DB. 22GB isn't too bad.I don't know how well this would scale to a 222GB DB with proportionally higher usage rates. It'd be possible, but it would have to be simplified, no?
Re: Instapaper's backup method
#3It seems unnecessarily exposed to an event affecting Marco's home - fire, burglary, natural disaster etc. It would appear more prudent to back up to a cloud location. Either, as he mentions, S3, or a VPS somewhere.
Re: Instapaper's backup method
#4It seems unnecessarily exposed to an event affecting Marco's home - fire, burglary, natural disaster etc. It would appear more prudent to back up to a cloud location. Either, as he mentions, S3, or a VPS somewhere.
The problem with backing up with S3 is that if you ever stop paying for S3 you lose your backups. If you get sick for a month and your bank balance goes negative, or the IRS takes control over your account for something, there go your backups. I find that to be way more scary and likely than losing all of my DVD backups.
Re: Instapaper's backup method
#5That's really an amazing system. Super redundant. A relatively easy boost, which he briefly mentioned, would be to also store the data in S3. That should be easy enough to be automated, which could provide a a somewhat-reliable off-site backup. However, Instapaper has the benefit of a (relatively) small DB. 22GB isn't too bad.I don't know how well this would scale to a 222GB DB with proportionally higher usage rates.…
I'd call S3 super-reliable rather than somewhat-reliable:
"Amazon S3 is designed to provide 99.999999999% durability of objects over a given year. This durability level corresponds to an average annual expected loss of 0.000000001% of objects. For example, if you store 10,000 objects with Amazon S3, you can on average expect to incur a loss of a single object once every 10,000,000 years. In addition, Amazon S3 is designed to sustain the concurrent loss of data in two facilities."
It's slow as a result...but that's the trade-off you're looking for in a backup.
Re: Instapaper's backup method
#6It seems unnecessarily exposed to an event affecting Marco's home - fire, burglary, natural disaster etc. It would appear more prudent to back up to a cloud location. Either, as he mentions, S3, or a VPS somewhere.
The problem with backing up with S3 is that if you ever stop paying for S3 you lose your backups. If you get sick for a month and your bank balance goes negative, or the IRS takes control over your account for something, there go your backups. I find that to be way more scary and likely than losing all of my DVD backups.
The cost of storage on Amazon is very low - roughly $3 per month for Marco's 22GB.
I imagine most people running a company would have a separate corporate account linked to a credit card, so that personal circumstances have less of a major effect month to month.
Re: Instapaper's backup method
#7on delicious, we had a thing that would serialize a user to disk for every day they were active. inactive users were not re-serialized.
this let us have day-to-day backups of individual users. this was necessary when broken clients would delete all the user's items. so we could easily restore an individual user (or do a historical recovery.)
Re: Instapaper's backup method
#8Earlier quoted context omitted.
The problem with backing up with S3 is that if you ever stop paying for S3 you lose your backups. If you get sick for a month and your bank balance goes negative, or the IRS takes control over your account for something, there go your backups. I find that to be way more scary and likely than losing all of my DVD backups.
The cost of storage on Amazon is very low - roughly $3 per month for Marco's 22GB. I imagine most people running a company would have a separate corporate account linked to a credit card, so that personal circumstances have less of a major effect month to month.
It doesn't matter how much it costs. I still don't want to lose my backups due to financial circumstances. I would say a fire destroying multiple safe places to store a backup (i.e., leaving a flash drive or DVD at my partner's home) is a lot less likely than financial mishap.
Re: Instapaper's backup method
#9Are the primary and backup DBs in the same data center? If so, how would you restore from an "unplanned event" there? I ask because I faced that situation once years ago, and very quickly learned that uploading 10's of GB of data from an offsite backup will keep your site offline for hours.
In the end I ended up _driving_ a copy of the DB over to a data center. Adding a slaved-replica in another location is pretty easy these days.
Re: Instapaper's backup method
#10Are the primary and backup DBs in the same data center? If so, how would you restore from an "unplanned event" there? I ask because I faced that situation once years ago, and very quickly learned that uploading 10's of GB of data from an offsite backup will keep your site offline for hours. In the end I ended up _driving_ a copy of the DB over to a data center. Adding a slaved-replica in another location is pretty ea…
"Never underestimate the bandwidth of a station wagon full of tapes hurtling down the highway" -- Andrew Tanenbaum