11–20 of 46 posts
It seems unnecessarily exposed to an event affecting Marco's home - fire, burglary, natural disaster etc. It would appear more prudent to back up to a cloud location. Either, as he mentions, S3, or a VPS somewhere.
It would be much better if these dumps were made to S3, or somewhere else that is actually in a secure datacenter (and a step that includes the word 'encryption').
on delicious, we had a thing that would serialize a user to disk for every day they were active. inactive users were not re-serialized. this let us have day-to-day backups of individual users. this was necessary when broken clients would delete all the user's items. so we could easily restore an individual user (or do a historical recovery.)
performance advantage here as well since indexes aren't rebuilt and no table lock
1. Relying on a home computer on the critical path for data backup and persistence for a business
2. Relying on a high latency, low quality networking path between the slave db and the 'home mac' rather than a more reliable link between two machines in a datacenter.
3. A poor persistence model for long lived backups
4. No easy way to programatically recover old backups
What's even more disturbing is that this isn't a new problem. Its not like we don't know how to backup databases. This solution seems very poorly though out.
I don't think backing up the entire db to a laptop is a good idea, since laptops can get both lost and stolen. As somebody who uses the service, I am not super-comfortable with knowing that a full copy of my account and everything I save is sitting on a laptop somewhere. It would be much better if these dumps were made to S3, or somewhere else that is actually in a secure datacenter (and a step that includes the word…
That said, I agree with you, and I hope it's at least encrypted.
[1] http://twitter.com/#!/marcoarment/status/6035374438621184
Anyone else find this scheme completely atrocious? 1. Relying on a home computer on the critical path for data backup and persistence for a business 2. Relying on a high latency, low quality networking path between the slave db and the 'home mac' rather than a more reliable link between two machines in a datacenter. 3. A poor persistence model for long lived backups 4. No easy way to programatically recover old backu…
Also, "gzip --rsyncable" increases the compressed size by only about 1%, but makes deduplication between successive compressed dump files possible.
(I cofounded SpiderOak.)
on delicious, we had a thing that would serialize a user to disk for every day they were active. inactive users were not re-serialized. this let us have day-to-day backups of individual users. this was necessary when broken clients would delete all the user's items. so we could easily restore an individual user (or do a historical recovery.)
thats why I never have a DELETE in any query, only UPDATE and a state field (ie. deleted) performance advantage here as well since indexes aren't rebuilt and no table lock
Also from a privacy perspective you can't keep people's data around forever.
That's really an amazing system. Super redundant. A relatively easy boost, which he briefly mentioned, would be to also store the data in S3. That should be easy enough to be automated, which could provide a a somewhat-reliable off-site backup. However, Instapaper has the benefit of a (relatively) small DB. 22GB isn't too bad.I don't know how well this would scale to a 222GB DB with proportionally higher usage rates.…
I'd call S3 super-reliable rather than somewhat-reliable: "Amazon S3 is designed to provide 99.999999999% durability of objects over a given year. This durability level corresponds to an average annual expected loss of 0.000000001% of objects. For example, if you store 10,000 objects with Amazon S3, you can on average expect to incur a loss of a single object once every 10,000,000 years. In addition, Amazon S3 is des…