Unrelated to the outage, but I'm curious nonetheless: would it be possible to hook up Tarsnap's encryption software to a Dropbox folder? I'm not sure if it even makes sense to use Tarsnap for this, but I'd love to have an easy setup that allows me to use Dropbox's servers but only let them see encrypted data so they can't snoop.
Tarsnap outage postmortem
121–130 of 338 posts
Re: Tarsnap outage postmortem
#122Earlier quoted context omitted.
>Far be it from me to tell anyone how to write software, but why build a database on top of S3 when you can just chuck the metadata into RDS with however much replication you want? Cost and reliability? * Using S3 as a simple database is generally going to be much cheaper than RDS. * If you turn on point in time restore, then losing data stored in S3 is not a possibility worth worrying about on a practical level for…
>Cost It's a bad trade. Thousands of hours of a high human capital computer scientist vs. a few tens of dollars a month for RDS. >Reliability Empirically false: none of this would have happened if Tarsnap used Postgres instead of a home-spun database.
>> Cost
> It's a bad trade.
Maybe. But that's the reason. You never acknowledged that advantage in your question so it needed to be emphasized
Re: Tarsnap outage postmortem
#123Unrelated to the outage, but I'm curious nonetheless: would it be possible to hook up Tarsnap's encryption software to a Dropbox folder? I'm not sure if it even makes sense to use Tarsnap for this, but I'd love to have an easy setup that allows me to use Dropbox's servers but only let them see encrypted data so they can't snoop.
You probably want something like https://cryptomator.org/
Re: Tarsnap outage postmortem
#124Earlier quoted context omitted.
>Cost It's a bad trade. Thousands of hours of a high human capital computer scientist vs. a few tens of dollars a month for RDS. >Reliability Empirically false: none of this would have happened if Tarsnap used Postgres instead of a home-spun database.
>> Cost > It's a bad trade. Maybe. But that's the reason. You never acknowledged that advantage in your question so it needed to be emphasized
It never occurred to me that anyone would need it explained to them that RDS is cheaper than the time of any software engineer.
The opportunity cost of building your own database is 10,000x the cost of running RDS for a year.
Re: Tarsnap outage postmortem
#125blinks Ok, I really wasn't expecting this to land at the top of HN. I'd love to stick around to answer any questions people have, but it's 10PM and my toddler decided to go to bed at 5PM... so if I'm lucky I can get about 4 hours of sleep before she decides that it's time to get up. I'll check in and answer questions in the morning.
Are you gonna switch to us-east-2?
Re: Tarsnap outage postmortem
#126(caveat: I may be running on old tarsnap company info but) I must say, the ONLY thing that has ever made me shy away from seriously using tarsnap was the prospect of an unexpected Colin Percival outage. i.e. key person risk. I'm guessing I'm not alone in this.
I mean, if you are on HN, you will probably learn of a Colin outage within 24 hours, so practically speaking you would really only have a problem if your primary data storage, Tarsnap, and Colin all failed in the same 24 hour window or so before you had time to switch to a new backup provider.
Re: Tarsnap outage postmortem
#127Earlier quoted context omitted.
Why would I use your service over restic? God bless you Colin, but reading this, it appears you're the only one in charge of the infrastructure for this service. I'm glad you're clear about no SLA, but this seems like a big liability between me and my backups.
tarsnap natively protects against inadvertent or malicious deletion or corruption — old tarsnap backups are immutablez The low-cost competitors (restic, borg, etc) seem to have this feature as an afterthought, and they make it surprisingly difficult. (FWIW, S3 can be somewhat straightforwardly configured so that old data is effectively immutable. Google Cloud Storage’s similarly named versioning feature appears to be…
Yep, S3 is reasonably easy to configure for immutability. I personally use restic to send (encrypted) blobs to https://www.borgbase.com which has append-only mode and monitoring to warn me if some backups didn't happen.
Re: Tarsnap outage postmortem
#128blinks Ok, I really wasn't expecting this to land at the top of HN. I'd love to stick around to answer any questions people have, but it's 10PM and my toddler decided to go to bed at 5PM... so if I'm lucky I can get about 4 hours of sleep before she decides that it's time to get up. I'll check in and answer questions in the morning.
Thank you for the post-mortem Colin and I hope you get some sleep!
Re: Tarsnap outage postmortem
#129>The process of recovering the EC2 instance state consists of two steps: First, reading all of the metadata headers from S3; and second, "replaying" all of those operations locally. (These cannot be performed at the same time, since the use of log-structured storage means that log entries are "rewritten" to free up storage when data is deleted; log entries contain sequence numbers to allow them to be replayed in the…
Forget about software or data architecture. S3 is the most reliable data storage mechanism in the world, and insanely simpler than a relational database. There is no operational failure mode to S3, other than "region went down". There is no instance to go down, no replication to fail, no worry about whether there's enough capacity for writes or too many connections, no thought to a schema, no migrations to manage, no storage to grow, no logs to rotate, no software to upgrade on a maintenance window. Plus S3 is versioned, has multiple kinds of access control built in, is a protocol supported by many vendors and open source projects, and is (on AWS) strongly read-after-write consistent. I would also argue (though I don't have figures) that it's faster than RDS. Almost guaranteed it's cheaper. And it's integrated into many services in AWS making it easier to build more functionality into the system without programming.
On a less technical note: Always avoid the fancy option when it makes sense. (From a veteran of building and maintaining large scale high performance high availability systems)
Re: Tarsnap outage postmortem
#130Earlier quoted context omitted.
If the data is super important you should be setting on two different providers anyways for backups.
Honestly, whose data isn't "super important"?. All my data is super important. Even the crap I just throw on my Google drive. I want to keep it. What is this mythical unimportant data that people still want to back up?
The stuff I don't want to fuck around searching re-downloading from torrent for example.