Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Tarsnap outage postmortem

mail.tarsnap.com

Post reply on HN
Print view View original

131–140 of 338 posts

Re: Tarsnap outage postmortem

#131

Post by ilyt »

mrweasel wrote: post #61
AceJohnny2 wrote: post #39

Earlier quoted context omitted.

It's not about suing, but defining expectations about how you can rely on a service. For example, my team has people across the world for HW bringup, so we can't allow our code hosting or CI to be down for more than a few hours. Of course, backups have different uptime requirements, but as for everything, it's a tradeoff between features, of which an SLA is one. Tarsnap's features are granularity of cost, reliability…

> It's not about suing, but defining expectations about how you can rely on a service. Meeeeh, my ISP cut of around 100+ fiber connections in my town and spend three weeks fixing it. My neighbor have business line, there's an SLA on those that among other things, require them if reestablish his connection within 3 - 5 hours. It took them over 500 hours, so that SLA is useless for anything but forcing compensations. T…

The "problem" is that no sane company will sign for any damage compensation on some cheapo few dollars a month service.

Re: Tarsnap outage postmortem

#132

Post by ilyt »

throwawaaarrgh wrote: post #129
zetalyrae wrote: post #95

>The process of recovering the EC2 instance state consists of two steps: First, reading all of the metadata headers from S3; and second, "replaying" all of those operations locally. (These cannot be performed at the same time, since the use of log-structured storage means that log entries are "rewritten" to free up storage when data is deleted; log entries contain sequence numbers to allow them to be replayed in the…

Forget about software or data architecture. S3 is the most reliable data storage mechanism in the world, and insanely simpler than a relational database. There is no operational failure mode to S3, other than "region went down". There is no instance to go down, no replication to fail, no worry about whether there's enough capacity for writes or too many connections, no thought to a schema, no migrations to manage, no…

The "fancy" option here is trying to act S3 to act like database instead of simple blob storage...

Re: Tarsnap outage postmortem

#133

Post by ilyt »

mekster wrote: post #29
RockRobotRock wrote: post #13

Aren't these storage prices absurd? Please let me know if I'm misunderstanding.

It's insane. Not sure how anyone can accept such a rip off pricing. Tarsnap : $0.25 / GB storage, $0.25 / GB bandwidth cost rsync.net : $0.015 / GB storage, no bandwidth cost s3 : $0.023 / GB storage, some complicated bandwidth pricing If tarsnap is built on top of s3, they're charging 10 times for the storage cost. Easy money from the uninformed?

> If tarsnap is built on top of s3, they're charging 10 times for the storage cost. Easy money from the uninformed?

That's pretty much any SaaS... look at the various log or metrics gathering solution, where you pay serious multipliers of what would cost to run same software on your own instance.

Re: Tarsnap outage postmortem

#134

Post by bombcar »

cperciva wrote: post #3

blinks Ok, I really wasn't expecting this to land at the top of HN. I'd love to stick around to answer any questions people have, but it's 10PM and my toddler decided to go to bed at 5PM... so if I'm lucky I can get about 4 hours of sleep before she decides that it's time to get up. I'll check in and answer questions in the morning.

Time to get your toddler providing round-the-clock support! ;)

Have been having some luck reading https://www.amazon.com/No-Cry-Sleep-Solution-Toddlers-Presch... - available everywhere libraries (blockbuster for books!) are found.

Re: Tarsnap outage postmortem

#135

Post by ilyt »

gnfargbl wrote: post #63
mekster wrote: post #29

Earlier quoted context omitted.

It's insane. Not sure how anyone can accept such a rip off pricing. Tarsnap : $0.25 / GB storage, $0.25 / GB bandwidth cost rsync.net : $0.015 / GB storage, no bandwidth cost s3 : $0.023 / GB storage, some complicated bandwidth pricing If tarsnap is built on top of s3, they're charging 10 times for the storage cost. Easy money from the uninformed?

If you are primarily cost-driven, you missed one: Backblaze: $0.005 / GB storage, $0.01 / GB download.

You can send your backups to backblaze and S3 and it still would be cheaper

Re: Tarsnap outage postmortem

#136

Post by koolba »

saalweachter wrote: post #126
verytrivial wrote: post #60

(caveat: I may be running on old tarsnap company info but) I must say, the ONLY thing that has ever made me shy away from seriously using tarsnap was the prospect of an unexpected Colin Percival outage. i.e. key person risk. I'm guessing I'm not alone in this.

I mean, if you are on HN, you will probably learn of a Colin outage within 24 hours, so practically speaking you would really only have a problem if your primary data storage, Tarsnap, and Colin all failed in the same 24 hour window or so before you had time to switch to a new backup provider.

Pretty sure his brother works on tarsnap too.

They should take separate buses to ______.

Re: Tarsnap outage postmortem

#137

Post by bombcar »

verytrivial wrote: post #60

(caveat: I may be running on old tarsnap company info but) I must say, the ONLY thing that has ever made me shy away from seriously using tarsnap was the prospect of an unexpected Colin Percival outage. i.e. key person risk. I'm guessing I'm not alone in this.

I would never consider a backup provider to be more reliable than that, because if you depend on it, it will fail you at the hardest time.

Better to have multiple layers of backup, of which tarsnap and friends are only one, and verify regularly.

Re: Tarsnap outage postmortem

#138

Post by zetalyrae »

throwawaaarrgh wrote: post #129
zetalyrae wrote: post #95

>The process of recovering the EC2 instance state consists of two steps: First, reading all of the metadata headers from S3; and second, "replaying" all of those operations locally. (These cannot be performed at the same time, since the use of log-structured storage means that log entries are "rewritten" to free up storage when data is deleted; log entries contain sequence numbers to allow them to be replayed in the…

Forget about software or data architecture. S3 is the most reliable data storage mechanism in the world, and insanely simpler than a relational database. There is no operational failure mode to S3, other than "region went down". There is no instance to go down, no replication to fail, no worry about whether there's enough capacity for writes or too many connections, no thought to a schema, no migrations to manage, no…

>S3 is the most reliable data storage mechanism in the world

S3 is not the problem here. The problem is building a database on top of S3, and having to reimplement all the consistency, atomicity, transactions etc. on top.

>no thought to a schema, no migrations to manage

There is, in fact, always a schema. Some people choose to ignore it's there, to their detriment.

>Always avoid the fancy option when it makes sense.

It's not the 1980s. Postgres is not fancy, and Greenspunning it is a mistake.

>Almost guaranteed it's cheaper.

Cheaper than a 26-hour outage?

Re: Tarsnap outage postmortem

#139

Post by throwawaaarrgh »

ilyt wrote: post #132
throwawaaarrgh wrote: post #129

Earlier quoted context omitted.

Forget about software or data architecture. S3 is the most reliable data storage mechanism in the world, and insanely simpler than a relational database. There is no operational failure mode to S3, other than "region went down". There is no instance to go down, no replication to fail, no worry about whether there's enough capacity for writes or too many connections, no thought to a schema, no migrations to manage, no…

The "fancy" option here is trying to act S3 to act like database instead of simple blob storage...

Fancy would be writing an application to talk to RDS, creating an RDS instance, creating a database, creating whatever IAM link is needed for auth into the db so you don't need a second set of credentials, creating a schema, creating columns with different data types, and then modifying the application to handle edge cases for the different data types, logic to insert, update, delete rows, select items, yadda yadda yadda.

Dumb is `aws s3 cp` and being done in 5 minutes.

Re: Tarsnap outage postmortem

#140

Post by rssoconnor »

patrec wrote: post #87
gnfargbl wrote: post #58

Earlier quoted context omitted.

The prices are absurdly high if your use-case is storage of large volumes of data that regularly change. It wouldn't be sensible to use Tarsnap for that, and you probably want to use one of the bulk backup services instead. Tarsnap makes a lot of sense when you benefit from the encryption and (especially) de-duplication features that it offers. For me, all of my most important personal and business data, from multipl…

> Tarsnap makes a lot of sense when you benefit from the encryption and (especially) de-duplication features that it offers. Can you provide more detail why you think so? I don't believe there is any use case in which tarsnap makes sense, other than maybe some Plan-C backup solution which you fall back on in the highly unlikely event that neither Plan-A nor Plan-B worked. Concretely, what benefits does tarsnap offer…

I use tarsnap because the asymmetric crypto means I can give my cron job authorization to create backups, but it doesn't have authorization to read or delete(!) backups.

This ability is critical to prevent a compromised system from having its data wiped and having all backups wiped as well.

I haven't been able to figure out how to do this in any other system. But if someone has a tutorial, I am all ears.

Post reply on HN