Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Tarsnap outage postmortem

mail.tarsnap.com

Post reply on HN
Print view View original

161–170 of 338 posts

Re: Tarsnap outage postmortem

#161

Post by ivanhoe »

VWWHFSfQ wrote: post #120
k8sToGo wrote: post #99

Earlier quoted context omitted.

If the data is super important you should be setting on two different providers anyways for backups.

Honestly, whose data isn't "super important"?. All my data is super important. Even the crap I just throw on my Google drive. I want to keep it. What is this mythical unimportant data that people still want to back up?

Anything that you stashed just for convenience, but you could re-download or re-generate it if really needed, or simply live without it... frankly, like 90% of stuff on my disks fall in the category "I'll read/view it one day", which in reality I'll probably never have time or patience to open ever again.

Re: Tarsnap outage postmortem

#162

Post by throwaway290 »

ilyt wrote: post #131
mrweasel wrote: post #61

Earlier quoted context omitted.

> It's not about suing, but defining expectations about how you can rely on a service. Meeeeh, my ISP cut of around 100+ fiber connections in my town and spend three weeks fixing it. My neighbor have business line, there's an SLA on those that among other things, require them if reestablish his connection within 3 - 5 hours. It took them over 500 hours, so that SLA is useless for anything but forcing compensations. T…

The "problem" is that no sane company will sign for any damage compensation on some cheapo few dollars a month service.

A company could... if you have N users and you pay M for storage per user and downtime cost you X then it could be that a discount of Y means (M - Y) * N = X

Re: Tarsnap outage postmortem

#163

Post by bluGill »

mrweasel wrote: post #61
AceJohnny2 wrote: post #39

Earlier quoted context omitted.

It's not about suing, but defining expectations about how you can rely on a service. For example, my team has people across the world for HW bringup, so we can't allow our code hosting or CI to be down for more than a few hours. Of course, backups have different uptime requirements, but as for everything, it's a tradeoff between features, of which an SLA is one. Tarsnap's features are granularity of cost, reliability…

> It's not about suing, but defining expectations about how you can rely on a service. Meeeeh, my ISP cut of around 100+ fiber connections in my town and spend three weeks fixing it. My neighbor have business line, there's an SLA on those that among other things, require them if reestablish his connection within 3 - 5 hours. It took them over 500 hours, so that SLA is useless for anything but forcing compensations. T…

SLAs can be meaningless like that. However the better ISPs have in place a backup system that doesn't use the same fiber/wires. Sure the backup might be a radio or satellite feed and so be slower, but it will get/keep you online. This costs are lot more per month though, so if you are not paying for that service your SLA will probably just be we give you a free month (which hurts them enough that they will do some things to prevent downtime, but not enough that they put redundant fiber paths in the ground)

Re: Tarsnap outage postmortem

#164

Post by tomjakubowski »

ivanhoe wrote: post #159
stigz wrote: post #9

Earlier quoted context omitted.

Why would I use your service over restic? God bless you Colin, but reading this, it appears you're the only one in charge of the infrastructure for this service. I'm glad you're clear about no SLA, but this seems like a big liability between me and my backups.

It's a pretty well-known fact for years that tarsnap is basically a one-man show, and yet Colin has managed to provide fantastic service so far. Sometimes having ppl who built the service also managing it is actually a big plus, compared to other services where you first have to fight through outsourced & underpaid support that's limited to template answers, only to finally get some "engineer" who got that job 2 mont…

Why the scare quotes? I would expect any well-experienced power user to know a complicated system better than a fresh engineer two months into working on it, with no previous experience on the system. Especially if the power user is an engineer themself.

Re: Tarsnap outage postmortem

#165

Post by acedTrex »

bluehatbrit wrote: post #145
Tepix wrote: post #116

Earlier quoted context omitted.

It was a postmortem without the mandatory "how can we prevent this in the future" steps…

I think that's a little unfair given what was in the postmortem. It may not be a separate section with the key points, but the information is all there of what the issues were and what the solutions are. I think it's fair to assume they're actually acting on those without them needing to be reiterated at the bottom of the page.

I agree, we don't really need a "key points/future actions" section that boils down to "The service will be geo redundant"

Re: Tarsnap outage postmortem

#166

Post by lordgilman »

zetalyrae wrote: post #95

>The process of recovering the EC2 instance state consists of two steps: First, reading all of the metadata headers from S3; and second, "replaying" all of those operations locally. (These cannot be performed at the same time, since the use of log-structured storage means that log entries are "rewritten" to free up storage when data is deleted; log entries contain sequence numbers to allow them to be replayed in the…

FWIW Tarsnap was launched in 2008, the initial RDS for MySQL was launched in 2009.

Re: Tarsnap outage postmortem

#167

Post by XCSme »

cowsandmilk wrote: post #158
deltarholamda wrote: post #156

Earlier quoted context omitted.

While I get this as a risk, I'm not convinced it's any more risky than a larger corporate entity. This is Colin's job. Colin has his name attached to it. It's really important to Colin. You're not going to get the same kind of service from BigBackupCorp. Their employees are replaceable, their management is replaceable, and to be honest, you as a customer are replaceable, if they decide to move in a different directio…

It isn’t necessarily about Colin quitting. Key person gets hit by bus is also always a concern. You can say someone will pick it up, but I know nothing of whether such plans are in place. Does the person who would inherit the business have the know how to sell it? Is there enough documentation in place for a transfer of assets to be successful?

Companies and corporations get "killed" often too, even if the people in them are alive.

Re: Tarsnap outage postmortem

#168

Post by electroly »

tinco wrote: post #155
patrec wrote: post #77

Earlier quoted context omitted.

Everything about tarsnap is absurd. It's basically the world's most absurd backup service (insanely expensive, poor UX, bus factor of ~1, restoring moderate amounts of data appears to take days (!)[1]), brought about by an absurdly bad allocation of human capital (it's run by a double Putnam challenge winner, with several other impressive accomplishments), and as such, absurdly beloved by HN. [1] In case of an emerge…

If tarsnap has even a modicum of popularity, thanks to these prices it would be bringing in bank. If he's making bank, that means he's providing value (even if it's just to "uninformed"). And it seems this system mostly runs itself, so it's a side gig. It's probably a far more effective allocation than many other possible allocations of human capital. How many of the world's best and brightest are doing all sorts of…

> If tarsnap has even a modicum of popularity

It won't, though, because of the points mentioned by the post you're replying to. It's been 15 years; tarsnap is as popular as it's going to get.

Re: Tarsnap outage postmortem

#169

Post by electroly »

LukeShu wrote: post #69
mekster wrote: post #29

Earlier quoted context omitted.

It's insane. Not sure how anyone can accept such a rip off pricing. Tarsnap : $0.25 / GB storage, $0.25 / GB bandwidth cost rsync.net : $0.015 / GB storage, no bandwidth cost s3 : $0.023 / GB storage, some complicated bandwidth pricing If tarsnap is built on top of s3, they're charging 10 times for the storage cost. Easy money from the uninformed?

How's the saying in every HN thread go? "Don't set your prices based on your costs, set your prices based on the value you deliver." or something like that. Tarsnap is a wonderful piece of software. You're paying for that. That said, is the value of "Tarsnap" worth the price difference from "Borg+rsync.net"? (Or Restic, I've been meaning to look into Restic). I'm not so sure. These days I'm a customer of rsync.net, n…

rsync.net is also overpriced for strictly backup purposes. Make sure you do check out restic; it can use S3 or Backblaze B2 (I actually use both) as backends instead of something expensive like rsync.net. The value of these boutique storage services evaporates when you start using restic.

Re: Tarsnap outage postmortem

#170

Post by [deleted] »

ilyt wrote: post #152
alanfranz wrote: post #30

Earlier quoted context omitted.

You need a remote service that keeps backup readonly. You’re not covering attack scenarios if you just use raw object storage from your client machine. I have written about this some time ago if you’re interested: https://www.franzoni.eu/ransomware-resistant-backups/

How tarsnap keeps backups readonly ? Just having service being a barrier is not enough

[deleted]
Post reply on HN