Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Tarsnap outage postmortem

mail.tarsnap.com

Post reply on HN
Print view View original

221–230 of 338 posts

Re: Tarsnap outage postmortem

#221

Post by zetalyrae »

foldr wrote: post #217
zetalyrae wrote: post #214

Earlier quoted context omitted.

RDS can have replication. But more than that: servers should be stateless! A server going down should never take down your business. If you use Postgres, and stateless servers, then if a server goes down it's no problem, it gets rebooted and there may be other servers and a load balancer to pick up the load. If Postgres goes down, you have a replica, or it gets rebooted, and Postgres always recovers from crashes (in…

The hardware failure was on the server running the application code, so RDS replication wouldn’t have helped. You’re right of course that this failure points to a lack of redundancy – but that’s a separate issue from choosing S3 vs. RDS as the data layer. By the way, S3 is insanely reliable and in fact more reliable than a replicated RDS setup. So switching from S3 to RDS would almost certainly reduce the basic relia…

Once again, the problem is not S3, it is reinventing a database on top of S3, the logic of which runs on EC2.

Re: Tarsnap outage postmortem

#222

Post by [deleted] »

catiopatio wrote: post #219
zetalyrae wrote: post #214

Earlier quoted context omitted.

RDS can have replication. But more than that: servers should be stateless! A server going down should never take down your business. If you use Postgres, and stateless servers, then if a server goes down it's no problem, it gets rebooted and there may be other servers and a load balancer to pick up the load. If Postgres goes down, you have a replica, or it gets rebooted, and Postgres always recovers from crashes (in…

Just … stop. This is the first outage in 11 years. You’re being unnecessarily arrogant and antagonistic up and down this thread You’re not smarter than everyone else here, you don’t have better or more perfect knowledge, and you almost certainly wouldn’t have built a better or more reliable system.

[deleted]

Re: Tarsnap outage postmortem

#223

Post by [deleted] »

catiopatio wrote: post #219
zetalyrae wrote: post #214

Earlier quoted context omitted.

RDS can have replication. But more than that: servers should be stateless! A server going down should never take down your business. If you use Postgres, and stateless servers, then if a server goes down it's no problem, it gets rebooted and there may be other servers and a load balancer to pick up the load. If Postgres goes down, you have a replica, or it gets rebooted, and Postgres always recovers from crashes (in…

Just … stop. This is the first outage in 11 years. You’re being unnecessarily arrogant and antagonistic up and down this thread You’re not smarter than everyone else here, you don’t have better or more perfect knowledge, and you almost certainly wouldn’t have built a better or more reliable system.

[deleted]

Re: Tarsnap outage postmortem

#224

Post by rsync »

rssoconnor wrote: post #180
rcxdude wrote: post #174

Earlier quoted context omitted.

Rsync.net can do similar, because of their snapshot system. By default there's no way to delete a snapshot except through the schedule set up (you can write to them and ask if it's necessary for some reason). It doesn't use asymmetric crypto to do this but it's neither necessary not sufficient for the purpose of preventing accidental or malicious deletion of backups https://www.rsync.net/resources/howto/snapshots.htm…

Right I've considered that. It is however limited to like 7 snapshots. The thing is that tarsnap deduplicates over arbitrarily long time periods, letting me make arbitrarily long staggered sequences of retained archives. Perhaps I should really reconsider if I really need such long lived archives, but it is hard to bring myself to drop them.

No, not limited.

An rsync.net account can have any arbitrary schedule of snapshots - including days, weeks, months, quarters and years.

Re: Tarsnap outage postmortem

#225

Post by catiopatio »

zetalyrae wrote: post #221
foldr wrote: post #217

Earlier quoted context omitted.

The hardware failure was on the server running the application code, so RDS replication wouldn’t have helped. You’re right of course that this failure points to a lack of redundancy – but that’s a separate issue from choosing S3 vs. RDS as the data layer. By the way, S3 is insanely reliable and in fact more reliable than a replicated RDS setup. So switching from S3 to RDS would almost certainly reduce the basic relia…

Once again, the problem is not S3, it is reinventing a database on top of S3, the logic of which runs on EC2.

Once again, no, that’s not the problem.

PostgreSQL and RDS are quite a bit more than just a log-structured data store, and are not prima facie the correct solution for this problem domain, regardless of how much arrogant ignorance you bring to bear on the debate.

Re: Tarsnap outage postmortem

#226

Post by catiopatio »

ilyt wrote: post #152
alanfranz wrote: post #30

Earlier quoted context omitted.

You need a remote service that keeps backup readonly. You’re not covering attack scenarios if you just use raw object storage from your client machine. I have written about this some time ago if you’re interested: https://www.franzoni.eu/ransomware-resistant-backups/

How tarsnap keeps backups readonly ? Just having service being a barrier is not enough

It supports distinct authentication keys with read, write, and/or delete permissions for the same data protected by a given encryption key.

Those restrictions are enforced by the service.

Re: Tarsnap outage postmortem

#227

Post by Tepix »

bluehatbrit wrote: post #145
Tepix wrote: post #116

Earlier quoted context omitted.

It was a postmortem without the mandatory "how can we prevent this in the future" steps…

I think that's a little unfair given what was in the postmortem. It may not be a separate section with the key points, but the information is all there of what the issues were and what the solutions are. I think it's fair to assume they're actually acting on those without them needing to be reiterated at the bottom of the page.

Well, for sure he has fixed several bugs, but he didn't say that he would be testing his disaster recovery procedure every year in the future for example.

Re: Tarsnap outage postmortem

#228

Post by mst »

ilyt wrote: post #151
mst wrote: post #86

Earlier quoted context omitted.

Roughly: The number of hours of my time that would be required to get something with even theoretically equivalent features would be sufficient to make the cost - and opportunity cost - involved seem far more reasonable. Plus "written by cperciva and heavily battle tested by Serious Sysadmins" is a feature I couldn't recreate myself - notice that while there was an outage, part of the reason for it taking a while was…

> notice that while there was an outage, part of the reason for it taking a while was a conscious choice to take a much longer path to resolution than bringing up the previous server in the name of paranoia. Paranoia about data corruption is a nice thing to have in a backup system and something I'm happily willing to trade-off uptime for. As Actual Serious Sysadmin that Actually Manages Big Systems for Living that sc…

Your comments seem to me to be about how long it took to bring up a new system.

Without agreeing or disagreeing with those, "I'm not going to trust the filesystem on the existing machine" was the choice I was talking about.

Re: Tarsnap outage postmortem

#229

Post by catiopatio »

zetalyrae wrote: post #112
amluto wrote: post #104

Earlier quoted context omitted.

> This would benefit from being further from the metal. How, exactly, is that a good thing?

How is not rolling your own database a good thing? Mainly because the business of tarsnap is 1) encrypted 2) backups, not building a database storage engine.

It’s cute that you think implementing client-side encrypted, deduplicated backups doesn’t involve building a database storage engine.

Re: Tarsnap outage postmortem

#230

Post by mst »

zetalyrae wrote: post #97
mst wrote: post #86

Earlier quoted context omitted.

Roughly: The number of hours of my time that would be required to get something with even theoretically equivalent features would be sufficient to make the cost - and opportunity cost - involved seem far more reasonable. Plus "written by cperciva and heavily battle tested by Serious Sysadmins" is a feature I couldn't recreate myself - notice that while there was an outage, part of the reason for it taking a while was…

You don't need to build it yourself. You can use restic or rclone with S3/B2 and both support encryption.

I am aware there are existing tools that handle encryption.

The tarsnap architecture still does more things.

You're welcome to feel that you don't need those things, but that wasn't my point.

Post reply on HN