Live data from Hacker News

Tarsnap outage postmortem

mail.tarsnap.com

171–180 of 338 posts

Re: Tarsnap outage postmortem

#171

>The process of recovering the EC2 instance state consists of two steps: First, reading all of the metadata headers from S3; and second, "replaying" all of those operations locally. (These cannot be performed at the same time, since the use of log-structured storage means that log entries are "rewritten" to free up storage when data is deleted; log entries contain sequence numbers to allow them to be replayed in the…

FWIW Tarsnap was launched in 2008, the initial RDS for MySQL was launched in 2009.

You can always self-host Postgres.

Re: Tarsnap outage postmortem

#172
post #115

Earlier quoted context omitted.

This is terrible. Instinct cannot be trusted. Write it down.

There are two types of emergencies - checklist ones, and panic ones. You need to have both, but realize that in the panic ones people do NOT operate rationally. This is why house doors open in but business doors have to open out - if there’s a crush against a fire door it opens. You even see this in aviation, where everything is checkisted; the pilots will first stabilize the plane in an emergency and then run the ch…

Note that in airplanes (unlike cars) you normally cannot just get in a new one and fly. You first get training on that particular plane. If everything goes perfect any pilot can get in any plane and fly it, but if any little thing goes wrong they better know how the plane flys very well so they can get it stable enough to run the checklist.

Re: Tarsnap outage postmortem

#174
post #87

Earlier quoted context omitted.

> Tarsnap makes a lot of sense when you benefit from the encryption and (especially) de-duplication features that it offers. Can you provide more detail why you think so? I don't believe there is any use case in which tarsnap makes sense, other than maybe some Plan-C backup solution which you fall back on in the highly unlikely event that neither Plan-A nor Plan-B worked. Concretely, what benefits does tarsnap offer…

I use tarsnap because the asymmetric crypto means I can give my cron job authorization to create backups, but it doesn't have authorization to read or delete(!) backups. This ability is critical to prevent a compromised system from having its data wiped and having all backups wiped as well. I haven't been able to figure out how to do this in any other system. But if someone has a tutorial, I am all ears.

Rsync.net can do similar, because of their snapshot system. By default there's no way to delete a snapshot except through the schedule set up (you can write to them and ask if it's necessary for some reason). It doesn't use asymmetric crypto to do this but it's neither necessary not sufficient for the purpose of preventing accidental or malicious deletion of backups

https://www.rsync.net/resources/howto/snapshots.html

Re: Tarsnap outage postmortem

#175
post #152

Earlier quoted context omitted.

You need a remote service that keeps backup readonly. You’re not covering attack scenarios if you just use raw object storage from your client machine. I have written about this some time ago if you’re interested: https://www.franzoni.eu/ransomware-resistant-backups/

How tarsnap keeps backups readonly ? Just having service being a barrier is not enough

[deleted]

Re: Tarsnap outage postmortem

#176
post #80
post #69

Earlier quoted context omitted.

How's the saying in every HN thread go? "Don't set your prices based on your costs, set your prices based on the value you deliver." or something like that. Tarsnap is a wonderful piece of software. You're paying for that. That said, is the value of "Tarsnap" worth the price difference from "Borg+rsync.net"? (Or Restic, I've been meaning to look into Restic). I'm not so sure. These days I'm a customer of rsync.net, n…

Completely ignoring costs, can you name a single use case for which tarsnap would be better than Borg or restic on rsync.net?

Backing up anything Windows with more granularity than top level directories?

Ugh.

Try picking a choosing specific file types or file extensions from filesystems holding thousands of files.

I ended up having to cobble together some god-awful pre-process powershell with multiple pipes just because restic fails to be able to grep using Windows reliably.

:(

Re: Tarsnap outage postmortem

#177

(caveat: I may be running on old tarsnap company info but) I must say, the ONLY thing that has ever made me shy away from seriously using tarsnap was the prospect of an unexpected Colin Percival outage. i.e. key person risk. I'm guessing I'm not alone in this.

It's an MTBF like calculation: do you trust the well engineered one person company that has a well engineered solution with few moving parts over the much larger company that has far more moving parts and their probably less well engineered solution with far more moving parts?

I personally would go with the simpler solution because in my experience you need an awful lot of extra complexity before you get to the same level of reliability that you have with the simpler system. Most complexity is just making things worse.

You can see this clearly when it comes to clustering servers. A single server with a solid power supply and network hookup will be more reliable than any attempt at making that service redundant until you get to something like 5x more costly and complex. Then maybe you'll have the same MTBF as you had with the single server. Beyond that you can have actualy improvements. YMMV and you may be able to get better reliability at the same level of performance in some cases but on average it always first gets far more complex, costly and fragile before you see any real improvements.

I strongly believe that the best path to real reliability is simplicity (which is: as simple as possible) and good backups. For stuff that needs to be available 24x7 and 365 days per year this limits your choices in available technologies considerably.

Re: Tarsnap outage postmortem

#178
post #99

Earlier quoted context omitted.

If the data is super important you should be setting on two different providers anyways for backups.

Honestly, whose data isn't "super important"?. All my data is super important. Even the crap I just throw on my Google drive. I want to keep it. What is this mythical unimportant data that people still want to back up?

I really used to enjoy formatting my machines about every 6 months.

Well I used to until macOS kinda went off the rails a bit. Now it’s mostly an exercise in running my arch script for my thinkpad.

Being stuck between operating systems is kinda a mess though, makes backup and file sync in general really hard. But everyone’s gotta have their own cloud, right?!

Why can’t I just put a cloud under my bed and forget about it?

Re: Tarsnap outage postmortem

#179

Earlier quoted context omitted.

Honestly, whose data isn't "super important"?. All my data is super important. Even the crap I just throw on my Google drive. I want to keep it. What is this mythical unimportant data that people still want to back up?

Anything that you stashed just for convenience, but you could re-download or re-generate it if really needed, or simply live without it... frankly, like 90% of stuff on my disks fall in the category "I'll read/view it one day", which in reality I'll probably never have time or patience to open ever again.

Strange, 90% of the things trapped in my flash memory are system files.

Re: Tarsnap outage postmortem

#180

Earlier quoted context omitted.

I use tarsnap because the asymmetric crypto means I can give my cron job authorization to create backups, but it doesn't have authorization to read or delete(!) backups. This ability is critical to prevent a compromised system from having its data wiped and having all backups wiped as well. I haven't been able to figure out how to do this in any other system. But if someone has a tutorial, I am all ears.

Rsync.net can do similar, because of their snapshot system. By default there's no way to delete a snapshot except through the schedule set up (you can write to them and ask if it's necessary for some reason). It doesn't use asymmetric crypto to do this but it's neither necessary not sufficient for the purpose of preventing accidental or malicious deletion of backups https://www.rsync.net/resources/howto/snapshots.htm…

Right I've considered that. It is however limited to like 7 snapshots.

The thing is that tarsnap deduplicates over arbitrarily long time periods, letting me make arbitrarily long staggered sequences of retained archives.

Perhaps I should really reconsider if I really need such long lived archives, but it is hard to bring myself to drop them.

Post reply on HN