Tarsnap outage postmortem • Hacker News Board

Re: Tarsnap outage postmortem

#191

Post by foldr » Thu, Jul 27, 2023, 1:52 PM UTC

zetalyrae wrote: post #124

nonethewiser wrote: post #122

Earlier quoted context omitted.

>> Cost > It's a bad trade. Maybe. But that's the reason. You never acknowledged that advantage in your question so it needed to be emphasized

It never occurred to me that anyone would need it explained to them that RDS is cheaper than the time of any software engineer. The opportunity cost of building your own database is 10,000x the cost of running RDS for a year.

That sort of logic doesn't really apply here because:

* RDS costs obviously scale linearly with ongoing time and probably scale linearly with the total amount of data being backed up. So depending on the revenue of the business, these extra costs could easily end up outweighing the (notional) cost of the time saved, which is mostly a one-off expense.

* The cost of a software engineer's time is notional in the context of a one-person business. The author of Tarsnap isn't going to be able to employ fewer than zero additional software engineers to maintain Tarsnap because of the time saved by using RDS.

Re: Tarsnap outage postmortem

#192

Post by jacquesm » Thu, Jul 27, 2023, 1:54 PM UTC

cperciva wrote: post #3

blinks Ok, I really wasn't expecting this to land at the top of HN. I'd love to stick around to answer any questions people have, but it's 10PM and my toddler decided to go to bed at 5PM... so if I'm lucky I can get about 4 hours of sleep before she decides that it's time to get up. I'll check in and answer questions in the morning.

In future postmortems (of which I hope there will be very few or even none) you may want to spell out your 'lessons learned' to show why particular items will never recur.

Re: Tarsnap outage postmortem

#193

Post by rcxdude » Thu, Jul 27, 2023, 1:56 PM UTC

rssoconnor wrote: post #180

rcxdude wrote: post #174

Earlier quoted context omitted.

Rsync.net can do similar, because of their snapshot system. By default there's no way to delete a snapshot except through the schedule set up (you can write to them and ask if it's necessary for some reason). It doesn't use asymmetric crypto to do this but it's neither necessary not sufficient for the purpose of preventing accidental or malicious deletion of backups https://www.rsync.net/resources/howto/snapshots.htm…

Right I've considered that. It is however limited to like 7 snapshots. The thing is that tarsnap deduplicates over arbitrarily long time periods, letting me make arbitrarily long staggered sequences of retained archives. Perhaps I should really reconsider if I really need such long lived archives, but it is hard to bring myself to drop them.

You can do the same with rsync, they just charge you for the extra space (the differential space, like with tarsnap) instead of providing them for free (from what I can see the limits in the web UI are like 1000 daily and weekly snapshots, 200 monthy snapshots, 100 quarterly snapshows, and 10 yearly snapshots, which I suspect are arbitrary 'good enough for most' numbers, not some hard limit based on what they can profitably provide). I personally use the direct ZFS option so I can set up the snapshots exactly how I want, but it is extra effort and doesn't provide quite as good a guarantee that they won't be overwritten (it's resilient against a compromise of the server uploading the backups because I've set up scripts that way, but it doesn't protect against compromise of the logic credentials for the VM in the same way).

Re: Tarsnap outage postmortem

#194

Post by bombcar » Thu, Jul 27, 2023, 2:00 PM UTC

jpgvm wrote: post #186

bombcar wrote: post #143

Earlier quoted context omitted.

There are two types of emergencies - checklist ones, and panic ones. You need to have both, but realize that in the panic ones people do NOT operate rationally. This is why house doors open in but business doors have to open out - if there’s a crush against a fire door it opens. You even see this in aviation, where everything is checkisted; the pilots will first stabilize the plane in an emergency and then run the ch…

Pilots are a little special, their panic mode is also a checklist, known as the memory items. This doesn't work for normal people because normal people don't drill non-normal events until the response is instinctive.

Normal people should drill certain non-normal events (for example, all drivers should know how to deaccelerate and get off the road quickly).

But you should NEVER design a system that requires normal people to drill non-normal events; even planes have been redesigned to "fix" problems where the pilot had to do something unintuitive or unexpected, because eventually it WILL catch up to you.

Re: Tarsnap outage postmortem

#195

Post by tptacek » Thu, Jul 27, 2023, 2:00 PM UTC

mekster wrote: post #29

RockRobotRock wrote: post #13

Aren't these storage prices absurd? Please let me know if I'm misunderstanding.

It's insane. Not sure how anyone can accept such a rip off pricing. Tarsnap : $0.25 / GB storage, $0.25 / GB bandwidth cost rsync.net : $0.015 / GB storage, no bandwidth cost s3 : $0.023 / GB storage, some complicated bandwidth pricing If tarsnap is built on top of s3, they're charging 10 times for the storage cost. Easy money from the uninformed?

Yes. That's the thing about Tarsnap, a service with a TikZ diagram on its front page, built around a Unix utility, that meters in picodollars. It's meant to bilk money from uninformed mom-and-pop backup users.

Re: Tarsnap outage postmortem

#196

Post by bombcar » Thu, Jul 27, 2023, 2:03 PM UTC

tialaramex wrote: post #189

bluGill wrote: post #172

Earlier quoted context omitted.

Note that in airplanes (unlike cars) you normally cannot just get in a new one and fly. You first get training on that particular plane. If everything goes perfect any pilot can get in any plane and fly it, but if any little thing goes wrong they better know how the plane flys very well so they can get it stable enough to run the checklist.

You probably shouldn't just get in a new car and drive it, but people do. I remember at a hire car place once the team I worked with were given an automatic, the guy driving has never driven an automatic transmission before, but his license authorises it (UK licenses allow everybody to drive an automatic, but you need to test in a manual to drive manual), and so they just lent him a car with a completely different dr…

It is kind of mind-boggling insane that you can be 25 years or (younger in some states/places), having only ever driven a smart car (so you have your license) and you can walk into U-Haul and rent a 26 foot box truck with a trailer, and the most they do is tell you not to go under low overpasses or into drive-thrus.

Re: Tarsnap outage postmortem

#197

Post by zetalyrae » Thu, Jul 27, 2023, 2:05 PM UTC

foldr wrote: post #191

zetalyrae wrote: post #124

Earlier quoted context omitted.

It never occurred to me that anyone would need it explained to them that RDS is cheaper than the time of any software engineer. The opportunity cost of building your own database is 10,000x the cost of running RDS for a year.

That sort of logic doesn't really apply here because: * RDS costs obviously scale linearly with ongoing time and probably scale linearly with the total amount of data being backed up. So depending on the revenue of the business, these extra costs could easily end up outweighing the (notional) cost of the time saved, which is mostly a one-off expense. * The cost of a software engineer's time is notional in the context…

[flagged]

Re: Tarsnap outage postmortem

#198

Post by tptacek » Thu, Jul 27, 2023, 2:05 PM UTC

Tepix wrote: post #116

mike_d wrote: post #12

Earlier quoted context omitted.

This was an extremely well written and thoughtful postmortem, but I hope to never see one from you again. :)

It was a postmortem without the mandatory "how can we prevent this in the future" steps…

In 15+ years of running this service, this is one of two (2) postmortems he's ever published, and the first in eleven (yes, 11) years.

Re: Tarsnap outage postmortem

#199

Post by mattigames » Thu, Jul 27, 2023, 2:08 PM UTC

zetalyrae wrote: post #148

throwawaaarrgh wrote: post #146

Earlier quoted context omitted.

> having to reimplement all the consistency, atomicity, transactions etc. on top. Did you miss where I said it's read-after-write strongly consistent?

Having those things at the base layer is not the same as having them in higher-level behaviour. This is why God invented SQL transactions.

I never knew Donald Chamberlin and Raymond Boyce were God, it doesn't sound right but I don't know enough about theology to refute it.

Re: Tarsnap outage postmortem

#200

Post by otabdeveloper4 » Thu, Jul 27, 2023, 2:10 PM UTC

ilyt wrote: post #132

throwawaaarrgh wrote: post #129

Earlier quoted context omitted.

Forget about software or data architecture. S3 is the most reliable data storage mechanism in the world, and insanely simpler than a relational database. There is no operational failure mode to S3, other than "region went down". There is no instance to go down, no replication to fail, no worry about whether there's enough capacity for writes or too many connections, no thought to a schema, no migrations to manage, no…

The "fancy" option here is trying to act S3 to act like database instead of simple blob storage...

I'm only wildly guessing here, but most likely the "cloud storage" backing all those managed databases is actually S3-like blob storage under the hood.