Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Tarsnap outage postmortem

mail.tarsnap.com

Post reply on HN
Print view View original

271–280 of 338 posts

Re: Tarsnap outage postmortem

#271

Post by nemothekid »

zetalyrae wrote: post #138
throwawaaarrgh wrote: post #129

Earlier quoted context omitted.

Forget about software or data architecture. S3 is the most reliable data storage mechanism in the world, and insanely simpler than a relational database. There is no operational failure mode to S3, other than "region went down". There is no instance to go down, no replication to fail, no worry about whether there's enough capacity for writes or too many connections, no thought to a schema, no migrations to manage, no…

>S3 is the most reliable data storage mechanism in the world S3 is not the problem here. The problem is building a database on top of S3 , and having to reimplement all the consistency, atomicity, transactions etc. on top. >no thought to a schema, no migrations to manage There is, in fact, always a schema. Some people choose to ignore it's there, to their detriment. >Always avoid the fancy option when it makes sense.…

>Cheaper than a 26-hour outage?

AFAICT, using HN theres been roughly 30 hours of non-availability over 15 years. RDS didn't even support Postgres when Tarsnap was released.

EDIT: Tarsnap predates RDS.

Re: Tarsnap outage postmortem

#272

Post by nemothekid »

tptacek wrote: post #270
nemothekid wrote: post #269

Earlier quoted context omitted.

I'm having a hard time to believe that anyone remotely interested in Tarsnap's value prop is also an "uninformed mom-and-pop". This "uninformed mom-and-pop" is potentially compiling the client application from source, but can't do basic math to compare tarsnap's pricing to the top 20 or so competitors that rank above tarsnap in SEO?

That was sarcasm. :)

Should have looked at the username, massive egg on face.

Re: Tarsnap outage postmortem

#273

Post by LukeShu »

AnonHP wrote: post #205
electroly wrote: post #169

Earlier quoted context omitted.

rsync.net is also overpriced for strictly backup purposes. Make sure you do check out restic; it can use S3 or Backblaze B2 (I actually use both) as backends instead of something expensive like rsync.net. The value of these boutique storage services evaporates when you start using restic.

The complexity with S3 or Backblaze B2 is the granular pricing for various operations on the service. It’s difficult to optimize costs compared to a fixed “$X per GB” pricing that’s easier for people to understand and forecast expenses a bit more easily. Other than that, rsync.net provides the same pricing for different days center locations, which means there is some subsidization going on. There are services like r…

> There are services like rsync.net that support borg at a lower price.

And rsync.net is even one of them!

"Special "borg accounts" are available at a very deep discount for technically proficient users." -- https://www.rsync.net/products/borg.html

...hrm, it seems they didn't update that page with last year's price drop. https://web.archive.org/web/20220319135035/https://www.rsync... It used to be a deep discount, now it's the same for <100TB. I wonder if they did drop the Borg prices too and just forgot to update that page?

Re: Tarsnap outage postmortem

#274

Post by khiqxj »

zetalyrae wrote: post #153
bastawhiz wrote: post #150

Earlier quoted context omitted.

> having to reimplement all the consistency, atomicity, transactions etc. on top. Most of those problems are moot if you're only ever writing from a single head node. If all your data is strictly ordered and you have no meaningful concurrency, this is a far, far simpler system.

Did I fall into a timewarp into the 70's? How on Earth, by what sane standard, is a Postgres instance too complex? If you're `fopen`ing files as a "database" you are wasting your time and lowering the world's economic productivity. Complex is Greenspunning a database and having it blow up in your face and cause a twenty-six hour outage. You never hear about such things with Postgres because Postgres is rock-solid.

[flagged]

Re: Tarsnap outage postmortem

#275

Post by molsongolden »

hunter2_ wrote: post #267
twic wrote: post #252

Earlier quoted context omitted.

> God bless you Colin, but reading this, it appears you're the only one in charge of the infrastructure for this service Hence the toddler.

Are you suggesting that those who build enterprises don't have time for kids? Seems plausible, but is the difference in lifestyle so consistently prevalent as to be stereotypical? Elon has 10!

Raising the toddler to have some help running the business.

Re: Tarsnap outage postmortem

#276

Post by dang »

catiopatio wrote: post #256
dang wrote: post #254

Earlier quoted context omitted.

You broke the site guidelines badly in more than one place in this thread. I realize you're trying to defend someone's work against what you feel is unfair criticism, but breaking the site guidelines yourself, with swipes and name-calling and flamewar, is exactly the wrong way to do this. If you'd please review https://news.ycombinator.com/newsguidelines.html and stick to the rules when posting here, we'd appreciate…

You’re not wrong, and thank you for holding me to account. In retrospect, I’d delete and/or edit the comments if I could.

No need to delete - the only thing we care about is fixing things going forward. Thanks for the kind reply!

Re: Tarsnap outage postmortem

#277

Post by Mawr »

stigz wrote: post #9
cperciva wrote: post #3

blinks Ok, I really wasn't expecting this to land at the top of HN. I'd love to stick around to answer any questions people have, but it's 10PM and my toddler decided to go to bed at 5PM... so if I'm lucky I can get about 4 hours of sleep before she decides that it's time to get up. I'll check in and answer questions in the morning.

Why would I use your service over restic? God bless you Colin, but reading this, it appears you're the only one in charge of the infrastructure for this service. I'm glad you're clear about no SLA, but this seems like a big liability between me and my backups.

Uptime isn't an important property of a backup solution, so I'm not sure where the expectation comes from?

Re: Tarsnap outage postmortem

#278

Post by idlewords »

jacquesm wrote: post #266
idlewords wrote: post #220

Earlier quoted context omitted.

It always amuses me how people want reassurance that the next crisis will be a fresh, new problem, and not one the person can demonstrably solve. A lot of 'lessons learned' analysis boils down to this: in order to prevent a recurrence of X, we introduced complex subsystem Y, the unexpected effects of which you can read about in our next post-mortem.

That was entirely not what I was getting at and is a cheap shot that is well beneath you, especially because I suspect that you know that that wasn't what I was getting at.

My comment wasn't intended personally; your words about "will never recur" just reminded me of this peculiarity of software systems, where it's often error handling/monitoring/backups/etc. that cause cascading failures in the systems they're intended to safeguard.

I'm sorry if I misconstrued your meaning, but I am flattered that you think there are things beneath me!

Re: Tarsnap outage postmortem

#279

Post by jacquesm »

idlewords wrote: post #278
jacquesm wrote: post #266

Earlier quoted context omitted.

That was entirely not what I was getting at and is a cheap shot that is well beneath you, especially because I suspect that you know that that wasn't what I was getting at.

My comment wasn't intended personally; your words about "will never recur" just reminded me of this peculiarity of software systems, where it's often error handling/monitoring/backups/etc. that cause cascading failures in the systems they're intended to safeguard. I'm sorry if I misconstrued your meaning, but I am flattered that you think there are things beneath me!

Fair enough. I see the whole function of a postmortem in a very simple way: to avoid recurrence of the same fault. Yes, there will be plenty of new ones to make. But if you don't change your processes as the result of a failure you are almost certainly going to see a repeat because the bulk of the conditions are still the same. All it takes then is a minor regression and you're back to where you were before. This I've seen many times in practice and I suspect that Colin isn't immune to it. And yes, I look up to you, your writing is usually sharp and on point and has both amused me and educated me. So you have an image to live up to ;)

Re: Tarsnap outage postmortem

#280

Post by Kwpolska »

TheDong wrote: post #72
Kwpolska wrote: post #45

Earlier quoted context omitted.

How should a browser differentiate between a of hard-wrapped prose (that it could reflow in reader mode) and a of code?

In an email you can't of course since an email could contain one, the other, or a mix. And, in fact, most mailing list emails I read these days do contain code and code reviews. However, if the user clicked the "reader mode" button, that's a good sign the user thinks this is reflowable text. Firefox's reader mode figures this out. Safari's doesn't.

Or you might use Reader Mode on a programming blog, which has both prose and code, and you wouldn’t want to change the code in that case.
Post reply on HN