Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Tarsnap outage postmortem

mail.tarsnap.com

Post reply on HN
Print view View original

111–120 of 338 posts

Re: Tarsnap outage postmortem

#111

Post by prmoustache »

Aeolun wrote: post #105
prmoustache wrote: post #92

Earlier quoted context omitted.

How can simple black on white text with a lisible font be hard to read?

It doesn’t adjust to my screen size. So in portrait mode I have sentences with 80 characters per line that are 5 points wide. That’s barely possible to read on a high dpi screen, but fairly uncomfortable.

I understand some websites are doing lots of funky stuff, but at this point the issue is on your particular client if it cannot zoom bare text correctly and smoothly. Any decent client should be able to allow you to zoom it so that you can maximize those 80 character column to the width of your phone screen in protrait mode regardless of font size and dpi.

Re: Tarsnap outage postmortem

#112

Post by zetalyrae »

amluto wrote: post #104
zetalyrae wrote: post #95

>The process of recovering the EC2 instance state consists of two steps: First, reading all of the metadata headers from S3; and second, "replaying" all of those operations locally. (These cannot be performed at the same time, since the use of log-structured storage means that log entries are "rewritten" to free up storage when data is deleted; log entries contain sequence numbers to allow them to be replayed in the…

> This would benefit from being further from the metal. How, exactly, is that a good thing?

How is not rolling your own database a good thing? Mainly because the business of tarsnap is 1) encrypted 2) backups, not building a database storage engine.

Re: Tarsnap outage postmortem

#113

Post by seanthemon »

andrioni wrote: post #107
terinjokes wrote: post #101

Earlier quoted context omitted.

Backblaze has a policy of allowing backups of external disks, but the disks have to be connected at least once every 30 days, or they'll delete the backups. I understand they want to avoid abuse, but the lack of any grace period, or ability of support to ad an override, really soured the service for me.

You can just pay extra for extended or infinite retention. https://www.backblaze.com/cloud-backup/features/extended-ver...

I did this, it's come in use a few times..

Re: Tarsnap outage postmortem

#114

Post by zetalyrae »

foldr wrote: post #108
zetalyrae wrote: post #95

>The process of recovering the EC2 instance state consists of two steps: First, reading all of the metadata headers from S3; and second, "replaying" all of those operations locally. (These cannot be performed at the same time, since the use of log-structured storage means that log entries are "rewritten" to free up storage when data is deleted; log entries contain sequence numbers to allow them to be replayed in the…

>Far be it from me to tell anyone how to write software, but why build a database on top of S3 when you can just chuck the metadata into RDS with however much replication you want? Cost and reliability? * Using S3 as a simple database is generally going to be much cheaper than RDS. * If you turn on point in time restore, then losing data stored in S3 is not a possibility worth worrying about on a practical level for…

>Cost

It's a bad trade. Thousands of hours of a high human capital computer scientist vs. a few tens of dollars a month for RDS.

>Reliability

Empirically false: none of this would have happened if Tarsnap used Postgres instead of a home-spun database.

Re: Tarsnap outage postmortem

#115

Post by baz00 »

tialaramex wrote: post #102
baz00 wrote: post #36

Earlier quoted context omitted.

As someone who just discovered my DR process does not work by testing it, 100% this. The only plan that is likely to work is a repeatable tested one.

Ideally, the thing you do in an emergency is largely routine, so that it happens by instinct rather than being a special case you need to remember. It should not be different in arbitrary ways. For example in both trains and cars, thanks to anti-lock braking, the correct way to stop the vehicle ASAP is to brake just like normal but as hard as you can, the computers will automatically solve the much trickier problem o…

This is terrible. Instinct cannot be trusted. Write it down.

Re: Tarsnap outage postmortem

#116

Post by Tepix »

mike_d wrote: post #12
cperciva wrote: post #3

blinks Ok, I really wasn't expecting this to land at the top of HN. I'd love to stick around to answer any questions people have, but it's 10PM and my toddler decided to go to bed at 5PM... so if I'm lucky I can get about 4 hours of sleep before she decides that it's time to get up. I'll check in and answer questions in the morning.

This was an extremely well written and thoughtful postmortem, but I hope to never see one from you again. :)

It was a postmortem without the mandatory "how can we prevent this in the future" steps…

Re: Tarsnap outage postmortem

#117

Post by nottorp »

JimDabell wrote: post #74
crossroadsguy wrote: post #37

Earlier quoted context omitted.

Well while I use Tarsnap for a very small amount of data (due to pricing) and I quite like Tarsnap for my use-case scenario, your question might have been downvoted due to two reasons: - your comment was a very valid question but rather quip-like, offhanded, seemed off etc etc. I mean something like that… - Tarsnap is an hn darling If I have to pick one I think it’s the latter :)

There’s also the fact that he has quite publicly been repeatedly nagged to raise his prices by another “HN darling” and he has been resistant to it. It’s actually quite an interesting read that covers a lot of the things brought up in this discussion: https://www.kalzumeus.com/2014/04/03/fantasy-tarsnap/

But basically what he suggests is turning Tarsnap into yet another SaaS offering I wouldn't give a second glance to. The screenhot of his proposal for the new entry page says... nothing.

Which is probably okay if you want to pivot from a geek-ish service to one that geeks don't use, of course. Does the owner want that?

Re: Tarsnap outage postmortem

#118

Post by kevincox »

Semaphor wrote: post #8
switch007 wrote: post #4

Not to be that guy, but it’s unreadable either zoomed in or in reader mode either horizontal or landscape on iOS. Colin, could the website be updated to the 2010s? :P

Just FYI, Firefox Reader mode works great with it.

"Great" is a bit of a stretch as there are random short lines where they were originally hard-wrapped. But it certainly does make it readable.

Re: Tarsnap outage postmortem

#119

Post by amluto »

stigz wrote: post #9
cperciva wrote: post #3

blinks Ok, I really wasn't expecting this to land at the top of HN. I'd love to stick around to answer any questions people have, but it's 10PM and my toddler decided to go to bed at 5PM... so if I'm lucky I can get about 4 hours of sleep before she decides that it's time to get up. I'll check in and answer questions in the morning.

Why would I use your service over restic? God bless you Colin, but reading this, it appears you're the only one in charge of the infrastructure for this service. I'm glad you're clear about no SLA, but this seems like a big liability between me and my backups.

tarsnap natively protects against inadvertent or malicious deletion or corruption — old tarsnap backups are immutablez The low-cost competitors (restic, borg, etc) seem to have this feature as an afterthought, and they make it surprisingly difficult.

(FWIW, S3 can be somewhat straightforwardly configured so that old data is effectively immutable. Google Cloud Storage’s similarly named versioning feature appears to be far weaker.)

Re: Tarsnap outage postmortem

#120

Post by VWWHFSfQ »

k8sToGo wrote: post #99
stigz wrote: post #9

Earlier quoted context omitted.

Why would I use your service over restic? God bless you Colin, but reading this, it appears you're the only one in charge of the infrastructure for this service. I'm glad you're clear about no SLA, but this seems like a big liability between me and my backups.

If the data is super important you should be setting on two different providers anyways for backups.

Honestly, whose data isn't "super important"?. All my data is super important. Even the crap I just throw on my Google drive. I want to keep it.

What is this mythical unimportant data that people still want to back up?

Post reply on HN