Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Tarsnap outage postmortem

mail.tarsnap.com

Post reply on HN
Print view View original

151–160 of 338 posts

Re: Tarsnap outage postmortem

#151

Post by ilyt »

mst wrote: post #86
RockRobotRock wrote: post #20

Earlier quoted context omitted.

Can you not be snide and please help me understand? It seems 50 times more expensive than B2. I'm genuinely curious about the product.

Roughly: The number of hours of my time that would be required to get something with even theoretically equivalent features would be sufficient to make the cost - and opportunity cost - involved seem far more reasonable. Plus "written by cperciva and heavily battle tested by Serious Sysadmins" is a feature I couldn't recreate myself - notice that while there was an outage, part of the reason for it taking a while was…

> notice that while there was an outage, part of the reason for it taking a while was a conscious choice to take a much longer path to resolution than bringing up the previous server in the name of paranoia. Paranoia about data corruption is a nice thing to have in a backup system and something I'm happily willing to trade-off uptime for.

As Actual Serious Sysadmin that Actually Manages Big Systems for Living that screams more lack of preparation than anything else.

Yes you should be careful but you should also have procedures in place and know the system well enough to trust it. And the fact is that the "boring" architecture of RDS DB instead of that S3 database abomination thing would just start right up if master DB server failed.

It honestly looks like a trap many intelligent people fall into where they turn their cool-but-ulimately-flawed mental excercise into bedrock of the product. I don't want to use baby's-first-database on my production servers (I'm looking at you Lennart Poettering and journald) and I don't want my data/metadata stored on some experimental one.

Re: Tarsnap outage postmortem

#152

Post by ilyt »

alanfranz wrote: post #30
yjftsjthsd-h wrote: post #24

Earlier quoted context omitted.

Then the obvious question is "why would I use this instead of something else over S3" (ex. rclone), to which I think the answer is ease of use (don't need to deal with AWS yourself, encryption/deduplication/compression handled for you, nice interface), which isn't everything to everyone but is certainly useful.

You need a remote service that keeps backup readonly. You’re not covering attack scenarios if you just use raw object storage from your client machine. I have written about this some time ago if you’re interested: https://www.franzoni.eu/ransomware-resistant-backups/

How tarsnap keeps backups readonly ? Just having service being a barrier is not enough

Re: Tarsnap outage postmortem

#153

Post by zetalyrae »

bastawhiz wrote: post #150
zetalyrae wrote: post #138

Earlier quoted context omitted.

>S3 is the most reliable data storage mechanism in the world S3 is not the problem here. The problem is building a database on top of S3 , and having to reimplement all the consistency, atomicity, transactions etc. on top. >no thought to a schema, no migrations to manage There is, in fact, always a schema. Some people choose to ignore it's there, to their detriment. >Always avoid the fancy option when it makes sense.…

> having to reimplement all the consistency, atomicity, transactions etc. on top. Most of those problems are moot if you're only ever writing from a single head node. If all your data is strictly ordered and you have no meaningful concurrency, this is a far, far simpler system.

Did I fall into a timewarp into the 70's? How on Earth, by what sane standard, is a Postgres instance too complex? If you're `fopen`ing files as a "database" you are wasting your time and lowering the world's economic productivity.

Complex is Greenspunning a database and having it blow up in your face and cause a twenty-six hour outage. You never hear about such things with Postgres because Postgres is rock-solid.

Re: Tarsnap outage postmortem

#154

Post by ilyt »

justsomehnguy wrote: post #94
aborsy wrote: post #79

Earlier quoted context omitted.

I backup around 2TB with Restic, also tried locally with Borg. The size is nearly the same. Sadly, I can’t even test with Tarsnap! (absurd pricing for 2TB).

> absurd pricing for 2TB Well, it can't be that ba.. $0.25 x 2000 = $500 Yikes. And this is without BW costs. At $500/M you can just rent a dedicated physical server with a lot of HDDs and still have money left for your favourite pumpkin latte. For comparison rsync.net says it's $0.015 per GB/Mo, for 2TBs that's $30/m and no BW costs.

I'm renting $15/mo 2TB atom machine from OVH/kimsufi as second target for backups.

Now that I think about it... some kind of micro-distributed backup server (throw on few of your machines, auto-replicate between) would be a neat project...

Re: Tarsnap outage postmortem

#155

Post by tinco »

patrec wrote: post #77
RockRobotRock wrote: post #13

Aren't these storage prices absurd? Please let me know if I'm misunderstanding.

Everything about tarsnap is absurd. It's basically the world's most absurd backup service (insanely expensive, poor UX, bus factor of ~1, restoring moderate amounts of data appears to take days (!)[1]), brought about by an absurdly bad allocation of human capital (it's run by a double Putnam challenge winner, with several other impressive accomplishments), and as such, absurdly beloved by HN. [1] In case of an emerge…

If tarsnap has even a modicum of popularity, thanks to these prices it would be bringing in bank. If he's making bank, that means he's providing value (even if it's just to "uninformed"). And it seems this system mostly runs itself, so it's a side gig. It's probably a far more effective allocation than many other possible allocations of human capital.

How many of the world's best and brightest are doing all sorts of busywork? At least Colin has some time to do whatever he wants to do while running tarsnap.

Re: Tarsnap outage postmortem

#156

Post by deltarholamda »

verytrivial wrote: post #60

(caveat: I may be running on old tarsnap company info but) I must say, the ONLY thing that has ever made me shy away from seriously using tarsnap was the prospect of an unexpected Colin Percival outage. i.e. key person risk. I'm guessing I'm not alone in this.

While I get this as a risk, I'm not convinced it's any more risky than a larger corporate entity.

This is Colin's job. Colin has his name attached to it. It's really important to Colin.

You're not going to get the same kind of service from BigBackupCorp. Their employees are replaceable, their management is replaceable, and to be honest, you as a customer are replaceable, if they decide to move in a different direction and become BigFlowerArrangementShippingCorp.

The neat thing about a small business is that it runs entirely on its own profits. There are no stock price games or VC jiggery-pokery or anything like that. If it's a profitable business, there will be somebody to come along and take it over and make it their job with their name attached to it. I think the open Internet benefits a lot from this sort of thing.

Re: Tarsnap outage postmortem

#157

Post by ilyt »

throwawaaarrgh wrote: post #139
ilyt wrote: post #132

Earlier quoted context omitted.

The "fancy" option here is trying to act S3 to act like database instead of simple blob storage...

Fancy would be writing an application to talk to RDS, creating an RDS instance, creating a database, creating whatever IAM link is needed for auth into the db so you don't need a second set of credentials, creating a schema, creating columns with different data types, and then modifying the application to handle edge cases for the different data types, logic to insert, update, delete rows, select items, yadda yadda y…

I envy job where you think a database is something fancy and hard to do...

Re: Tarsnap outage postmortem

#158

Post by cowsandmilk »

deltarholamda wrote: post #156
verytrivial wrote: post #60

(caveat: I may be running on old tarsnap company info but) I must say, the ONLY thing that has ever made me shy away from seriously using tarsnap was the prospect of an unexpected Colin Percival outage. i.e. key person risk. I'm guessing I'm not alone in this.

While I get this as a risk, I'm not convinced it's any more risky than a larger corporate entity. This is Colin's job. Colin has his name attached to it. It's really important to Colin. You're not going to get the same kind of service from BigBackupCorp. Their employees are replaceable, their management is replaceable, and to be honest, you as a customer are replaceable, if they decide to move in a different directio…

It isn’t necessarily about Colin quitting. Key person gets hit by bus is also always a concern. You can say someone will pick it up, but I know nothing of whether such plans are in place. Does the person who would inherit the business have the know how to sell it? Is there enough documentation in place for a transfer of assets to be successful?

Re: Tarsnap outage postmortem

#159

Post by ivanhoe »

stigz wrote: post #9
cperciva wrote: post #3

blinks Ok, I really wasn't expecting this to land at the top of HN. I'd love to stick around to answer any questions people have, but it's 10PM and my toddler decided to go to bed at 5PM... so if I'm lucky I can get about 4 hours of sleep before she decides that it's time to get up. I'll check in and answer questions in the morning.

Why would I use your service over restic? God bless you Colin, but reading this, it appears you're the only one in charge of the infrastructure for this service. I'm glad you're clear about no SLA, but this seems like a big liability between me and my backups.

It's a pretty well-known fact for years that tarsnap is basically a one-man show, and yet Colin has managed to provide fantastic service so far. Sometimes having ppl who built the service also managing it is actually a big plus, compared to other services where you first have to fight through outsourced & underpaid support that's limited to template answers, only to finally get some "engineer" who got that job 2 months ago and is more clueless on their system than myself...

Re: Tarsnap outage postmortem

#160

Post by 88913527 »

stigz wrote: post #9
cperciva wrote: post #3

blinks Ok, I really wasn't expecting this to land at the top of HN. I'd love to stick around to answer any questions people have, but it's 10PM and my toddler decided to go to bed at 5PM... so if I'm lucky I can get about 4 hours of sleep before she decides that it's time to get up. I'll check in and answer questions in the morning.

Why would I use your service over restic? God bless you Colin, but reading this, it appears you're the only one in charge of the infrastructure for this service. I'm glad you're clear about no SLA, but this seems like a big liability between me and my backups.

Even large organizations can have fairly regular availability issues. I appreciate the noted flaws of "single point of failure", but I also see orgs where 100s of people have access to the infrastructure, make a change, and then it breaks something. I wouldn't do business with an org just because they have many people, that won't mean they're operationally sound, at least not to my expectations.
Post reply on HN