Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Tarsnap outage postmortem

mail.tarsnap.com

Post reply on HN
Print view View original

181–190 of 338 posts

Re: Tarsnap outage postmortem

#181

Post by jpgvm »

colonwqbang wrote: post #173

What would be the benefit of tarsnap over using something like restic+backblaze at order(s) of magnitude lower cost? What specific need would motivate you to pay $3000 per TB-year?

Extremely good deduplication means that for the core set of very important data I backup to Tarsnap the costs are negligible. I imagine the math is probably different if your data is changing more frequently. I for instance use other services to manage my video and photo libraries but my accounting databases, critical documents, etc are backed up to Tarsnap.

I have been using Tarsnap for a decade and not only has there been minimal availability issues there have been almost no issues of any kind that I can recall.

Re: Tarsnap outage postmortem

#182

Post by crossroadsguy »

e1g wrote: post #127
amluto wrote: post #119

Earlier quoted context omitted.

tarsnap natively protects against inadvertent or malicious deletion or corruption — old tarsnap backups are immutablez The low-cost competitors (restic, borg, etc) seem to have this feature as an afterthought, and they make it surprisingly difficult. (FWIW, S3 can be somewhat straightforwardly configured so that old data is effectively immutable. Google Cloud Storage’s similarly named versioning feature appears to be…

Yep, S3 is reasonably easy to configure for immutability. I personally use restic to send (encrypted) blobs to https://www.borgbase.com which has append-only mode and monitoring to warn me if some backups didn't happen.

borgbase is another “little” service that I use and like just like tarsnap and to some extent rsync.net. And they also have an excellent gui app Vorta (it’s FOSS; borgbase dev is the maintainer).

Re: Tarsnap outage postmortem

#183

Post by BowBun »

VWWHFSfQ wrote: post #120
k8sToGo wrote: post #99

Earlier quoted context omitted.

If the data is super important you should be setting on two different providers anyways for backups.

Honestly, whose data isn't "super important"?. All my data is super important. Even the crap I just throw on my Google drive. I want to keep it. What is this mythical unimportant data that people still want to back up?

I mean, you called it crap and then said it's super important. That's what hoarders say.

Subjectively you may feel that your data is super important, but objectively it probably isn't.

When people talk about 'super important' (totally a technical term), I think of things like DB backups in software companies, backups of financial reporting for firms, etc. Not your tax return from 2008.

Re: Tarsnap outage postmortem

#184

Post by jacquesm »

dinner wrote: post #68

[flagged]

What you see is someone who is actually willing and able to learn from any mistakes during this outage, no matter how small. That degree of attention to detail is exactly what I would expect from Colin. Novelty accounts created with the express purpose of slinging crap however are the equivalent of heckling in a theater, they don't contribute and in this case seem to be motivated by malice.

I do tech DD for a living and pretty much every company could do better if and when something goes wrong, but rarely do companies extract the maximum of learnings from an outage. That is what should impress you rather than to perceive it as a negative.

Note that most companies don't make any information about outages public and note that if and when they do it is usually heavily manipulated to make them look good. Colin could have easily done the same thing and the fact that he didn't deserves your respect, not your scorn. Consider the fact that even the best make mistakes. I'm aware of a very big name company that lost a ton of customer data through an interesting series of mishaps that all started with a routine test and not a peep on their website or in the media. Tens of thousands of people and hundreds of customers affected. And yet, you probably would trust them with your data precisely because they are not as honest as Tarsnap.

Re: Tarsnap outage postmortem

#185

Post by recov »

patrec wrote: post #77
RockRobotRock wrote: post #13

Aren't these storage prices absurd? Please let me know if I'm misunderstanding.

Everything about tarsnap is absurd. It's basically the world's most absurd backup service (insanely expensive, poor UX, bus factor of ~1, restoring moderate amounts of data appears to take days (!)[1]), brought about by an absurdly bad allocation of human capital (it's run by a double Putnam challenge winner, with several other impressive accomplishments), and as such, absurdly beloved by HN. [1] In case of an emerge…

http://www.daemonology.net/blog/2020-09-20-On-the-use-of-a-l...

Re: Tarsnap outage postmortem

#186

Post by jpgvm »

bombcar wrote: post #143
baz00 wrote: post #115

Earlier quoted context omitted.

This is terrible. Instinct cannot be trusted. Write it down.

There are two types of emergencies - checklist ones, and panic ones. You need to have both, but realize that in the panic ones people do NOT operate rationally. This is why house doors open in but business doors have to open out - if there’s a crush against a fire door it opens. You even see this in aviation, where everything is checkisted; the pilots will first stabilize the plane in an emergency and then run the ch…

Pilots are a little special, their panic mode is also a checklist, known as the memory items.

This doesn't work for normal people because normal people don't drill non-normal events until the response is instinctive.

Re: Tarsnap outage postmortem

#187

Post by justsomehnguy »

ilyt wrote: post #154
justsomehnguy wrote: post #94

Earlier quoted context omitted.

> absurd pricing for 2TB Well, it can't be that ba.. $0.25 x 2000 = $500 Yikes. And this is without BW costs. At $500/M you can just rent a dedicated physical server with a lot of HDDs and still have money left for your favourite pumpkin latte. For comparison rsync.net says it's $0.015 per GB/Mo, for 2TBs that's $30/m and no BW costs.

I'm renting $15/mo 2TB atom machine from OVH/kimsufi as second target for backups. Now that I think about it... some kind of micro-distributed backup server (throw on few of your machines, auto-replicate between) would be a neat project...

It's not even that neat.

Just slap rsync/syncthing to the backup dir.

Re: Tarsnap outage postmortem

#188

Post by carapace »

colonwqbang wrote: post #173

What would be the benefit of tarsnap over using something like restic+backblaze at order(s) of magnitude lower cost? What specific need would motivate you to pay $3000 per TB-year?

Some of us have lots of extra money and like an excuse to give some of it to cperciva so he doesn't have to work a shit job and can apply his skills and talents to bigger, better things?

(People here asking about the low Bus Factor: you don't keep your backups in one service/location, eh? You use Tarsnap and Restic with Backblaze, Rsync.net, S3, etc. right? "Backups are a tax you pay for the luxury of restore.")

Re: Tarsnap outage postmortem

#189

Post by tialaramex »

bluGill wrote: post #172
bombcar wrote: post #143

Earlier quoted context omitted.

There are two types of emergencies - checklist ones, and panic ones. You need to have both, but realize that in the panic ones people do NOT operate rationally. This is why house doors open in but business doors have to open out - if there’s a crush against a fire door it opens. You even see this in aviation, where everything is checkisted; the pilots will first stabilize the plane in an emergency and then run the ch…

Note that in airplanes (unlike cars) you normally cannot just get in a new one and fly. You first get training on that particular plane. If everything goes perfect any pilot can get in any plane and fly it, but if any little thing goes wrong they better know how the plane flys very well so they can get it stable enough to run the checklist.

You probably shouldn't just get in a new car and drive it, but people do. I remember at a hire car place once the team I worked with were given an automatic, the guy driving has never driven an automatic transmission before, but his license authorises it (UK licenses allow everybody to drive an automatic, but you need to test in a manual to drive manual), and so they just lent him a car with a completely different driving style. He had to get them to show him how to even drive it away out of their car park.

I learned in the small car from the same brand as my father's larger car, so that the controls are in the same place, the symbols on stuff are identical, all that was different once I have a license and borrow dad's car is it's longer and has more power.

It also probably shouldn't be legal for me to drive today, but it is. I learned 25 years ago, and I haven't driven anything in over a decade, so a rational system would say nah, you're too rusty, get a refresher course, but there's no mandate for that.

Re: Tarsnap outage postmortem

#190

Post by Roark66 »

VWWHFSfQ wrote: post #120
k8sToGo wrote: post #99

Earlier quoted context omitted.

If the data is super important you should be setting on two different providers anyways for backups.

Honestly, whose data isn't "super important"?. All my data is super important. Even the crap I just throw on my Google drive. I want to keep it. What is this mythical unimportant data that people still want to back up?

>What is this mythical unimportant data that people still want to back up?

I have lots of stuff like this. Often it is easier to just back up an entire folder than go through sub/sub folders separating stuff into: important, not very important. Storage costs are low enough to just backup everything (almost). Also, one often doesn't know what may be important/useful in future. For example a couple of years ago I had this huge buildroot system (600gb) to build firmware images for a single board computer I spent quite a while to put together. The project I was doing it for got cancelled so I had no need to keep it. Still I wish I did, as I'd love to be able to tinker with it now, but 600gb is not a trivial amount to store so it got deleted. Most of this data was pulled from various online resources that don't exist anymore too.

What's the morale of my story? If you have a fast internet connection (I don't) backup "everything" to cloud. Then find "really important stuff" like the pictures of your children etc and back it up again to a different cloud.

If you're in a middle of nowhere on a slow LTE connection like me, building a nas box is not a bad idea for backups.

Post reply on HN