Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Tarsnap performance issues in late March, most of April

mail.tarsnap.com

Post reply on HN
Print view View original

41–50 of 116 posts

Re: Tarsnap performance issues in late March, most of April

#41

Post by lucb1e »

ac29 wrote: post #29
segf4ult wrote: post #23

Earlier quoted context omitted.

Because tarsnap is cheap, incredibly well documented, open source, and run by an awesome guy. It's an all around win-win.

But its not cheap, which was my point. 100GB of storage costs: $300/year at tarsnap $36/year at S3

Finally, numbers other than picodollars and gigabyte months and unpredictable deduplication. This convinces me I don't want to store 4TB there at a huge cost($12,000 if it's really $300 a year for 100GB) compared to buying two 4TB drives (~€250 per 3-4 years) and placing them at a friend's with free bandwidth.

Don't get me wrong: managed, off-site encrypted backups are very attractive, and I might be willing to pay a premium, especially for software from a trusted person, but not the cost price hundredfold.

Re: Tarsnap performance issues in late March, most of April

#42

Post by dmix »

stephenr wrote: post #34
dmix wrote: post #31

Earlier quoted context omitted.

I personally just trust Colin's crypto skills more than anyone.

So you're saying you trust a single developer to both write an encryption tool and run the servers it talks to more than the combined possibilities using existing open source tools to create backups by encrypting data locally and storing it remotely via ssh/sftp?

Yes, when it comes to crypto I'd put my in trust in highly talented people over trusting my own ability to glue together a collection of OSS tools anyday.

Re: Tarsnap performance issues in late March, most of April

#43

Post by sillysaurus3 »

jldugger wrote: post #32
patio11 wrote: post #3

In case any other customer is wondering "Wait, I didn't hear anything from my monitoring about that and I'm retroactively worried. How worried should I be?" like I was: I just pulled our logs and reconstructed them, and it shows over the last ~30 days that the worse-case performance of our daily backup (~150 MB per day delta, ~45 GB total post deduplication) was about 40% longer than our typical case. This didn't tri…

We actually have our Chef rdiff backup cookbook randomly distribute jobs across a buckets of time using a hash function of the hostname.

I have to know: Why a hash function of the hostname?

Re: Tarsnap performance issues in late March, most of April

#44

Post by Spooky23 »

ac29 wrote: post #18

Sorry if this is offtopic, but can anybody explain the value proposition of tarsnap to me? It seems like a nice service and all, but the pricing is an order of magnitude more expensive than S3. If you are storing a few GB, this might not matter ("over half of Tarsnap users spend under $1 per month on storing their backups"), but if you have that little data, why not just dump it on a free Dropbox/Gdrive/etc account?…

It depends on the data you are backing up. For dedupe friendly data, the costs come down significantly.

Re: Tarsnap performance issues in late March, most of April

#45

Post by cperciva »

k1w1 wrote: post #36

As an AWS user this type of thing gives me cause for concern: At 2015-04-01 00:00 UTC, the Amazon EC2 "provisioned I/O" volume on which most of this metadata was stored suddenly changed from an average latency of 1.2 ms per request to an average latency of 2.2 ms per request. I have no idea why this happened -- indeed, I was so surprised by it that I didn't believe Amazon's monitoring systems at first -- but this imm…

Indeed, I didn't know such a change was possible -- that EBS volume went for years with consistent low latency before it suddenly slowed down.

Re: Tarsnap performance issues in late March, most of April

#46

Post by jballanc »

Poiesis wrote: post #10
mtsmith85 wrote: post #2

This line: I would have sent out an email to the mailing lists earlier; but since at each point I thought I was "one change away" from fixing the problems, I kept on delaying said email until it was clear that the problems were finally fixed" is such a common situation for most people, but I tend to see it with engineers especially. I find I struggle with it an incredible amount. In some ways, I guess it seems health…

I came here to write this comment essentially. Very much of the time I feel, "If I knew what the problem[s] [was|were] it'd be solved by now!" That's not exactly true of course but of course diagnosis is a large part of the total solution. This type of an answer that Colin gave above does not exactly win friends and influence people in most situations where you're part of a team or hierarchy. Can anyone share what th…

One trick that I've learned (though I still have trouble routinely applying it myself) for these situations is: less is more.

That is, as engineers we tend to want details. All the details. We want to know what happened, why it happened, how it's going to be fixed, and how long that will take. Because we want all that detail for ourselves, we hesitate to contact our customers/boss until we have all the details. Combine that with a desire to fix problems as they come up, and you end up with, "I never told you there was a problem because I was always one fix away from the solution."

But most people are not engineers. They want to be acknowledged. They want to feel informed, even if they have less details than what you would like to provide for them. Sometimes, something as simple as, "We've noticed that there is an issue and are currently working on a fix," goes a long way. Also don't be afraid to pull out, "Users have been reporting issues with backup performance. We do not currently believe this represents a service failure, but we are working to return performance to normal levels."

Your users trust you (otherwise they wouldn't pay you). If you "believe" something, they will too.

Re: Tarsnap performance issues in late March, most of April

#47

Post by kijin »

protomyth wrote: post #8
mtsmith85 wrote: post #4

Earlier quoted context omitted.

Hear hear on said Old Wizened Graybeard habit. The amount of pain inflicted from twenty jobs all starting up at :00 (or even :30, :45, etc.) when they could easily run at :04 or :17 can be huge. Anecdotally I once "lost" a sandbox server to a ton of developer sandbox jobs starting at :00 and not completing before the next batch started.

Funny part to that, was on a project with multiple teams with multiple crontabs. Each team took that advice to heart for some jobs. Sadly, we had too many Hitchhiker fans and :42 became a bit too common.

Use the following shell command to decide when to run cron jobs.

    echo $((RANDOM % 60))
It's not a CSPRNG, but good enough for this kind of load balancing!

Re: Tarsnap performance issues in late March, most of April

#48

Post by stephenr »

dmix wrote: post #42
stephenr wrote: post #34

Earlier quoted context omitted.

So you're saying you trust a single developer to both write an encryption tool and run the servers it talks to more than the combined possibilities using existing open source tools to create backups by encrypting data locally and storing it remotely via ssh/sftp?

Yes, when it comes to crypto I'd put my in trust in highly talented people over trusting my own ability to glue together a collection of OSS tools anyday.

You seem to have misunderstood me.

I didn't suggest you should write your own encryption tool. There are numerous open source tools for creating encrypted backups, some do deduplication first too.

If the tool doesn't happen to support remote storage, a simple rsync or scp fills that part.

Literally the only thing unique about this service is the use of the term picodollars and the single individual it's all reliant on.

Re: Tarsnap performance issues in late March, most of April

#49

Post by stephenr »

lucb1e wrote: post #41
ac29 wrote: post #29

Earlier quoted context omitted.

But its not cheap, which was my point. 100GB of storage costs: $300/year at tarsnap $36/year at S3

Finally, numbers other than picodollars and gigabyte months and unpredictable deduplication. This convinces me I don't want to store 4TB there at a huge cost($12,000 if it's really $300 a year for 100GB) compared to buying two 4TB drives (~€250 per 3-4 years) and placing them at a friend's with free bandwidth. Don't get me wrong: managed, off-site encrypted backups are very attractive, and I might be willing to pay a…

Add another $1k to that for data transfer (assuming you only upload that 4tb once)

Re: Tarsnap performance issues in late March, most of April

#50

Post by cperciva »

JacobAldridge wrote: post #28
cperciva wrote: post #21

I suppose I should have known that this would end up at the top of Hacker News...

It's the picodollars - tarsnap was the second business I fell in love with on HN (the late Kiko was #1) purely because of the awesome vibe I felt emanating from your enterprise (which I'm assuming is a reflection on you as well). Years later, you've also become a cause celebre for holding true to a clear business and lifestyle vision (again, perceived at distance), in spite of the recommendations and 'support' provid…

It's the picodollars

Hey Thomas, are you listening here?

In all seriousness, the picodollars do an excellent job of attracting exactly the sort of customers I want... and turning away the customers I don't want. They were originally part joke and part a way to avoid arguments with customers who don't understand that 1 GB in spite of the recommendations and 'support' provided by Patrick and others

Don't be too harsh on Patrick. His vision for Tarsnap is not my vision for Tarsnap, but he has helped me to orient myself: The projection of "business" onto the subspace "geek" doesn't look very much like "business", but it's not the same as "kid right out of university who has never had a real job" either, and that's what you would see if I hadn't had advice (from Patrick, Thomas, various YC people, and the rest of HN).

Advice can be very valuable even if you don't follow it to the letter.

Post reply on HN