Live data from Hacker News

Tarsnap performance issues in late March, most of April

mail.tarsnap.com

31–40 of 116 posts

Re: Tarsnap performance issues in late March, most of April

#31

Earlier quoted context omitted.

Because tarsnap is cheap, incredibly well documented, open source, and run by an awesome guy. It's an all around win-win.

Rsync.net is even cheaper, has no requirement for a custom client, and arguably are more dependable because they're not just reselling S3 Edit: not to mention they offer actual support not just "contact the author" email link as a last resort.

I personally just trust Colin's crypto skills more than anyone.

Re: Tarsnap performance issues in late March, most of April

#32
post #3

In case any other customer is wondering "Wait, I didn't hear anything from my monitoring about that and I'm retroactively worried. How worried should I be?" like I was: I just pulled our logs and reconstructed them, and it shows over the last ~30 days that the worse-case performance of our daily backup (~150 MB per day delta, ~45 GB total post deduplication) was about 40% longer than our typical case. This didn't tri…

We actually have our Chef rdiff backup cookbook randomly distribute jobs across a buckets of time using a hash function of the hostname.

Re: Tarsnap performance issues in late March, most of April

#33
post #3

In case any other customer is wondering "Wait, I didn't hear anything from my monitoring about that and I'm retroactively worried. How worried should I be?" like I was: I just pulled our logs and reconstructed them, and it shows over the last ~30 days that the worse-case performance of our daily backup (~150 MB per day delta, ~45 GB total post deduplication) was about 40% longer than our typical case. This didn't tri…

I ran into this recently, backing up munin data to s3. I ran it at a time point offset from an hour to avoid those 'on-the-hour' rushes, but I was getting problems with the copy. Took me a moment to realise I was doing it on a 5-minute boundary, and munin fires on a 5-minute boundary - the data was being updated as I was copying it...

mental note: think harder, next time.

Re: Tarsnap performance issues in late March, most of April

#34
post #31

Earlier quoted context omitted.

Rsync.net is even cheaper, has no requirement for a custom client, and arguably are more dependable because they're not just reselling S3 Edit: not to mention they offer actual support not just "contact the author" email link as a last resort.

I personally just trust Colin's crypto skills more than anyone.

So you're saying you trust a single developer to both write an encryption tool and run the servers it talks to more than the combined possibilities using existing open source tools to create backups by encrypting data locally and storing it remotely via ssh/sftp?

Re: Tarsnap performance issues in late March, most of April

#35
post #18

Sorry if this is offtopic, but can anybody explain the value proposition of tarsnap to me? It seems like a nice service and all, but the pricing is an order of magnitude more expensive than S3. If you are storing a few GB, this might not matter ("over half of Tarsnap users spend under $1 per month on storing their backups"), but if you have that little data, why not just dump it on a free Dropbox/Gdrive/etc account?…

Because tarsnap is cheap, incredibly well documented, open source, and run by an awesome guy. It's an all around win-win.

tarsnap is not open source:

"While the Tarsnap code itself has not been released under an open source license, some of the "reusable components" have been published separately under a BSD license"

http://www.tarsnap.com/oss.html

The source code for tarsnap is available to view, so you could audit/inspect it yourself, but it is not under an open source license.

Re: Tarsnap performance issues in late March, most of April

#36
As an AWS user this type of thing gives me cause for concern:

At 2015-04-01 00:00 UTC, the Amazon EC2 "provisioned I/O" volume on which most of this metadata was stored suddenly changed from an average latency of 1.2 ms per request to an average latency of 2.2 ms per request. I have no idea why this happened -- indeed, I was so surprised by it that I didn't believe Amazon's monitoring systems at first -- but this immediately resulted in the service being I/O limited.

A sudden doubling of latency can have dire consequences on any system. Knowing that such unexpected changes are possible makes it built trust in your environment, even if it is running fine today.

Re: Tarsnap performance issues in late March, most of April

#37

Earlier quoted context omitted.

Rsync.net is even cheaper, has no requirement for a custom client, and arguably are more dependable because they're not just reselling S3 Edit: not to mention they offer actual support not just "contact the author" email link as a last resort.

I contacted the author today. He responded to me in 30 seconds.

Try in 18 hours. Can you call him when something fails?

I'm not saying he isn't responsive I'm saying depending on a one-man-band who is responsible for the client software, server software and the underlying storage system (ie he is the owner of the s3 account) seems like a huge risk.

Re: Tarsnap performance issues in late March, most of April

#38
post #6
post #5

Earlier quoted context omitted.

One way to think about your fear is, shouldn't that just be a tarsnap feature? Add some metadata for a machine that tarsnap should expect a once a day/week/month backup from this machine, and if it doesn't get one, to send you an email?

whistles Until the day when Colin considers it in-scope for Tarsnap, I recommend Deadman's Snitch for this purpose. I literally spend more on DMS to monitor Tarsnap than I spend on Tarsnap. No, I don't think that is just, either.

Don't you have some other servers running other services? So you must already have some monitoring and alerting system like Nagios, to which you can add one more little "passive check" that does the same thing, for no incremental cost?

Re: Tarsnap performance issues in late March, most of April

#39

Earlier quoted context omitted.

Rsync.net is even cheaper, has no requirement for a custom client, and arguably are more dependable because they're not just reselling S3 Edit: not to mention they offer actual support not just "contact the author" email link as a last resort.

I contacted the author today. He responded to me in 30 seconds.

I assume he still has to sleep, at least on some days. :)

Re: Tarsnap performance issues in late March, most of April

#40
post #3

In case any other customer is wondering "Wait, I didn't hear anything from my monitoring about that and I'm retroactively worried. How worried should I be?" like I was: I just pulled our logs and reconstructed them, and it shows over the last ~30 days that the worse-case performance of our daily backup (~150 MB per day delta, ~45 GB total post deduplication) was about 40% longer than our typical case. This didn't tri…

Hear hear on said Old Wizened Graybeard habit. The amount of pain inflicted from twenty jobs all starting up at :00 (or even :30, :45, etc.) when they could easily run at :04 or :17 can be huge. Anecdotally I once "lost" a sandbox server to a ton of developer sandbox jobs starting at :00 and not completing before the next batch started.

Don't run on :17 and :39. Those are mine. Thanks!
Post reply on HN