Tarsnap performance issues in late March, most of April • Hacker News Board

Re: Tarsnap performance issues in late March, most of April

#111

Post by stephenr » Wed, May 06, 2015, 7:45 PM UTC

girvo wrote: post #90

lucb1e wrote: post #41

Earlier quoted context omitted.

Finally, numbers other than picodollars and gigabyte months and unpredictable deduplication. This convinces me I don't want to store 4TB there at a huge cost($12,000 if it's really $300 a year for 100GB) compared to buying two 4TB drives (~€250 per 3-4 years) and placing them at a friend's with free bandwidth. Don't get me wrong: managed, off-site encrypted backups are very attractive, and I might be willing to pay a…

That sort of backup is what AWS Glacier is for, is it not?

I hate to think the cost if you had to restore that data from glacier though.

Re: Tarsnap performance issues in late March, most of April

#112

Post by cperciva » Wed, May 06, 2015, 8:07 PM UTC

JoachimSchipper wrote: post #89

cperciva wrote: post #52

Earlier quoted context omitted.

Or schedule your cron job for :00, but add "sleep `jot -r 1 0 3600` &&" to the start of the command. (jot is a BSDism, but I assume you can do the same with GNU seq.)

Don't use that for hourly jobs, though - things are liable to break when you randomly run a command at, say, 12:59 and 13:00.

Right, I usually do that for my daily jobs.

Re: Tarsnap performance issues in late March, most of April

#113

Post by cperciva » Wed, May 06, 2015, 8:08 PM UTC

gonzo wrote: post #109

cperciva wrote: post #17

Earlier quoted context omitted.

I'd love some context (maybe from cperciva himself?) around the performance enhancement of integrating new Intel AESNI instructions. I was using OpenSSL for that (which was using a software implementation). The code (you can see it in spiped) now detects the CPU feature and selects between AESNI or OpenSSL automatically. Given that the tarsnap server code was spending about 40% of its time running AES, it's a nontriv…

One wonders why you aren't using a version of OpenSSL that has the AESNI bits already in it.

I've learned that wondering about OpenSSL internals is detrimental to my sanity.

Re: Tarsnap performance issues in late March, most of April

#114

Post by jldugger » Wed, May 06, 2015, 9:33 PM UTC

sillysaurus3 wrote: post #43

jldugger wrote: post #32

Earlier quoted context omitted.

We actually have our Chef rdiff backup cookbook randomly distribute jobs across a buckets of time using a hash function of the hostname.

I have to know: Why a hash function of the hostname?

It's more or less random, but stable.

Re: Tarsnap performance issues in late March, most of April

#115

Post by scott_karana » Wed, May 06, 2015, 10:23 PM UTC

middleclick wrote: post #91

Osiris wrote: post #26

For those that want to run a similar service using their own systems, I found that Attic [1] is a great open source backup tool that works in a very similar way, including deduplication and compression. I backup some VPS servers to my NAS at home using attic over an SSH tunnel. Incremental backups are quite small and it's easy to automate with a simple cron job. [1] https://attic-backup.org/

How does this compare to Duplicity?

It uses git-style addressable blob storage, so you don't have to worry about deltas, because there aren't any.

It's also got more efficient deduplication, because it doesn't use rsync's naïve algorithm.

The downsides: it requires the agent to be remotely installed (a la rsync: no "dumb" backends), and supports less storage backends to boot.

YMMV :-)

Re: Tarsnap performance issues in late March, most of April

#116

Post by scott_karana » Wed, May 06, 2015, 10:29 PM UTC

ac29 wrote: post #18

Sorry if this is offtopic, but can anybody explain the value proposition of tarsnap to me? It seems like a nice service and all, but the pricing is an order of magnitude more expensive than S3. If you are storing a few GB, this might not matter ("over half of Tarsnap users spend under $1 per month on storing their backups"), but if you have that little data, why not just dump it on a free Dropbox/Gdrive/etc account?…

Tarsnap can have different write/read keys for each backup archive, so (unlike with Attic) you don't need to worry about a compromised host deleting its entire backup history.

Other than that, Attic is pretty excellent too.