what the fu....Colin Percival used to be my cello teacher 12 years ago....and he is running tarsnap. My mind is blown.
Tarsnap performance issues in late March, most of April
11–20 of 116 posts
Re: Tarsnap performance issues in late March, most of April
#12This line: I would have sent out an email to the mailing lists earlier; but since at each point I thought I was "one change away" from fixing the problems, I kept on delaying said email until it was clear that the problems were finally fixed" is such a common situation for most people, but I tend to see it with engineers especially. I find I struggle with it an incredible amount. In some ways, I guess it seems health…
I would have sent out an email to the mailing lists earlier; but since at each point I thought I was "one change away" from fixing the problems, I kept on delaying said email until it was clear that the problems were finally fixed
This ties in to the last lesson I mentioned at the bottom:
5. When performance drops, it's not always due to a single problem; sometimes there are multiple interacting bottlenecks.
Every time I identified a problem, I was correct that it was a problem -- my failing was in not realizing that there were several things going on at once.
Re: Tarsnap performance issues in late March, most of April
#13In case any other customer is wondering "Wait, I didn't hear anything from my monitoring about that and I'm retroactively worried. How worried should I be?" like I was: I just pulled our logs and reconstructed them, and it shows over the last ~30 days that the worse-case performance of our daily backup (~150 MB per day delta, ~45 GB total post deduplication) was about 40% longer than our typical case. This didn't tri…
the worse-case performance of our daily backup (~150 MB per day delta, ~45 GB total post deduplication) was about 40% longer than our typical case
Yes, that sounds about right. I had maybe half a dozen people write to me who had noticed performance problems, and after the initial "backups failed because the server hit its connection limit" issue, it was people whose backups were already very long-running -- if your daily backups normally take 20 hours to complete, a 40% slowdown is painful.
Re: Tarsnap performance issues in late March, most of April
#14This line: I would have sent out an email to the mailing lists earlier; but since at each point I thought I was "one change away" from fixing the problems, I kept on delaying said email until it was clear that the problems were finally fixed" is such a common situation for most people, but I tend to see it with engineers especially. I find I struggle with it an incredible amount. In some ways, I guess it seems health…
I would have sent out an email to the mailing lists earlier; but since at each point I thought I was "one change away" from fixing the problems, I kept on delaying said email until it was clear that the problems were finally fixed This ties in to the last lesson I mentioned at the bottom: 5. When performance drops, it's not always due to a single problem; sometimes there are multiple interacting bottlenecks. Every ti…
That totally jibes with what I found "reassuring" in a sense. That even very smart people sometimes get hit with inadvertent "multiple problems looking like a single issue" situations.
Re: Tarsnap performance issues in late March, most of April
#15what the fu....Colin Percival used to be my cello teacher 12 years ago....and he is running tarsnap. My mind is blown.
Not me -- I'm a violinist, and I've never taught anyone violin either.
Maybe you're thinking of my bother (Graham)? He was teaching cello around that time period I think.
Re: Tarsnap performance issues in late March, most of April
#16what the fu....Colin Percival used to be my cello teacher 12 years ago....and he is running tarsnap. My mind is blown.
Not me -- I'm a violinist, and I've never taught anyone violin either. Maybe you're thinking of my bother (Graham)? He was teaching cello around that time period I think.
Oh wow. Yes! Now I remember it was Graham. I was his student around then. Saw your photo on twitter and looked exactly like him! Hahaha. How is Graham?
My mind is just completely blown right now.
Re: Tarsnap performance issues in late March, most of April
#17This line: I would have sent out an email to the mailing lists earlier; but since at each point I thought I was "one change away" from fixing the problems, I kept on delaying said email until it was clear that the problems were finally fixed" is such a common situation for most people, but I tend to see it with engineers especially. I find I struggle with it an incredible amount. In some ways, I guess it seems health…
I'd love some context (maybe from cperciva himself?) around the performance enhancement of integrating new Intel AESNI instructions.
I was using OpenSSL for that (which was using a software implementation). The code (you can see it in spiped) now detects the CPU feature and selects between AESNI or OpenSSL automatically. Given that the tarsnap server code was spending about 40% of its time running AES, it's a nontrivial CPU time saving.
I should probably have been clearer in my writeup though -- using AESNI was never a "once I roll this out everything will be good" fix. Rather, it was a case of "I have this well-tested code available which will help a bit while I finish testing the real fixes".
Re: Tarsnap performance issues in late March, most of April
#18Sorry if this is offtopic, but can anybody explain the value proposition of tarsnap to me? It seems like a nice service and all, but the pricing is an order of magnitude more expensive than S3. If you are storing a few GB, this might not matter ("over half of Tarsnap users spend under $1 per month on storing their backups"), but if you have that little data, why not just dump it on a free Dropbox/Gdrive/etc account?
For more data, why not just use one of the many compressed, deduplicated, encrypted, incremental backup systems (attic comes to mind, I'm sure there are others) then just sync to S3 at a tenth the cost?
Re: Tarsnap performance issues in late March, most of April
#19Earlier quoted context omitted.
Not me -- I'm a violinist, and I've never taught anyone violin either. Maybe you're thinking of my bother (Graham)? He was teaching cello around that time period I think.
Oh wow. Yes! Now I remember it was Graham. I was his student around then. Saw your photo on twitter and looked exactly like him! Hahaha. How is Graham? My mind is just completely blown right now.
Graham is in Japan right now, but returning to Canada soon; he'll be joining me in the West Coast Symphony for our June concert.
I don't feel that I should really be talking too much about my family in a public forum, but if you'd like to send me an email I can forward it to him.
Re: Tarsnap performance issues in late March, most of April
#20In case any other customer is wondering "Wait, I didn't hear anything from my monitoring about that and I'm retroactively worried. How worried should I be?" like I was: I just pulled our logs and reconstructed them, and it shows over the last ~30 days that the worse-case performance of our daily backup (~150 MB per day delta, ~45 GB total post deduplication) was about 40% longer than our typical case. This didn't tri…
the worse-case performance of our daily backup (~150 MB per day delta, ~45 GB total post deduplication) was about 40% longer than our typical case Yes, that sounds about right. I had maybe half a dozen people write to me who had noticed performance problems, and after the initial "backups failed because the server hit its connection limit" issue, it was people whose backups were already very long-running -- if your d…
I run my backups overnight and get a status email each morning, and I didn't even realise there were performance issues until now. As you said, unless you run your backups multiple times per day, or have long-running backups, it may not have had a lot of impact.
FWIW, I live in Australia (so an 'off-peak' timezone), and schedule my cronjob on an odd minute offset, so it may not have been an issue for me anyway!