Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Tarsnap performance issues in late March, most of April

mail.tarsnap.com

Post reply on HN
Print view View original

101–110 of 116 posts

Re: Tarsnap performance issues in late March, most of April

#101

Post by mryan »

mtsmith85 wrote: post #2

This line: I would have sent out an email to the mailing lists earlier; but since at each point I thought I was "one change away" from fixing the problems, I kept on delaying said email until it was clear that the problems were finally fixed" is such a common situation for most people, but I tend to see it with engineers especially. I find I struggle with it an incredible amount. In some ways, I guess it seems health…

This "it's almost fixed, I'll email the client soon" pattern is something I have personally struggled with a lot, and I agree it appears to be common with engineers.

My workaround has been to make something else responsible for sending the email. In a team, this could be a manager setting a cut-off point after which communication must be made. When working on my own, I set an alarm for X minutes. When that alarm goes off I ignore the internal voice which says "just try one more thing, then send the email", and send an update to let the relevant people know my current progress, ETA to fix, and when they can expect the next update.

I think this is similar to how GTD encourages us to use systems for storing to-do lists instead of trying to remember them - our fragile human brains are not always to be trusted.

Re: Tarsnap performance issues in late March, most of April

#102

Post by jeffbarr »

cperciva wrote: post #65
jeffbarr wrote: post #63

Earlier quoted context omitted.

You could have contacted AWS support or emailed me. Either way, we would have investigated.

It wasn't missing its guaranteed # of I/Os per second, so I figured the slowdown was just "one of those things" and not an out-of-spec issue. Happy to send you the volume ID if you think someone would want to investigate (and still has data from the start of April) though.

Yes, please do.

Re: Tarsnap performance issues in late March, most of April

#103

Post by protomyth »

kijin wrote: post #47
protomyth wrote: post #8

Earlier quoted context omitted.

Funny part to that, was on a project with multiple teams with multiple crontabs. Each team took that advice to heart for some jobs. Sadly, we had too many Hitchhiker fans and :42 became a bit too common.

Use the following shell command to decide when to run cron jobs. echo $((RANDOM % 60)) It's not a CSPRNG, but good enough for this kind of load balancing!

We just went with a single group text file with all the jobs and which ones could be spread out. Saves the programming and gives the sys admins / DBAs an idea what goes when.

Re: Tarsnap performance issues in late March, most of April

#105

Post by pronoiac »

stephenr wrote: post #48
dmix wrote: post #42

Earlier quoted context omitted.

Yes, when it comes to crypto I'd put my in trust in highly talented people over trusting my own ability to glue together a collection of OSS tools anyday.

You seem to have misunderstood me. I didn't suggest you should write your own encryption tool. There are numerous open source tools for creating encrypted backups, some do deduplication first too. If the tool doesn't happen to support remote storage, a simple rsync or scp fills that part. Literally the only thing unique about this service is the use of the term picodollars and the single individual it's all reliant o…

Check out the key roles; you can split up writing and deleting archives, so - for example - a hacked machine can't delete the archives. This is nice.

Re: Tarsnap performance issues in late March, most of April

#108

Post by snuxoll »

MCRed wrote: post #55
k1w1 wrote: post #36

As an AWS user this type of thing gives me cause for concern: At 2015-04-01 00:00 UTC, the Amazon EC2 "provisioned I/O" volume on which most of this metadata was stored suddenly changed from an average latency of 1.2 ms per request to an average latency of 2.2 ms per request. I have no idea why this happened -- indeed, I was so surprised by it that I didn't believe Amazon's monitoring systems at first -- but this imm…

It's getting to the point where, when I see a post mortem like this, I am just waiting for the AWS problems. Between this and the downtime that AWS has, I'm kind of amazed that people use it-- you pay too much and you get less. (Compared to a lot of other choices, such as raw metal boxes from Hetzner) This is why I don't use AWS for anything non-trivial, and I am wary of people who put critical infrastructure on it.…

To this day I still do not get why you would use AWS, the entire user experience is clunky and the pricing is crazy for what you get. Azure isn't much better with regards to downtime, but if you want something more than just a VPS I'd choose it any day over AWS for the significantly better UX in both the admin console and the command line tools + SDK.

Ultimately though, even with Azure or AWS you're going to need people knowledgeable enough to administer your compute instances anyway, so why not just run your full stack on a bunch of VM's from DigitalOcean or Linode or rent a couple dedicated servers and throw oVirt on them; saving yourself a significant chunk of money at the same time.

Re: Tarsnap performance issues in late March, most of April

#109

Post by gonzo »

cperciva wrote: post #17
mtsmith85 wrote: post #2

This line: I would have sent out an email to the mailing lists earlier; but since at each point I thought I was "one change away" from fixing the problems, I kept on delaying said email until it was clear that the problems were finally fixed" is such a common situation for most people, but I tend to see it with engineers especially. I find I struggle with it an incredible amount. In some ways, I guess it seems health…

I'd love some context (maybe from cperciva himself?) around the performance enhancement of integrating new Intel AESNI instructions. I was using OpenSSL for that (which was using a software implementation). The code (you can see it in spiped) now detects the CPU feature and selects between AESNI or OpenSSL automatically. Given that the tarsnap server code was spending about 40% of its time running AES, it's a nontriv…

One wonders why you aren't using a version of OpenSSL that has the AESNI bits already in it.

Re: Tarsnap performance issues in late March, most of April

#110

Post by stephenr »

lucb1e wrote: post #95
stephenr wrote: post #78

Earlier quoted context omitted.

Rough estimate here: If you upload 4tb in a year, that's 333.33gb/month So for tarsnap that equals - $1k/year in data transfer charges (4000gb * $0.25 transfer charge * 12 months) - $83/month per month of data (333gb * $0.25 storage cost/month) - $6.4k/year for the first year ($83 * 78 cumulative months in a year) So $7.4k for 12 months resulting in 4tb If usage stays the same each year will add $12k to the increment…

> that's 333.33gb/month I have 4TB of data, which changes an unknown amount (probably around 20-50GB per month) and grows slightly (probably 5-15GB per month). In any case, thanks for the calculation. Tarsnap is apparently not for the common person who wants to back up everything including their media.

That actually works out worse - $14K for the first year, $13K for the second year.
Post reply on HN