Tarsnap performance issues in late March, most of April

Post by pliu » Wed, May 06, 2015, 5:09 AM UTC

sillysaurus3 wrote: post #43

jldugger wrote: post #32

Earlier quoted context omitted.

We actually have our Chef rdiff backup cookbook randomly distribute jobs across a buckets of time using a hash function of the hostname.

I have to know: Why a hash function of the hostname?

  # Generate a uniformly distributed unique number to sleep.
  if node['chef_client']['splay'].to_i > 0
    checksum   = Digest::MD5.hexdigest(node['fqdn'] || 'unknown-hostname')
    sleep_time = checksum.to_s.hex % node['chef_client']['splay'].to_i
  else
    sleep_time = nil
  end

Post by jeffbarr » Wed, May 06, 2015, 5:18 AM UTC

cperciva wrote: post #45

k1w1 wrote: post #36

As an AWS user this type of thing gives me cause for concern: At 2015-04-01 00:00 UTC, the Amazon EC2 "provisioned I/O" volume on which most of this metadata was stored suddenly changed from an average latency of 1.2 ms per request to an average latency of 2.2 ms per request. I have no idea why this happened -- indeed, I was so surprised by it that I didn't believe Amazon's monitoring systems at first -- but this imm…

Indeed, I didn't know such a change was possible -- that EBS volume went for years with consistent low latency before it suddenly slowed down.

Post by cperciva » Wed, May 06, 2015, 5:27 AM UTC

jeffbarr wrote: post #63

cperciva wrote: post #45

Earlier quoted context omitted.

Indeed, I didn't know such a change was possible -- that EBS volume went for years with consistent low latency before it suddenly slowed down.

You could have contacted AWS support or emailed me. Either way, we would have investigated.

Post by ploxiln » Wed, May 06, 2015, 5:55 AM UTC

patio11 wrote: post #66

ploxiln wrote: post #38

Earlier quoted context omitted.

Don't you have some other servers running other services? So you must already have some monitoring and alerting system like Nagios, to which you can add one more little "passive check" that does the same thing, for no incremental cost?

I have roughly fourish separate monitoring systems for Appointment Reminder. DMS is the one which is least tied to me, so I use it for Tarsnap (the most critical thing about AR that can fail "quietly") and as the fourthish line of defense for the core AR functionality. (This may be slightly overbuilt, but I felt it justified to get peace of mind, given AR's fair degree of importance to customers/myself and the enterp…

Post by lucb1e » Wed, May 06, 2015, 6:24 AM UTC

patmcc wrote: post #56

lucb1e wrote: post #41

Earlier quoted context omitted.

Finally, numbers other than picodollars and gigabyte months and unpredictable deduplication. This convinces me I don't want to store 4TB there at a huge cost($12,000 if it's really $300 a year for 100GB) compared to buying two 4TB drives (~€250 per 3-4 years) and placing them at a friend's with free bandwidth. Don't get me wrong: managed, off-site encrypted backups are very attractive, and I might be willing to pay a…

Tarsnap isn't intended to be used as one-time backup like that, and it's super expensive if used that way. It's very cheap when used to backup (almost) the same 4GB for 1000 days in a row, which is what a lot of people/businesses need for their backup solutions.