51–60 of 116 posts
Earlier quoted context omitted.
Because tarsnap is cheap, incredibly well documented, open source, and run by an awesome guy. It's an all around win-win.
But its not cheap, which was my point. 100GB of storage costs: $300/year at tarsnap $36/year at S3
Earlier quoted context omitted.
Funny part to that, was on a project with multiple teams with multiple crontabs. Each team took that advice to heart for some jobs. Sadly, we had too many Hitchhiker fans and :42 became a bit too common.
Use the following shell command to decide when to run cron jobs. echo $((RANDOM % 60)) It's not a CSPRNG, but good enough for this kind of load balancing!
Earlier quoted context omitted.
We actually have our Chef rdiff backup cookbook randomly distribute jobs across a buckets of time using a hash function of the hostname.
I have to know: Why a hash function of the hostname?
As an AWS user this type of thing gives me cause for concern: At 2015-04-01 00:00 UTC, the Amazon EC2 "provisioned I/O" volume on which most of this metadata was stored suddenly changed from an average latency of 1.2 ms per request to an average latency of 2.2 ms per request. I have no idea why this happened -- indeed, I was so surprised by it that I didn't believe Amazon's monitoring systems at first -- but this imm…
Indeed, I didn't know such a change was possible -- that EBS volume went for years with consistent low latency before it suddenly slowed down.
If you absolutely need guaranteed IO performance, use an instance store or move to dedicated hardware. Them be the breaks of cloud computing.
http://en.wikipedia.org/wiki/Fallacies_of_distributed_comput...
As an AWS user this type of thing gives me cause for concern: At 2015-04-01 00:00 UTC, the Amazon EC2 "provisioned I/O" volume on which most of this metadata was stored suddenly changed from an average latency of 1.2 ms per request to an average latency of 2.2 ms per request. I have no idea why this happened -- indeed, I was so surprised by it that I didn't believe Amazon's monitoring systems at first -- but this imm…
This is why I don't use AWS for anything non-trivial, and I am wary of people who put critical infrastructure on it. (EG: I Don't care about netflix, that service can run on AWS fine, but coinbase, for instance, if I was their customer and they ran on AWS I would stop being their customer.)
Whenever AWS problems come up people talk about how "AWS is so much more efficient, you just outsource that stuff to the experts".
But that seems to imply that hosting on your own hardware in your own office is the only alternative. Of course we stopped doing that in the 1990s.
With AWS you have to know Linux and have ops people, that's true everywhere. With AWS you have the additional burden of learning the AWS APIs and learning how to use AWS, which isn't transferrable, so that's a higher cost. With AWS you have to architect around the limitations of the way AWS is built and your architecture becomes AWS specific if you use those APIS, so that's an additional cost. You don't need any less ops people, probably more, than going with another hosting service like Digital Ocean or Backspace. And if you go with something like Hetzner you pay 1/5th to 1/10th for machines with a lot more performance and local storage. (Though you get the additional latency of being located in Europe, if your primary customers are the USA.)
Of course, I'm also prejudiced. I worked at Amazon and saw how the sausage was made and was not impressed. When AWS was announced as "running on the same infrastructure that powers Amazon.com!!!" as if it was a feature, I cringed. Amazon.com was having outages of parts or major components on a weekly basis at that time. Much of AWS is actually running on bespoke software (so not actually tested by Amazon.com when introduced, though I'm sure portions have been moved over at gunpoint) ... which actually makes it worse. People were trusting their data to a service that pretended to be backing a major e-commerce site but was actually untested outside of the company at the time.
And what have we seen since? An unacceptable level of failures. (in my opinion, of course)
But people seem to be very forgiving. When it's happening everyone's in "how can we fix this mode" and then when it's fixed everyone forgets and goes back to thinking of AWS as always running.
Earlier quoted context omitted.
But its not cheap, which was my point. 100GB of storage costs: $300/year at tarsnap $36/year at S3
Finally, numbers other than picodollars and gigabyte months and unpredictable deduplication. This convinces me I don't want to store 4TB there at a huge cost($12,000 if it's really $300 a year for 100GB) compared to buying two 4TB drives (~€250 per 3-4 years) and placing them at a friend's with free bandwidth. Don't get me wrong: managed, off-site encrypted backups are very attractive, and I might be willing to pay a…
Earlier quoted context omitted.
Yes, when it comes to crypto I'd put my in trust in highly talented people over trusting my own ability to glue together a collection of OSS tools anyday.
You seem to have misunderstood me. I didn't suggest you should write your own encryption tool. There are numerous open source tools for creating encrypted backups, some do deduplication first too. If the tool doesn't happen to support remote storage, a simple rsync or scp fills that part. Literally the only thing unique about this service is the use of the term picodollars and the single individual it's all reliant o…
Earlier quoted context omitted.
For those interested in patio11's thoughts on how he would run tarsnap http://www.kalzumeus.com/2014/04/03/fantasy-tarsnap/ And the discussion on HN https://news.ycombinator.com/item?id=7523953
Did Colin ever reply to that? I've always wondered what his response was.
I suppose I should have known that this would end up at the top of Hacker News...
Happy to help if you'd like.
Earlier quoted context omitted.
For those interested in patio11's thoughts on how he would run tarsnap http://www.kalzumeus.com/2014/04/03/fantasy-tarsnap/ And the discussion on HN https://news.ycombinator.com/item?id=7523953
Did Colin ever reply to that? I've always wondered what his response was.