91–100 of 116 posts
For those that want to run a similar service using their own systems, I found that Attic [1] is a great open source backup tool that works in a very similar way, including deduplication and compression. I backup some VPS servers to my NAS at home using attic over an SSH tunnel. Incremental backups are quite small and it's easy to automate with a simple cron job. [1] https://attic-backup.org/
This line: I would have sent out an email to the mailing lists earlier; but since at each point I thought I was "one change away" from fixing the problems, I kept on delaying said email until it was clear that the problems were finally fixed" is such a common situation for most people, but I tend to see it with engineers especially. I find I struggle with it an incredible amount. In some ways, I guess it seems health…
I would have sent out an email to the mailing lists earlier; but since at each point I thought I was "one change away" from fixing the problems, I kept on delaying said email until it was clear that the problems were finally fixed This ties in to the last lesson I mentioned at the bottom: 5. When performance drops, it's not always due to a single problem; sometimes there are multiple interacting bottlenecks. Every ti…
Very common! One thing that's been helpful for us is establishing predefined system performance thresholds that, if exceeded, initiate the chain of events that will lead to customer communication. "If X% of requests are failing, then we had better advertise that the system is degraded." Discussing and setting these thresholds in advance and the expectation that they'll result in communication helps drive the right outcome. It's not perfect, because one is always tempted to make a judgment call in the circumstance, which is vulnerable to the same effect, but it's a good start.
Thanks for sharing!
Earlier quoted context omitted.
sleep $[RANDOM/3600] works everywhere without requiring jot/seq etc. on BSD/Mac/Linux.
That will be a number between 1 and 10 ($RANDOM only goes to 32767), sleep $[RANDOM/10] would be better. :) This might be platform dependent though, I can't find any standard RAND_MAX in bash so it's difficult to make this work everywhere.
sleep $(( 0x$(xxd -l2 -p /dev/random) * 3600 / 65536 ))In case any other customer is wondering "Wait, I didn't hear anything from my monitoring about that and I'm retroactively worried. How worried should I be?" like I was: I just pulled our logs and reconstructed them, and it shows over the last ~30 days that the worse-case performance of our daily backup (~150 MB per day delta, ~45 GB total post deduplication) was about 40% longer than our typical case. This didn't tri…
"AccuracySec=" in *.timer files lets you specify the amount of slack systemd has in firing timers. To quote the documentation "Within this time window, the expiry time will be placed at a host-specific, randomized but stable position that is synchronized between all local timer units."
You may still want to randomize timers locally on a host too, but the above makes automated deployment of timers that affects network services very convenient.
Earlier quoted context omitted.
It's not one time, I'd be incrementally writing updates to the disks. With a raspberry pi or something, the power costs are near negligible.
Rough estimate here: If you upload 4tb in a year, that's 333.33gb/month So for tarsnap that equals - $1k/year in data transfer charges (4000gb * $0.25 transfer charge * 12 months) - $83/month per month of data (333gb * $0.25 storage cost/month) - $6.4k/year for the first year ($83 * 78 cumulative months in a year) So $7.4k for 12 months resulting in 4tb If usage stays the same each year will add $12k to the increment…
I have 4TB of data, which changes an unknown amount (probably around 20-50GB per month) and grows slightly (probably 5-15GB per month).
In any case, thanks for the calculation. Tarsnap is apparently not for the common person who wants to back up everything including their media.
Earlier quoted context omitted.
Finally, numbers other than picodollars and gigabyte months and unpredictable deduplication. This convinces me I don't want to store 4TB there at a huge cost($12,000 if it's really $300 a year for 100GB) compared to buying two 4TB drives (~€250 per 3-4 years) and placing them at a friend's with free bandwidth. Don't get me wrong: managed, off-site encrypted backups are very attractive, and I might be willing to pay a…
That sort of backup is what AWS Glacier is for, is it not?
Earlier quoted context omitted.
Use the following shell command to decide when to run cron jobs. echo $((RANDOM % 60)) It's not a CSPRNG, but good enough for this kind of load balancing!
Or schedule your cron job for :00, but add "sleep `jot -r 1 0 3600` &&" to the start of the command. (jot is a BSDism, but I assume you can do the same with GNU seq.)
Earlier quoted context omitted.
Too much overhead. Also, concurrent systems are actively malicious.
I don't believe that you have too many active connections for threading to work. Passive connections can be handled by a single or small number of threads. Modern Linux on modern hardware has no problem with many thousands of threads and the overhead is minimal in $$$ compared to the time you wasted debugging a scheduling problem. As for concurrent systems being harmful, you just have to design your program for threa…
Better than time wasted debugging the races and deadlocks that only threads can cause. These are much harder to debug because they are so much harder to examine without changing behaviour.
Sure, there's a trade-off - so there's no point in pretending that there are no downsides to using threads.
Earlier quoted context omitted.
One trick that I've learned (though I still have trouble routinely applying it myself) for these situations is: less is more. That is, as engineers we tend to want details. All the details. We want to know what happened, why it happened, how it's going to be fixed, and how long that will take. Because we want all that detail for ourselves, we hesitate to contact our customers/boss until we have all the details. Combi…
This. There is also a saying on a similar note: "It's OK to disappoint, it's not OK to surprise."
This line: I would have sent out an email to the mailing lists earlier; but since at each point I thought I was "one change away" from fixing the problems, I kept on delaying said email until it was clear that the problems were finally fixed" is such a common situation for most people, but I tend to see it with engineers especially. I find I struggle with it an incredible amount. In some ways, I guess it seems health…
I would have sent out an email to the mailing lists earlier; but since at each point I thought I was "one change away" from fixing the problems, I kept on delaying said email until it was clear that the problems were finally fixed This ties in to the last lesson I mentioned at the bottom: 5. When performance drops, it's not always due to a single problem; sometimes there are multiple interacting bottlenecks. Every ti…