Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Tarsnap performance issues in late March, most of April

mail.tarsnap.com

Post reply on HN
Print view View original

81–90 of 116 posts

Re: Tarsnap performance issues in late March, most of April

#81

Post by cperciva »

btmorex wrote: post #77
cperciva wrote: post #75

Earlier quoted context omitted.

Too much overhead. Also, concurrent systems are actively malicious.

I don't believe that you have too many active connections for threading to work. Passive connections can be handled by a single or small number of threads. Modern Linux on modern hardware has no problem with many thousands of threads and the overhead is minimal in $$$ compared to the time you wasted debugging a scheduling problem. As for concurrent systems being harmful, you just have to design your program for threa…

One connection can have many outstanding requests.

Re: Tarsnap performance issues in late March, most of April

#82

Post by stephenr »

NhanH wrote: post #80
stephenr wrote: post #72

Earlier quoted context omitted.

Would you care to elaborate?

When Drew first do a "Show HN" [0] (before it was a thing, actually), there were a lot of response about how it doesn't do anything new that couldn't be already done by a technical inclined person (see the first two top comments in the posts). To make a comparison with tarsnap, while it's probably possible to do encrypted backup manually with a combination of shellscript and such, there are just too many moving piece…

As I already stated I never said you or I or most people should write our own encryption tools.

There are many open source backup tools. They offer a wide range of features such as data deduplication, references/hard links to simulate total backups without copying unchanged files, data compression, data encryption, logging, reporting, remote storage and/or remote sync.

Not all tools offer all features. Not all features work the same way, but there are many options.

Those that don't offer remote storage/sync can be setup very simply to backup locally and then sync/copy to your remote file store of choice - another server, s3, rsync.net, etc

The majority of these tools are shipped as part of Linux distribution repos, so there are almost certainly many more people using them, and multiple people with a vested interest in maintaining them.

And for reference, I agree with the comment(s) about Dropbox. The only difference is that they offer a more intuitive GUI which so far is lacking in open solutions.

Re: Tarsnap performance issues in late March, most of April

#83

Post by btmorex »

cperciva wrote: post #81
btmorex wrote: post #77

Earlier quoted context omitted.

I don't believe that you have too many active connections for threading to work. Passive connections can be handled by a single or small number of threads. Modern Linux on modern hardware has no problem with many thousands of threads and the overhead is minimal in $$$ compared to the time you wasted debugging a scheduling problem. As for concurrent systems being harmful, you just have to design your program for threa…

One connection can have many outstanding requests.

I would redesign your protocol to be request/response based akin to http. Achieve performance by using multiple connections in the client. Simplicity > efficiency especially if you don't have the engineering resources of a company like Google.

And I'm out. The reply rate limiting is infuriating.

Re: Tarsnap performance issues in late March, most of April

#84

Post by NhanH »

stephenr wrote: post #82
NhanH wrote: post #80

Earlier quoted context omitted.

When Drew first do a "Show HN" [0] (before it was a thing, actually), there were a lot of response about how it doesn't do anything new that couldn't be already done by a technical inclined person (see the first two top comments in the posts). To make a comparison with tarsnap, while it's probably possible to do encrypted backup manually with a combination of shellscript and such, there are just too many moving piece…

As I already stated I never said you or I or most people should write our own encryption tools. There are many open source backup tools. They offer a wide range of features such as data deduplication, references/hard links to simulate total backups without copying unchanged files, data compression, data encryption, logging, reporting, remote storage and/or remote sync. Not all tools offer all features. Not all featur…

I didn't mention writing encryption tools, I was simply saying that plugging all the available tools to use together is non-trivial.

Tarsnap to encrypted backup is what dropbox to file syncing (to a certain extent, obviously). I can understand why for someone knowledgeable like you, the benefit isn't obvious, just like we don't see the benefit of dropbox over other tools. But certain demographics will see tarsnap/ dropbox as value added, and is willing to pay for them (with good reason, too).

I know a lot of developers who have never spin up an EC2 instance, can't get their way around setting up a server, and certainly is not interested in maintaining an offsite server for backup. To them, tarsnap with its command line provide enough simplicity to be used (of course, it can be much better, as patio11 and alot of people pointed out).

Re: Tarsnap performance issues in late March, most of April

#85

Post by stephenr »

NhanH wrote: post #84
stephenr wrote: post #82

Earlier quoted context omitted.

As I already stated I never said you or I or most people should write our own encryption tools. There are many open source backup tools. They offer a wide range of features such as data deduplication, references/hard links to simulate total backups without copying unchanged files, data compression, data encryption, logging, reporting, remote storage and/or remote sync. Not all tools offer all features. Not all featur…

I didn't mention writing encryption tools, I was simply saying that plugging all the available tools to use together is non-trivial. Tarsnap to encrypted backup is what dropbox to file syncing (to a certain extent, obviously). I can understand why for someone knowledgeable like you, the benefit isn't obvious, just like we don't see the benefit of dropbox over other tools. But certain demographics will see tarsnap/ dr…

Plugging together?

I'm talking about working tools. They either do everything when invoked, or write to a file/dir on disk that can then have rsync invoked to copy offsite.

Im talking about maybe a 4 line shell script, if that.

If someone can't handle that amount of setup, maybe they shouldn't be the person setting up mission critical backups?

Re: Tarsnap performance issues in late March, most of April

#86

Post by hjnilsson »

junkblocker wrote: post #76
cperciva wrote: post #52

Earlier quoted context omitted.

Or schedule your cron job for :00, but add "sleep `jot -r 1 0 3600` &&" to the start of the command. (jot is a BSDism, but I assume you can do the same with GNU seq.)

sleep $[RANDOM/3600] works everywhere without requiring jot/seq etc. on BSD/Mac/Linux.

That will be a number between 1 and 10 ($RANDOM only goes to 32767), sleep $[RANDOM/10] would be better. :)

This might be platform dependent though, I can't find any standard RAND_MAX in bash so it's difficult to make this work everywhere.

Re: Tarsnap performance issues in late March, most of April

#87

Post by hamburglar »

btmorex wrote: post #83
cperciva wrote: post #81

Earlier quoted context omitted.

One connection can have many outstanding requests.

I would redesign your protocol to be request/response based akin to http. Achieve performance by using multiple connections in the client. Simplicity > efficiency especially if you don't have the engineering resources of a company like Google. And I'm out. The reply rate limiting is infuriating.

It's really easy to glibly criticize someone else's design decisions when 1) you don't have a full understanding of their problem, architecture, or rationale for that architecture, and when 2) the medium of the conversation doesn't lend itself well to providing you a satisfactory explanation.

It seems as though you've gotten the tiniest glimpse of some details about the system and went on to assume he made a boneheaded decision and you know better. Do you have some secret evidence that he's incompetent and doesn't have a good reason for his decision?

Re: Tarsnap performance issues in late March, most of April

#88

Post by annnnd »

jballanc wrote: post #46
Poiesis wrote: post #10

Earlier quoted context omitted.

I came here to write this comment essentially. Very much of the time I feel, "If I knew what the problem[s] [was|were] it'd be solved by now!" That's not exactly true of course but of course diagnosis is a large part of the total solution. This type of an answer that Colin gave above does not exactly win friends and influence people in most situations where you're part of a team or hierarchy. Can anyone share what th…

One trick that I've learned (though I still have trouble routinely applying it myself) for these situations is: less is more. That is, as engineers we tend to want details. All the details. We want to know what happened, why it happened, how it's going to be fixed, and how long that will take. Because we want all that detail for ourselves, we hesitate to contact our customers/boss until we have all the details. Combi…

This. There is also a saying on a similar note: "It's OK to disappoint, it's not OK to surprise."

Re: Tarsnap performance issues in late March, most of April

#89

Post by JoachimSchipper »

cperciva wrote: post #52
kijin wrote: post #47

Earlier quoted context omitted.

Use the following shell command to decide when to run cron jobs. echo $((RANDOM % 60)) It's not a CSPRNG, but good enough for this kind of load balancing!

Or schedule your cron job for :00, but add "sleep `jot -r 1 0 3600` &&" to the start of the command. (jot is a BSDism, but I assume you can do the same with GNU seq.)

Don't use that for hourly jobs, though - things are liable to break when you randomly run a command at, say, 12:59 and 13:00.

Re: Tarsnap performance issues in late March, most of April

#90

Post by girvo »

lucb1e wrote: post #41
ac29 wrote: post #29

Earlier quoted context omitted.

But its not cheap, which was my point. 100GB of storage costs: $300/year at tarsnap $36/year at S3

Finally, numbers other than picodollars and gigabyte months and unpredictable deduplication. This convinces me I don't want to store 4TB there at a huge cost($12,000 if it's really $300 a year for 100GB) compared to buying two 4TB drives (~€250 per 3-4 years) and placing them at a friend's with free bandwidth. Don't get me wrong: managed, off-site encrypted backups are very attractive, and I might be willing to pay a…

That sort of backup is what AWS Glacier is for, is it not?
Post reply on HN