Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

The curl-wget Venn diagram

daniel.haxx.se

Post reply on HN
Print view View original

81–90 of 159 posts

Re: The curl-wget Venn diagram

#81

Post by bla3 »

vesinisa wrote: post #13

I would also add at least "sane default options", "continues downloads" and "retries on error" to the Wget column. I recently had to write a script that downloads a very large file over a somewhat unreliable connection. The common wisdom among the engineers is that you need to use Wget for this job. I tried using curl but out of the box it could not resume or retry the download. I would have to study the manual and s…

Also following redirects by default. "Run curl, be confused run curl -L" is something I do many times a month.

Re: The curl-wget Venn diagram

#82

Post by sherburt3 »

vesinisa wrote: post #61
OskarS wrote: post #19

Earlier quoted context omitted.

> I tried using curl but out of the box it could not resume or retry the download. Maybe I'm misunderstanding, but curl has exactly that feature, it's the `-C` flag. If you want retries, there's `--retry`. I find curls defaults pretty sane, personally, I wouldn't want either of those by default for a tool like curl.

You recall incorrectly. curl's -C flag does not work as-is. You must specify the offset from where it should continue. Why doesn't it take the resumed file's existing length as the guess by default? What else could the user want outside of some very exotic cases? Yes, I want retries. They should be the default for a user-facing tool. Try searching the curl's manual page for "retry". There are no less than 5 different…

I thought that `-C -` checks the output file for the offset

Re: The curl-wget Venn diagram

#84

Post by freeopinion »

cinntaile wrote: post #32
bdhcuidbebe wrote: post #26

Earlier quoted context omitted.

I just look at my curl / wget note for some sane examples. i cant believe one needed chatgpt to be productive

If chatgpt makes someone more productive, then why not. It's just a tool as any other tool.

I agree with this sentiment. But I also have personal antidotes that make me cautious about it. When I was a child I memorized dozens of phone numbers of people I called a couple times a month. Now my phone memorizes any phone number for me. We used to memorize poems and such. I can't give you any scientific reason why it is good to have memorized the Gettysburg Address. But I feel intuitively that there is some benefit to exercise your brain similar to the intuition I have about exercising muscles.

If a person thinks they benefit from 100 situps a day, I'm not going to disagree. And if they think there is some benefit in reading man pages, well, thanks to those who take the time to write all that ducumentation.

Re: The curl-wget Venn diagram

#85

Post by OskarS »

vesinisa wrote: post #61
OskarS wrote: post #19

Earlier quoted context omitted.

> I tried using curl but out of the box it could not resume or retry the download. Maybe I'm misunderstanding, but curl has exactly that feature, it's the `-C` flag. If you want retries, there's `--retry`. I find curls defaults pretty sane, personally, I wouldn't want either of those by default for a tool like curl.

You recall incorrectly. curl's -C flag does not work as-is. You must specify the offset from where it should continue. Why doesn't it take the resumed file's existing length as the guess by default? What else could the user want outside of some very exotic cases? Yes, I want retries. They should be the default for a user-facing tool. Try searching the curl's manual page for "retry". There are no less than 5 different…

> You recall incorrectly. curl's -C flag does not work as-is. You must specify the offset from where it should continue. Why doesn't it take the resumed file's existing length as the guess by default? What else could the user want outside of some very exotic cases?

But... it does, though. From the man page ( https://curl.se/docs/manpage.html#-C )

> Use "-C -" to tell curl to automatically find out where/how to resume the transfer. It then uses the given output/input files to figure that out.

I just tried it, works perfectly. I don't really see a difference between writing `--continue` in wget and `-C -` in curl. And the use case for specifying it is not so exotic, you might want to do range requests for all sorts of reasons.

Look, it's fine if you prefer wget's command line: I don't think retrying a request is a reasonable default for a tool like curl, but reasonable people can disagree on that for sure. But curl is perfectly capable of resuming downloads automatically, you're just (very arrogantly) wrong on that one.

Re: The curl-wget Venn diagram

#86

Post by semi »

vesinisa wrote: post #61
OskarS wrote: post #19

Earlier quoted context omitted.

> I tried using curl but out of the box it could not resume or retry the download. Maybe I'm misunderstanding, but curl has exactly that feature, it's the `-C` flag. If you want retries, there's `--retry`. I find curls defaults pretty sane, personally, I wouldn't want either of those by default for a tool like curl.

You recall incorrectly. curl's -C flag does not work as-is. You must specify the offset from where it should continue. Why doesn't it take the resumed file's existing length as the guess by default? What else could the user want outside of some very exotic cases? Yes, I want retries. They should be the default for a user-facing tool. Try searching the curl's manual page for "retry". There are no less than 5 different…

What is a sane default for retries? is it to loop indefinitely? should it retry against the same TCP connection or establish a new one? To the same IP it picked the first time or a different one, or reresolve the DNS entirely? against the same resolver?

IMO theres too much complexity for 'sane defaults' to not just be 'surprising behavior' for someone else's use case.

Re: The curl-wget Venn diagram

#87

Post by freeopinion »

jiofj wrote: post #45
ducktective wrote: post #24

Daniel Stenberg is among those rare breed of developers who put their heart and soul into their creation, a fading trait in the modern world of big tech that shadowy developers seem to be replaceable cogs of a money-making machine. It's as if he treats curl as his mark on the world of IT.

Maybe if you work for a company you don't put your heart into your creation, but if you have a popular personal project that brings you a lot of cash I'm sure you'll be as dedicated as him.

Curl is a money maker? It makes me very happy to hear that. I hope it is true.

Re: The curl-wget Venn diagram

#88

Post by tomatocracy »

vesinisa wrote: post #13

I would also add at least "sane default options", "continues downloads" and "retries on error" to the Wget column. I recently had to write a script that downloads a very large file over a somewhat unreliable connection. The common wisdom among the engineers is that you need to use Wget for this job. I tried using curl but out of the box it could not resume or retry the download. I would have to study the manual and s…

One point where I would argue wget doesn't have a sensible default is on filenames - it really should make --content-disposition the default, at least for single file downloads. Otherwise it will often use the wrong name if there is a redirect or similar in the chain, which seems increasingly common.

Re: The curl-wget Venn diagram

#89

Post by joshspankit »

tomatocracy wrote: post #88
vesinisa wrote: post #13

I would also add at least "sane default options", "continues downloads" and "retries on error" to the Wget column. I recently had to write a script that downloads a very large file over a somewhat unreliable connection. The common wisdom among the engineers is that you need to use Wget for this job. I tried using curl but out of the box it could not resume or retry the download. I would have to study the manual and s…

One point where I would argue wget doesn't have a sensible default is on filenames - it really should make --content-disposition the default, at least for single file downloads. Otherwise it will often use the wrong name if there is a redirect or similar in the chain, which seems increasingly common.

Also (annoyingly) URL params

Re: The curl-wget Venn diagram

#90

Post by smaudet »

vesinisa wrote: post #61
OskarS wrote: post #19

Earlier quoted context omitted.

> I tried using curl but out of the box it could not resume or retry the download. Maybe I'm misunderstanding, but curl has exactly that feature, it's the `-C` flag. If you want retries, there's `--retry`. I find curls defaults pretty sane, personally, I wouldn't want either of those by default for a tool like curl.

You recall incorrectly. curl's -C flag does not work as-is. You must specify the offset from where it should continue. Why doesn't it take the resumed file's existing length as the guess by default? What else could the user want outside of some very exotic cases? Yes, I want retries. They should be the default for a user-facing tool. Try searching the curl's manual page for "retry". There are no less than 5 different…

Not only are you wrong, I had the opposite experience - whether was all well and good for downloading pages, but it fell over for making e.g. authenticated POST requests. I think that might be possible now/have gotten better, but wget has always been too restrained for me.

I am in agreement that wget has "sane" defaults i.e. it acts like a bot or web crawler, or basic browser. Curl has always been easier to get things done with, though. At least in the land of http requests.

Post reply on HN