Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

The curl-wget Venn diagram

daniel.haxx.se

Post reply on HN
Print view View original

61–70 of 159 posts

Re: The curl-wget Venn diagram

#61

Post by vesinisa »

OskarS wrote: post #19
vesinisa wrote: post #13

I would also add at least "sane default options", "continues downloads" and "retries on error" to the Wget column. I recently had to write a script that downloads a very large file over a somewhat unreliable connection. The common wisdom among the engineers is that you need to use Wget for this job. I tried using curl but out of the box it could not resume or retry the download. I would have to study the manual and s…

> I tried using curl but out of the box it could not resume or retry the download. Maybe I'm misunderstanding, but curl has exactly that feature, it's the `-C` flag. If you want retries, there's `--retry`. I find curls defaults pretty sane, personally, I wouldn't want either of those by default for a tool like curl.

You recall incorrectly. curl's -C flag does not work as-is. You must specify the offset from where it should continue. Why doesn't it take the resumed file's existing length as the guess by default? What else could the user want outside of some very exotic cases?

Yes, I want retries. They should be the default for a user-facing tool. Try searching the curl's manual page for "retry". There are no less than 5 different interdependent flags for specifying retry behaviour: --retry-all-errors, --retry-connrefused, --retry-delay , --retry-max-time and --retry

If I "just" want it to retry, surely there's a boolean flag like "--retry" that enables sane defaults? Nope! --retry takes a mandatory integer argument of maximum number of retries. Surely I can set it to zero for a sane default? Nope again: "Setting the number to 0 makes curl do no retries."

curl is a good tool if you know it through & through and want exact control over the transfer behaviour. I don't think it's a good tool if you just want to fetch a file and except your tool of choice to apply some sane behaviours for you to that end, that would probably make sense if you are a human rather than an application using a library.

Re: The curl-wget Venn diagram

#62

Post by twic »

vesinisa wrote: post #13

I would also add at least "sane default options", "continues downloads" and "retries on error" to the Wget column. I recently had to write a script that downloads a very large file over a somewhat unreliable connection. The common wisdom among the engineers is that you need to use Wget for this job. I tried using curl but out of the box it could not resume or retry the download. I would have to study the manual and s…

Also add -i which lets wget read URLs from a file. In particular wget -i - which makes it read from standard input, and is very useful in pipelines.

curl cannot, AFAIK, do this. People usually suggest using xargs, which is a mediocre substitute because it waits for all the URLs to arrive before invoking curl, giving up any chance at parallelism between the command generating the URLs and the one downloading them.

Re: The curl-wget Venn diagram

#63

Post by dotancohen »

bdhcuidbebe wrote: post #26
TheAceOfHearts wrote: post #23

Earlier quoted context omitted.

Both tools have their use-cases. I think with the advent of LLMs like ChatGPT it has also become a lot easier to get the proper command line incantation for whatever tool you're using. Even if you've read the manual previously, it's easy to forget the exact flags you want to use, and validating the generated command line is usually less effort than having to build it up yourself from scratch by reading the manual. Wi…

I just look at my curl / wget note for some sane examples. i cant believe one needed chatgpt to be productive

I would love to see your curl / wget note if you don't mind.

Re: The curl-wget Venn diagram

#65

Post by KaiserPro »

I've never seen them as competitors!

wget is my goto if I need to download a file now, with the minimum of fuss.

curl is used when I need to do something fancy with a url to make it work, or when I'm fiddling with params to make an API work/debug it.

Re: The curl-wget Venn diagram

#66

Post by acdha »

azatom wrote: post #2

don't forget the weekly security fix on the right side ;)

Curl is very widely used and has a ton of features which means that it gets a lot of CVEs, but their severity is often significantly overstated for users outside of specific niche configurations - for marketing purposes, it’s nice to be able to say that you found a HIGH in libcurl without mentioning that it only affected Windows domain authentication on ARM. The lead developer has written about this providing a lot of noise without much tangible security benefit:

https://daniel.haxx.se/blog/2023/08/26/cve-2020-19909-is-eve...

Re: The curl-wget Venn diagram

#67

Post by bandergirl »

curl is a connection tool while wget is an app. At a basic level they do the same thing, but they excel in different areas.

This diagram is clearly and unapologetically biased towards curl. Feels strange that the author of curl doesn’t know what wget actually offers.

Re: The curl-wget Venn diagram

#70

Post by dicytea »

vesinisa wrote: post #13

I would also add at least "sane default options", "continues downloads" and "retries on error" to the Wget column. I recently had to write a script that downloads a very large file over a somewhat unreliable connection. The common wisdom among the engineers is that you need to use Wget for this job. I tried using curl but out of the box it could not resume or retry the download. I would have to study the manual and s…

Another pet peeve of mine is that curl's URL parser is a lot more strict compared to wget.

For example:

  $ curl -sSLOJ 'example.com/file name.txt'
  curl: (3) URL using bad/illegal format or missing URL

  $ curl -sSLOJ 'example.com/file%20name.txt'
  $ ls
  file%20name.txt
On the other hand, wget (without any additional flags) will produce a file called "file name.txt" for both URLs. Well, technically you also need to add a --content-on-error flag to wget because this example URL 404s.
Post reply on HN