Live data from Hacker News

The curl-wget Venn diagram

daniel.haxx.se

71–80 of 159 posts

Re: The curl-wget Venn diagram

#71
post #9

For me the killer feature of wget is that by default it downloads a file with a name derived from the url. You do: wget url://to/file.htm and a file named "file.htm" appears in your cwd. Using curl, you would have to do curl url://to/file.htm > file.htm or some other, less ergonomical, incantation.

I’ve always seen this as a misfeature of wget, on the general principle that command-line utilities should write their principal result to stdout unless otherwise instructed.

[deleted]

Re: The curl-wget Venn diagram

#72

I would also add at least "sane default options", "continues downloads" and "retries on error" to the Wget column. I recently had to write a script that downloads a very large file over a somewhat unreliable connection. The common wisdom among the engineers is that you need to use Wget for this job. I tried using curl but out of the box it could not resume or retry the download. I would have to study the manual and s…

I agree on the "sane defaults".

Just the fact that `wget url` downloads a URL and saves it makes it a winner for me in command-line use.

Re: The curl-wget Venn diagram

#73
post #10

Earlier quoted context omitted.

curl -O https://curl.se/docs/manpage.html#-O

Also -OJ : with -O you get the name derived from the URL (the initial one, I think, even if redirects are being followed), with -OJ you get the one from the Content-Disposition header or the final URL, the way browsers do it. Of course, plain -O is safer. (For parity with Wget, you might also want to add -R to set the downloaded file’s mtime according to the Last-Modified header.)

Wget will give you an equivalent with the --content-dispisition flag. I would like for it to be the default, but it would likely break backwards compatibility with some scripts that except a different output filename.

Re: The curl-wget Venn diagram

#74
post #33

Earlier quoted context omitted.

`&unused=.htm`. It usually works.

At this point you might as well use the -o option (-o file.htm). It's easier and easier to understand. I'd prefer wget to be a bit more clever when handling URLs query strings though, but I guess changing this behavior now might break some scripts.

The -O option, not the -o option. The capital O sets the output file, while the small o in your comment sets the log filename.

Re: The curl-wget Venn diagram

#75
post #40
post #33

Earlier quoted context omitted.

At this point you might as well use the -o option (-o file.htm). It's easier and easier to understand. I'd prefer wget to be a bit more clever when handling URLs query strings though, but I guess changing this behavior now might break some scripts.

well, depends on the usecase. sometimes you want the whole url, like when i want to mirror a site and it has stuff like foo.html?page=1 foo.html?page=2 ... wget does have options to use the name proposed by the server, and so another option to remove the query arguments would be useful, and in line with those.

A new option to strip query parameters from the output filename would be interesting. But its not so simple. When combined with recursion, one will often see a lot of pages with the same name but different query parameters. How should they be stored on disk? There's a couple of different issues I can think of.

However, if the potential issues can be resolved with sane defaults, I think this would be a great new switch to add.

Re: The curl-wget Venn diagram

#76
post #15

wget's "downloads recursively" is worth half the features of curl.

I read “downloads recursively GPLv3 licensed” and wondered whether even Stallman would really claim that a file downloaded by wget becomes retroactively GPLv3.

Re: The curl-wget Venn diagram

#78

I would also add at least "sane default options", "continues downloads" and "retries on error" to the Wget column. I recently had to write a script that downloads a very large file over a somewhat unreliable connection. The common wisdom among the engineers is that you need to use Wget for this job. I tried using curl but out of the box it could not resume or retry the download. I would have to study the manual and s…

For downloading large files I would rather just use aria2c TBH.

I only used aria2c once a long time ago, but it was awesome for huge files. As I recollect one thing it does is download different sections of a file in parallel over multiple connections, which speeds up downloads from servers that throttle per connection.

Re: The curl-wget Venn diagram

#79
For the FreeBSD users out there also ‘fetch’ is available.

Don’t know what the advantages/disadvantages are, but it comes with the default install. It’s usually what I use.

Post reply on HN