Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

The curl-wget Venn diagram

daniel.haxx.se

Post reply on HN
Print view View original

131–140 of 159 posts

Re: The curl-wget Venn diagram

#131

Post by grayhatter »

dicytea wrote: post #70
vesinisa wrote: post #13

I would also add at least "sane default options", "continues downloads" and "retries on error" to the Wget column. I recently had to write a script that downloads a very large file over a somewhat unreliable connection. The common wisdom among the engineers is that you need to use Wget for this job. I tried using curl but out of the box it could not resume or retry the download. I would have to study the manual and s…

Another pet peeve of mine is that curl's URL parser is a lot more strict compared to wget. For example: $ curl -sSLOJ 'example.com/file name.txt' curl: (3) URL using bad/illegal format or missing URL $ curl -sSLOJ 'example.com/file%20name.txt' $ ls file%20name.txt On the other hand, wget (without any additional flags) will produce a file called "file name.txt" for both URLs. Well, technically you also need to add a -…

I thought curl had the ability to encode html entities if you asked it to?

Re: The curl-wget Venn diagram

#132

Post by grayhatter »

enriquto wrote: post #9

For me the killer feature of wget is that by default it downloads a file with a name derived from the url. You do: wget url://to/file.htm and a file named "file.htm" appears in your cwd. Using curl, you would have to do curl url://to/file.htm > file.htm or some other, less ergonomical, incantation.

or curl -O which is more ergonomic

that "killer feature" for cat would be turn `cat file.html` into `cat file.html > file.html` which means if you actually wanted to cat instead of cp you'd also need `cat file.html -o -` kinda glad curl doesn't have that killer feature.

Re: The curl-wget Venn diagram

#133

Post by latexr »

vesinisa wrote: post #97
OskarS wrote: post #85

Earlier quoted context omitted.

> You recall incorrectly. curl's -C flag does not work as-is. You must specify the offset from where it should continue. Why doesn't it take the resumed file's existing length as the guess by default? What else could the user want outside of some very exotic cases? But... it does, though. From the man page ( https://curl.se/docs/manpage.html#-C ) > Use "-C -" to tell curl to automatically find out where/how to resume…

No, `-C -` is not a flag. It is specifying the `-C` argument with obscure special value of `-`, which causes curl to determine the offset to continue from the output file length. This might be obvious to you if you are well-versed in curl command line, but it's by no means expected or obvious like a simple flag. > But curl is perfectly capable of resuming downloads automatically, you're just (very arrogantly) wrong o…

> It is specifying the `-C` argument with obscure special value of `-`

It is not an “obscure special value”. Not only is `-C -` (or `--continue-at -` for the long form) well documented in the correct place in the manual, `-` is a common value in command-line tools (e.g. when specifying that a tool’s input will be STDIN instead of a file).

Re: The curl-wget Venn diagram

#134

Post by scatters »

Karellen wrote: post #54
scatters wrote: post #20

Earlier quoted context omitted.

`&unused=.htm`. It usually works.

So... you're adding more noise to the filename? What?

It's a simple solution to give the file the right extension, and preserving query parameters can be the right thing to do if you hit the same path repeatedly e.g. for pagination.

Re: The curl-wget Venn diagram

#135

Post by latexr »

freeopinion wrote: post #87
jiofj wrote: post #45

Earlier quoted context omitted.

Maybe if you work for a company you don't put your heart into your creation, but if you have a popular personal project that brings you a lot of cash I'm sure you'll be as dedicated as him.

Curl is a money maker? It makes me very happy to hear that. I hope it is true.

From Daniel’s homepage¹:

> I work for wolfSSL doing commercial curl support. If you need help to fix curl problems, fix your app's use of libcurl, add features to curl, fix curl bugs, optimize your curl use or libcurl education for your developers... Then I'm your man. Contact us!

From Wikipedia’s wolfSSL page²:

> In February 2019, Daniel Stenberg, the creator of cURL, joined the wolfSSL project.

Given that, saying cURL is “a popular personal project that brings [Daniel] a lot of cash” seems like a bit of a stretch.

¹ https://daniel.haxx.se

² https://en.wikipedia.org/wiki/WolfSSL#History

Re: The curl-wget Venn diagram

#137

Post by dspillett »

twic wrote: post #122
dspillett wrote: post #112

Earlier quoted context omitted.

xargs doesn't have to wait, you can specify the number of items to include in a single sub-command and it'll batch things as they come in. For instance: ds@swann3:~# (for x in {1..100}; do sleep 0.1s; echo $x >&2; echo $x; done) | xargs -L5 echo 1 2 3 4 5 1 2 3 4 5 6 7 8 9 10 6 7 8 9 10 11 12 [... and so on ...] If the xargs call uses -I then --max-lines=1 is implied anyway. If you replace echo with something that sl…

Right, but then you are invoking curl several times, and so not reusing a single connection, as you would with wget -i, so it still loses.

Valid criticism, ish, but that wasn't in what was previously asked for so well done on being like my day-job clients and failing to specify the problem completely :)

You can specify multiple URLs on the same command in curl so using xargs in this way would do what you ask to an extent (the connection would be dropped and renegotiated at least between each batch) as long as you don't use any options that imply --max-lines=1.

With the --max-procs option you could be requesting multiple files at once which may improve performance over wget -i – though obviously take care doing this against a single site (with wget -i too for that matter) as this can be rather unfriendly (if requesting from multiple hosts this is moot, as is the multiple files-from-one-connection point).

Re: The curl-wget Venn diagram

#138

Post by dicytea »

fellerts wrote: post #106
bravetraveler wrote: post #69

On the cURL side; ridiculous manual I regularly forget the order for the values for --resolve , try searching for that word and figuring it out quickly I've been relegated to grepping a flippin' manual

A trick I've found useful when searching large man pages for a flag --foo is to search for `␣␣--foo` (note the two leading spaces). In my experience this always hits the line where the flag is defined instead of irrelevant mentions of that flag, and it's faster than paging through the manual by hand.

Or for something more flexible: ^ *--foo. But unfortunately this kind of pattern will be defeated by -f, --foo indexes.

Re: The curl-wget Venn diagram

#139

Post by jraph »

darnir wrote: post #74
jraph wrote: post #33

Earlier quoted context omitted.

At this point you might as well use the -o option (-o file.htm). It's easier and easier to understand. I'd prefer wget to be a bit more clever when handling URLs query strings though, but I guess changing this behavior now might break some scripts.

The -O option, not the -o option. The capital O sets the output file, while the small o in your comment sets the log filename.

Yep, thanks for the correction. I meant big -O, I don't know how I ended up writing small -o.

Re: The curl-wget Venn diagram

#140

Post by Karellen »

scatters wrote: post #134
Karellen wrote: post #54

Earlier quoted context omitted.

So... you're adding more noise to the filename? What?

It's a simple solution to give the file the right extension, and preserving query parameters can be the right thing to do if you hit the same path repeatedly e.g. for pagination.

> It's a simple solution to give the file the right extension,

Oh, I see now.

Do you work with many tools that can't work with files if they don't have the "right" extension? I thought that was mostly a Windows problem.

Post reply on HN