Looks like wget 2 introduces an equivalent to libcurl: libwget [0][1].
The curl-wget Venn diagram
121–130 of 159 posts
Re: The curl-wget Venn diagram
#122Earlier quoted context omitted.
Also add -i which lets wget read URLs from a file. In particular wget -i - which makes it read from standard input, and is very useful in pipelines. curl cannot, AFAIK, do this. People usually suggest using xargs, which is a mediocre substitute because it waits for all the URLs to arrive before invoking curl, giving up any chance at parallelism between the command generating the URLs and the one downloading them.
xargs doesn't have to wait, you can specify the number of items to include in a single sub-command and it'll batch things as they come in. For instance: ds@swann3:~# (for x in {1..100}; do sleep 0.1s; echo $x >&2; echo $x; done) | xargs -L5 echo 1 2 3 4 5 1 2 3 4 5 6 7 8 9 10 6 7 8 9 10 11 12 [... and so on ...] If the xargs call uses -I then --max-lines=1 is implied anyway. If you replace echo with something that sl…
Right, but then you are invoking curl several times, and so not reusing a single connection, as you would with wget -i, so it still loses.
Re: The curl-wget Venn diagram
#123Earlier quoted context omitted.
I agree on the "sane defaults". Just the fact that `wget url` downloads a URL and saves it makes it a winner for me in command-line use.
Well, the point of the article is that they are not cpmpetitors and are used differently. For me, 99% of the time i'm curl-ing some API and I definitely don't want to save the result to disk (but often want to pipe it to grep/jq).
Agree. I also use both: wget to download files, and curl to talk to APIs, and as a "no magic" HTTP client.
This is not about "sane defaults", but about use cases.
Re: The curl-wget Venn diagram
#124don't forget the weekly security fix on the right side ;)
Curl is very widely used and has a ton of features which means that it gets a lot of CVEs, but their severity is often significantly overstated for users outside of specific niche configurations - for marketing purposes, it’s nice to be able to say that you found a HIGH in libcurl without mentioning that it only affected Windows domain authentication on ARM. The lead developer has written about this providing a lot o…
Looks like cURL and SQLite have the same woes: https://www.sqlite.org/cves.html
Previously I worked on an open source project that pulled in many third party libraries. Users would run their corpo vulnerability scanners on the project and find dependencies with open CVEs and demand fixes, not understanding that in our usage of the libraries, the vulnerability is not exposed.
I think in 4 years, we had users open roughly 50 issues like this, which corresponded to exactly 0 real world exploitable issues.
A central vuln DB makes sense for sysadmins, but too many make it the end-all-be-all.
Re: The curl-wget Venn diagram
#125Earlier quoted context omitted.
Curl is very widely used and has a ton of features which means that it gets a lot of CVEs, but their severity is often significantly overstated for users outside of specific niche configurations - for marketing purposes, it’s nice to be able to say that you found a HIGH in libcurl without mentioning that it only affected Windows domain authentication on ARM. The lead developer has written about this providing a lot o…
Looks like cURL and SQLite have the same woes: https://www.sqlite.org/cves.html Previously I worked on an open source project that pulled in many third party libraries. Users would run their corpo vulnerability scanners on the project and find dependencies with open CVEs and demand fixes, not understanding that in our usage of the libraries, the vulnerability is not exposed. I think in 4 years, we had users open roug…
I think this ends up devolving to Goodhart’s law: once CVEs became marketing, a ton of people had a huge incentive to game their stats at the expense of everyone else’s time.
Re: The curl-wget Venn diagram
#126Earlier quoted context omitted.
Retry with `wget` was one of the most incredible Linux distro included features when I started running it at home. Pretty crucial thing on 56K dialup, and it worked better than the Windows tools I was aware of at the time.
Our dialup used to disconnect every 2 hours by design. wget and wvdial were the only alternative to mail-ordering CDs
We were way out in the sticks, so interruptions were very common. Still nothing but dialup, satellite, and LTE (no 5G) out there.
Re: The curl-wget Venn diagram
#127For many of us, I bet the key distinction is "the one that writes to stdout by default" vs "the one that makes a file by default".
or "the one that can be piped to `sh` by default" ;-)
Re: The curl-wget Venn diagram
#128Earlier quoted context omitted.
> I've never claimed it doesn't. Yes you did: > You must specify the offset from where it should continue No, you mustn't, you can specify - and it does exactly what you want. The docs are very clear and even provide examples. At some point you should stop blaming curl for your inability to read a man page and admit that you were simply mistaken.
You still fail to understand that curl's -C does not behave as a simple flag but as an switch with a mandatory argument. And there's a magic special value for that argument that finally enables the expected behavior. It's unintuitive, hard to remember and not in agreement with usability. While I agree that curl is powerful I will not concede that it's CLI is user friendly.
It's not hard to remember if you're familiar with unixs tools and syntax. But no one is demanding you concede anything. The point of the conversation is explain the difference in expectations between how you expect a command line tool to act and how most people expect a command line tool to act. If I try to `cp src dest` and it fails. I don't want the tool to guess how to fix the issue, that's not it's job. Ditto for `dd` it shouldn't try to guess offsets. curl exists as a knife, you're expecting it to do the job have a food processor and blender combination. You're not wrong to expect a tool to behave that way you're wrong to expect curl to behave that way. And no one is asking you to concede that a knife is easier than a blender when you want a blender. everyone is pointing out it's not a blender it's a knife but you can still do everything you want to do.
Re: The curl-wget Venn diagram
#129I would also add at least "sane default options", "continues downloads" and "retries on error" to the Wget column. I recently had to write a script that downloads a very large file over a somewhat unreliable connection. The common wisdom among the engineers is that you need to use Wget for this job. I tried using curl but out of the box it could not resume or retry the download. I would have to study the manual and s…
Also add -i which lets wget read URLs from a file. In particular wget -i - which makes it read from standard input, and is very useful in pipelines. curl cannot, AFAIK, do this. People usually suggest using xargs, which is a mediocre substitute because it waits for all the URLs to arrive before invoking curl, giving up any chance at parallelism between the command generating the URLs and the one downloading them.
`--config filename` allows this. `--config -` for stdin. Not only urls, but any config options
echo '--url https://google.com/' | curl --config -Re: The curl-wget Venn diagram
#130Earlier quoted context omitted.
> I tried using curl but out of the box it could not resume or retry the download. Maybe I'm misunderstanding, but curl has exactly that feature, it's the `-C` flag. If you want retries, there's `--retry`. I find curls defaults pretty sane, personally, I wouldn't want either of those by default for a tool like curl.
Strong agree. The only misbehaviour I believe curl displays out of the box is globbing, which has burned me enough times that I’ve come to believe it would’ve been better disabled by default and enabled with -g instead of vice versa.
do you have an example of the globbing that burns you?