Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Curl vs. Wget

daniel.haxx.se

Post reply on HN
Print view View original

191–200 of 205 posts

Re: Curl vs. Wget

#191

Post by laumars »

hobarrera wrote: post #182
laumars wrote: post #161

Earlier quoted context omitted.

Yes the server would need to support it. The request is made via a HTTP header (Range[1] IIRC). Also, I wouldn't consider that a stupid question. :) [1] https://www.w3.org/Protocols/rfc2616/rfc2616-sec14.html#Rang...

This is also how download accelerators worked (back in the late nineties, early naught's), by having different connections work at several ranges to maximize bandwidth usage.

Funny enough, it was writing a download accelerator that taught me about the Range HTTP request header.

Re: Curl vs. Wget

#192

Post by tripzilch »

falcolas wrote: post #28

My favorite use of wget: mirroring web documentation to my local machine. wget -r -l5 -k -np -p https://docs.python.org/2/ Rewrites the links to point local where appropriate, and the ones which are not local remain links to the online documentation. Makes for a nice, seamless experience while browsing documentation. I also prefer wget to `curl -O` for general file downloads, simply because wget will handle redirects…

This is a great use, and I've used it for a lot of other sites and documentation. However I just want to point out that Python's documentation happens to come in an easily downloadable archive of HTML files:

https://docs.python.org/2.7/download.html

(you can also find archives for Python 3 and SciPy and NumPy)

Re: Curl vs. Wget

#193

Post by MoSal »

notfoss wrote: post #176

aria2 is much more reliable when downloading stuff, especially for links which involve redirections. For example here's a link to download 7zip for windows from filehippo.com. Results: * Curl doesn't download it at all. curl -O 'http://filehippo.com/download/file/bf0c7e39c244b0910cfcfaef2af45de88d8cae8cc0f55350074bf1664fbb698d/' gives: curl: Remote file name has no length! * Wget manages to download the file, but wit…

The URL does not work right now. But I tried another one from the same site.

No client can get this right, always. aria2c is not more reliable. It's just choosing to take the filename from the redirect URL. It appears to be the right thing to do in this case. But it would fail if the start URL was actually the one that had the right filename.

Hosts can use the Content-Disposition header if they want to make sure all (capable) clients get the right filename.

In saldl, I implemented `--filename-from-redirect` to handle your use-case. But It's not set by default.

Re: Curl vs. Wget

#194

Post by mistat »

ce4 wrote: post #170
mistat wrote: post #155

curl for checking http headers simply with: curl -vskL http:1.2.3.4 -H "Host: example.com" > /dev/null

Why don't you just use the -I flag?

For some sites a HEAD may return different headers than GET, so it is safer to return the results in full. Also using vsk shows the request headers, including IP so you can easily see if things such as roundrobin DNS is in use, again to assist with debugging.

Re: Curl vs. Wget

#195

Post by X-Istence »

lucb1e wrote: post #142
X-Istence wrote: post #132

I use curl because it is generally installed. I prefer not to install wget, especially on customer machines because it stops 90% of script kiddies. For some reason wget is the only tool they will attempt to use to download their sploit.

Pretty sure skiddies will not assume most victims have wget already, they'll just ship it with the exploit. If not installing wget is an annoyance to a hacker, they're already in too deep ;)

Your stock standard drive by PHP exploit attempts usually attempt to "wget" another PHP file to public_html.

They try wget, fail, and move on.

Re: Curl vs. Wget

#196

Post by awkward »

Legogris wrote: post #159
awkward wrote: post #105

Earlier quoted context omitted.

Huh. I've always thought of them as the opposite - wget is the full featured spidering tool, curl is the easy to run one when I need a command line thing or to bang web stuff into a janky copy and paste workflow.

This is interesting. Do you want to give your reasons or is it just an arbitrary habit?

wget leaves files everywhere, and isn't useful without picking a few flags to use, carefully. I use it as a tool to check builds for broken links

curl is a unix-way type program that interacts with standard out in a pretty predictable way.

Re: Curl vs. Wget

#197

Post by botw »

vacri wrote: post #141
lomnakkus wrote: post #117

Earlier quoted context omitted.

> 'curl' is 'cat url', Whoa... TIL something! I don't know if that's the official etymology, but that's a great mnemonic! EDIT: ... and yes, I use both tools :).

Well, it looks like I was taught wrong, and it's not 'cat URL' (though that's a good way to think of it), but the rather more direct-to-STDOUT-sounding 'see URL'. TIL something too :) https://en.wikipedia.org/wiki/CURL

My first impression is 'cat URL' not 'see URL'. The wikipedia article does not have to be official.

Re: Curl vs. Wget

#198

Post by notfoss »

MoSal wrote: post #193
notfoss wrote: post #176

aria2 is much more reliable when downloading stuff, especially for links which involve redirections. For example here's a link to download 7zip for windows from filehippo.com. Results: * Curl doesn't download it at all. curl -O 'http://filehippo.com/download/file/bf0c7e39c244b0910cfcfaef2af45de88d8cae8cc0f55350074bf1664fbb698d/' gives: curl: Remote file name has no length! * Wget manages to download the file, but wit…

The URL does not work right now. But I tried another one from the same site. No client can get this right, always. aria2c is not more reliable. It's just choosing to take the filename from the redirect URL. It appears to be the right thing to do in this case. But it would fail if the start URL was actually the one that had the right filename. Hosts can use the Content-Disposition header if they want to make sure all…

Thanks for the explanation. But generally I have found aria2 to be more reliable in such scenarios.

Re: Curl vs. Wget

#199

Post by jandrese »

paulmd wrote: post #187
brobinson wrote: post #116

Earlier quoted context omitted.

curl is the one where I have to remember whether to use -o or -O when trying to download a file with the original filename and just use wget instead because it's faster than reading the curl man page.

I have the same problem. Curl is "unix-y" in the sense that the default options are optimized for a shell script and make no sense for interactive usage. bropages is where it's at for that kind of stuff. curl is actually their usage example :) http://bropages.org/

Their example will just dump the webpage to stdout! Almost certainly not the desired behavior given the comment. Then they include second example of a use case that almost nobody has, instead of giving the option that everyone is actually looking for.

Re: Curl vs. Wget

#200

Post by jandrese »

LukeShu wrote: post #18
rogerbinns wrote: post #16

Earlier quoted context omitted.

Lets compare the length of the man page: $ man curl | wc -l 1728 $ man wget | wc -l 1096 How about the --help output? $ curl --help | wc -l 178 $ wget --help | wc -l 176 The wget help is nicer, grouping options together by category and with longer text. curl just has a long list of options in alphabetical order. How many (long) options do they have? $ curl --help | grep -- -- | wc -l 175 $ wget --help | grep -- -- |…

"I'm glad I typed `man wget` instead of `wget --help`" -- no one ever You want the `wget --help` text over the man page, 99% of the time. The other 1%, you want the full info manual. The man page is an awful mix between the two; too dense for scanning through for the flag you need, but not containing the full information when you need specifics.

I really wish the GNU foundation would give up on info pages. Just admit failure and condense them down into full info manpages that I can search easily instead of having to use their 1980s version of a web browser with its awful EMACS-like keybinds.
Post reply on HN