Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Curl vs. Wget

daniel.haxx.se

Post reply on HN
Print view View original

181–190 of 205 posts

Re: Curl vs. Wget

#181

Post by hobarrera »

majewsky wrote: post #156
manyxcxi wrote: post #107

Earlier quoted context omitted.

> This doesn't really need to be an "emacs vs. vim" or "tabs vs. spaces"-type dichotomy: wget and curl do different things well and there's no reason why both shouldn't coexist in one's workflow. Totally agree. I love curl for testing API request/responses manually. It's usually a huge part of navigating my way around a new API that doesn't have a client library available for whatever language I'm using at that time.…

Random plug: Another invaluable tool for API analysis is http://mitmproxy.org .

Wow, this looks extremely interesting, I'll definitely keep this in mind!

Re: Curl vs. Wget

#182

Post by hobarrera »

laumars wrote: post #161
foota wrote: post #154

Earlier quoted context omitted.

Stupid question, but how do things like this resume from where they left off? Wouldn't the server need to be cooperating in this? Is that build into HTTP?

Yes the server would need to support it. The request is made via a HTTP header (Range[1] IIRC). Also, I wouldn't consider that a stupid question. :) [1] https://www.w3.org/Protocols/rfc2616/rfc2616-sec14.html#Rang...

This is also how download accelerators worked (back in the late nineties, early naught's), by having different connections work at several ranges to maximize bandwidth usage.

Re: Curl vs. Wget

#183

Post by hobarrera »

geerlingguy wrote: post #13

Also putting this out there—for nicer REST API interaction on the CLI, and a little more user-friendliness, you might also want to add HTTPie[1] to your toolbelt. It's not going to replace curl or wget usage, but it is a nicer interface in certain circumstances. [1] https://github.com/jkbrzt/httpie

I find httpie's lack of manpage quite disturbing though.

Re: Curl vs. Wget

#184

Post by [deleted] »

viperscape wrote: post #171
zo1 wrote: post #160

Earlier quoted context omitted.

You specify a byte "range" when requesting a file via HTTP from the server. Not all servers support this, but most do these days. #Edit. More info here: https://en.wikipedia.org/wiki/Byte_serving

Just looked that up, that's a rather limited usefulness. The multi homed usage makes sense but feels out of place really.

[deleted]

Re: Curl vs. Wget

#185

Post by fosco »

viraptor wrote: post #100
fosco wrote: post #97

Earlier quoted context omitted.

it acts like a live packet capture of a log file. not sure how else to describe other than I do not think wget is capable of such functionality.

sooo... "wget -O - .... | grep ..." ?

havent not worked with it in that matter, however like someone above said.

typically I use wget to download, and curl to troubleshoot http https.

however, if that is capable of keeping everything open I suppose thats a point for wget.

Re: Curl vs. Wget

#186

Post by mhils »

majewsky wrote: post #156
manyxcxi wrote: post #107

Earlier quoted context omitted.

> This doesn't really need to be an "emacs vs. vim" or "tabs vs. spaces"-type dichotomy: wget and curl do different things well and there's no reason why both shouldn't coexist in one's workflow. Totally agree. I love curl for testing API request/responses manually. It's usually a huge part of navigating my way around a new API that doesn't have a client library available for whatever language I'm using at that time.…

Random plug: Another invaluable tool for API analysis is http://mitmproxy.org .

Thanks for mentioning it! :) One of the authors here, happy to answer any questions.

Re: Curl vs. Wget

#187

Post by paulmd »

brobinson wrote: post #116
JonathonW wrote: post #31

For my usage: * Wget's the interactive, end-user tool, and my go-to if I just need to download a file. For that purpose, its defaults are more sane, its command line usage is more straightforward, its documentation is better-organized, and it can continue incomplete downloads, which curl can't. * Curl's the developer tool-- it's what I'd use if I were building a shell script that needed to download. The command line…

curl is the one where I have to remember whether to use -o or -O when trying to download a file with the original filename and just use wget instead because it's faster than reading the curl man page.

I have the same problem. Curl is "unix-y" in the sense that the default options are optimized for a shell script and make no sense for interactive usage.

bropages is where it's at for that kind of stuff. curl is actually their usage example :)

http://bropages.org/

Re: Curl vs. Wget

#188

Post by voltagex_ »

stephengillie wrote: post #169
jsingleton wrote: post #166

Earlier quoted context omitted.

Yes, wget is fantastic for mirroring www and ftp sites and I use it a lot for that purpose. It's magic [0]. I hadn't realized that it didn't support compression though, which might explain why it's so slow in some cases. Not normally a problem as it just runs in the background on a schedule. Curl supports gzip and deflate. It would be great if support for sdch and br were added too. Brotli is in Firefox 44 and can be…

The Win64 latest version of curl doesn't seem to support gzip nor deflate. I have to remove those options when copying from Chrome developer tools and pasting into a script. I'd report a bug but their page doesn't seem to have an obvious link.

I wonder if the Win64 version is built without zlib or similar.

Re: Curl vs. Wget

#189

Post by goldbrick »

troyvit wrote: post #115
xd1936 wrote: post #56

wget has the amazing flag `--page-requisites` though, which downloads all of an html documents' css and images that you might need to display it properly. Lifesaver.

wget has another great flag, -k, which changes references to the css, js, and images to absolute URLs, resulting in a 1 page download that still looks like the original page. It's useful for making dummy pages for clients. I wish curl had this for my OSX friends who need the functionality above. Getting a wget binary onto OSX is a pain but curl is there by default.

wget is on homebrew.

Re: Curl vs. Wget

#190

Post by voltagex_ »

baldfat wrote: post #179
JonathonW wrote: post #31

For my usage: * Wget's the interactive, end-user tool, and my go-to if I just need to download a file. For that purpose, its defaults are more sane, its command line usage is more straightforward, its documentation is better-organized, and it can continue incomplete downloads, which curl can't. * Curl's the developer tool-- it's what I'd use if I were building a shell script that needed to download. The command line…

I totally agree except over the years I have been using aria2 more and more instead of wget. aria2 supports HTTP/HTTPS, FTP, SFTP, BitTorrent and Metalink with the same sane wget syntax and defaults. example $ aria2c http://yourlink.com/file.* -x2 allows using 2 connections per host. https://aria2.github.io/

I'm having to use aria2 more and more to maintain decent download speeds, occasionally even from Akamai and CloudFront. S3 seems to be the exception.
Post reply on HN