Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Curl vs. Wget

daniel.haxx.se

Post reply on HN
Print view View original

71–80 of 205 posts

Re: Curl vs. Wget

#71

Post by phs2501 »

ars wrote: post #55
LukeShu wrote: post #51

Earlier quoted context omitted.

For all of the reasons you gave (except search--that's what grep is for), I usually reach for `man`. But, for wget the information density of the man page is just wrong. At least these days it has some more information in it--it used to just be a reformatted version of the --help text, plugged into a generic template.

grep on help output is annoying sometimes since many programs send it on stderr and you need to redirect it if you want to pipe it to grep. Plus even if grep matched something you can't read the context without extra options. man is much easier than doing all that, but the time you have your full search command you would have already gotten your info from man.

|& is useful in these cases, as it redirects both stdout and stderr to the piped process's stdin. It's a cshism, but it works in both zsh and modern bash. Much nicer than typing cmd 2>&1 | cmd.

For context try grep -2, where 2 is the desired lines of context.

  $ wget --help |& grep -2 base   
    -i,  --input-file=FILE           download URLs found in local or external FILE
    -F,  --force-html                treat input file as HTML
    -B,  --base=URL                  resolves HTML input-file links (-i -F)
                                       relative to URL
         --config=FILE               specify config file to use
  --
                                       existing files (overwriting them)
    -c,  --continue                  resume getting a partially-downloaded file
         --start-pos=OFFSET          start downloading from zero-based position OFFSET
         --progress=TYPE             select progress gauge type
         --show-progress             display the progress bar in any verbosity mode
(Not that I'm arguing that this is an excuse for wget's [and GNU projects' in general] man pages sucking, but it's a useful workaround.)

Re: Curl vs. Wget

#72

Post by fosco »

viraptor wrote: post #20
fosco wrote: post #6

Curl scripts allow open connection to view all new logs in a session. can wget do similar? I did not know it can or could however from my point of view if it cannot this is like comparing a philips head screwdriver to a powertool with 500pc set.

What do you even mean by logs in a session?

for example, in troubleshooting with a bluecoat proxy, I can run a curl session in conjuction with grep to check for very specific types of traffic and leave that script open while I might have an end user test.

Re: Curl vs. Wget

#73

Post by Spooky23 »

StreamBright wrote: post #4

There is no other industry where tools are debated so much as in IT. We literally waste tonns of hours on arguing over minor differences and nuances that really should not matter that much.

It is a waste of time, clearly whet and curl are just petty toys. Emacs is obviously the one true tool to meet any purpose.

Re: Curl vs. Wget

#74

Post by hcarvalhoalves »

contingencies wrote: post #68

If nobody's tried it, axel mentioned in the report as possibly abandoned has the awesome feature of splitting a download in to parts and then establishing that many concurrent TCP connections. Very useful on individual TCP flow rate-limited networks.

Haven't found anything better than axel to saturate the link yet.

Re: Curl vs. Wget

#75

Post by tomphoolery »

falcolas wrote: post #28

My favorite use of wget: mirroring web documentation to my local machine. wget -r -l5 -k -np -p https://docs.python.org/2/ Rewrites the links to point local where appropriate, and the ones which are not local remain links to the online documentation. Makes for a nice, seamless experience while browsing documentation. I also prefer wget to `curl -O` for general file downloads, simply because wget will handle redirects…

I bet that would be extra useful for creating Dash.app docsets...

Re: Curl vs. Wget

#76

Post by ars »

phs2501 wrote: post #71
ars wrote: post #55

Earlier quoted context omitted.

grep on help output is annoying sometimes since many programs send it on stderr and you need to redirect it if you want to pipe it to grep. Plus even if grep matched something you can't read the context without extra options. man is much easier than doing all that, but the time you have your full search command you would have already gotten your info from man.

|& is useful in these cases, as it redirects both stdout and stderr to the piped process's stdin. It's a cshism, but it works in both zsh and modern bash. Much nicer than typing cmd 2>&1 | cmd . For context try grep -2 , where 2 is the desired lines of context. $ wget --help |& grep -2 base -i, --input-file=FILE download URLs found in local or external FILE -F, --force-html treat input file as HTML -B, --base=URL res…

I know how to do it, that's not the question.

But rather, doing all that seems easier than man to you?

Re: Curl vs. Wget

#77

Post by eru »

theoh wrote: post #47
eru wrote: post #35

Earlier quoted context omitted.

Looks like a frontend to curl that has wget's defaults would be useful.

Merge the code, hard link the filenames to the same executable and get it to change behavior based on the name it's invoked under.

No need to merge code. Take curl as is, and set different defaults based on executable name.

Re: Curl vs. Wget

#78

Post by hartator »

> Wget requires no extra options to simply download a remote URL to a local file, while curl requires -o or -O.

I think this is oddly the major reason why wget is more popular. Saving 3 chars + not having to remember the specific curl flag seems to matter more than what we can think.

Re: Curl vs. Wget

#79

Post by jsmthrowaway »

falcolas wrote: post #28

My favorite use of wget: mirroring web documentation to my local machine. wget -r -l5 -k -np -p https://docs.python.org/2/ Rewrites the links to point local where appropriate, and the ones which are not local remain links to the online documentation. Makes for a nice, seamless experience while browsing documentation. I also prefer wget to `curl -O` for general file downloads, simply because wget will handle redirects…

Is this the part where I tell you about pydoc and blow your mind?

I had a patch at one point to make pydoc style itself just like docs.p.o, but I'm not working much in Python these days. Maybe it's not even necessary any more, which would be cool.

Re: Curl vs. Wget

#80

Post by daveguy »

falcolas wrote: post #28

My favorite use of wget: mirroring web documentation to my local machine. wget -r -l5 -k -np -p https://docs.python.org/2/ Rewrites the links to point local where appropriate, and the ones which are not local remain links to the online documentation. Makes for a nice, seamless experience while browsing documentation. I also prefer wget to `curl -O` for general file downloads, simply because wget will handle redirects…

Thanks for the jq tip. I hadn't seen that before. Link if anyone's interested:

https://stedolan.github.io/jq/

Post reply on HN