Live data from Hacker News

Curl vs. Wget

daniel.haxx.se

81–90 of 205 posts

Re: Curl vs. Wget

#81

My favorite use of wget: mirroring web documentation to my local machine. wget -r -l5 -k -np -p https://docs.python.org/2/ Rewrites the links to point local where appropriate, and the ones which are not local remain links to the online documentation. Makes for a nice, seamless experience while browsing documentation. I also prefer wget to `curl -O` for general file downloads, simply because wget will handle redirects…

Or just curl -OL

Re: Curl vs. Wget

#82
post #72

Earlier quoted context omitted.

What do you even mean by logs in a session?

for example, in troubleshooting with a bluecoat proxy, I can run a curl session in conjuction with grep to check for very specific types of traffic and leave that script open while I might have an end user test.

Sorry, can't imagine what you mean. Do you just start curl with a list of urls to process and grep for errors? Any specific examples?

Re: Curl vs. Wget

#83

Earlier quoted context omitted.

I've never needed to do more configuration for python than whatever the installer does. What problem are you having?

I've had enough problems in the past that I try to steer clear at this point. I don't do much with python so it's hard to justify spending much time on it. I think my bigger issue is the lack of being able to do cinst httpie

The key is to stick with Python 2.7. Unfortunately pyhon 3 has significantly damaged the brand. Also installing msysgit will give you git and the msys command line which is a nice easy to install mostly UNIX command line. It is not a beast like cygwin.

With msysgit (for git and msys) you can practically live in the command line on Windows and with python 2.7 the py things mostly just work. pywin32 helps for some things and if pip install doesn't work there's always the unofficial windows binaries:

http://www.lfd.uci.edu/~gohlke/pythonlibs/

Re: Curl vs. Wget

#85
post #70

Earlier quoted context omitted.

Should have lead that comment with "I'm on Windows, so". Of course it's going to be miserable to use any command-line tool on Windows. It's Windows.

I'm not sure why you feel like that, I switched back to Windows after ~1 year of using OSX, I wouldn't say its "miserable" there is really nothing I could do in OSX's command line that can't do in windows.

What is your Windows command-line environment? Plain CMD prompt or Cygwin? Or something else?

Re: Curl vs. Wget

#86

My favorite use of wget: mirroring web documentation to my local machine. wget -r -l5 -k -np -p https://docs.python.org/2/ Rewrites the links to point local where appropriate, and the ones which are not local remain links to the online documentation. Makes for a nice, seamless experience while browsing documentation. I also prefer wget to `curl -O` for general file downloads, simply because wget will handle redirects…

Is this the part where I tell you about pydoc and blow your mind? I had a patch at one point to make pydoc style itself just like docs.p.o, but I'm not working much in Python these days. Maybe it's not even necessary any more, which would be cool.

Sure. Does it work with golang docs, the w3 HTML5 spec, Mozilla's JS reference, etc?

;)

Python's built in documentation tools are good, but the command I referenced also gives you the full language spec locally, as well as the FAQs and packaging docs.

Re: Curl vs. Wget

#87

If nobody's tried it, axel mentioned in the report as possibly abandoned has the awesome feature of splitting a download in to parts and then establishing that many concurrent TCP connections. Very useful on individual TCP flow rate-limited networks.

Axel is great, but nowadays I reach for aria2 instead, which can do the same, but also supports bittorrent and (s)ftp.

Re: Curl vs. Wget

#88

My favorite use of wget: mirroring web documentation to my local machine. wget -r -l5 -k -np -p https://docs.python.org/2/ Rewrites the links to point local where appropriate, and the ones which are not local remain links to the online documentation. Makes for a nice, seamless experience while browsing documentation. I also prefer wget to `curl -O` for general file downloads, simply because wget will handle redirects…

I bet that would be extra useful for creating Dash.app docsets...

if you look at some of the default docsets you can tell that this was most likely done. All of the directories under the file root in the docset are named after domain names (one of the things whet does when crawling a site and traversing multiple domains).

Re: Curl vs. Wget

#89
Really interesting. Under curl he has:

"Much more developer activity. While this can be debated, I consider three metrics here: mailing list activity, source code commit frequency and release frequency. Anyone following these two projects can see that the curl project has a lot higher pace in all these areas, and it has been so for 10+ years. Compare on openhub"

Under wget he has: "GNU. Wget is part of the GNU project and all copyrights are assigned to FSF. The curl project is entirely stand-alone and independent with no organization parenting at all with almost all copyrights owned by Daniel."

Daniel seems pretty wrong here. Curl does not require copyright assignment to him to contribute, and so, really, 389 people own the copyright to curl if the openhub data he points to is correct :)

Even if you give it the benefit of the doubt, it's super unlikely that he owns "almost all", unless there really is not a lot of outside development activity (so this is pretty incongruous with the above statement).

(I'm just about to email him with some comments about this, i just found it interesting)

Re: Curl vs. Wget

#90
post #80

My favorite use of wget: mirroring web documentation to my local machine. wget -r -l5 -k -np -p https://docs.python.org/2/ Rewrites the links to point local where appropriate, and the ones which are not local remain links to the online documentation. Makes for a nice, seamless experience while browsing documentation. I also prefer wget to `curl -O` for general file downloads, simply because wget will handle redirects…

Thanks for the jq tip. I hadn't seen that before. Link if anyone's interested: https://stedolan.github.io/jq/

Underscore CLI looks interesting as well, though I haven't personally had much of a chance to play with it. It does require NodeJS, which might be a deal-breaker for some, but if it's already in your toolchain then it might come in handy.

https://github.com/ddopson/underscore-cli

Post reply on HN