Live data from Hacker News

“I Could Rewrite Curl”

daniel.haxx.se

141–150 of 193 posts

Re: “I Could Rewrite Curl”

#141

Just a little tangent but I think everyone should know cURL… not postman, not httpie or any other more “friendly” tool. cURL is the standard, pre-installed, do-everything, scriptable tool to interact with web APIs. I could say mostly the same stuff about knowing Bash. It separates the barely effective from hyper-effective individuals on simple day to day (programming) tasks.

It shows how standard it is when you can right click a network request in browser dev tools and copy as cURL is there as one of a few other odd options. :)

Re: “I Could Rewrite Curl”

#142
post #106
post #30

Let's be honest here. Curl to 98% of people is http/1 requests with some post params and maybe json body, with some custom headers. Most language standard libs (which might be using libcurl) can facilitate writing that relatively quickly. And probably with a more 'modern' CLI. Curl though, is a very wide breadth project. It currently supports the following protocols: DICT, FILE, FTP, FTPS, GOPHER, HTTP, HTTPS, IMAP,…

A question that arises out of this is, should the 90% use cases be handled by a small, simple tool, or by the tip of the iceberg of a large, complex tool such as curl? I can see the advantages of standardizing on a complex, powerful tool for simple use cases. For one thing, it may be the only way to standardize: simple versions are too easy to write, and therefore you get dozens of competitors, none of whom are popul…

I would be surprised if nobody had tried to make a mini-curl that could go in busybox. The idea of having a tiny version of the program which handles the 90% cases by itself but can call out to the real-deal bigger brother when necessary is a nice one. This sort-of happens already with lots of common tools which are shadowed by shell builtins, why not curl?

Re: “I Could Rewrite Curl”

#143
post #68

Earlier quoted context omitted.

True. I'm kind of surprised (again) by Daniel after 20 years of maintaining curl being so salty about "people not appreciating the achievement". Yes, for overwhelming majority of users curl in 2021 is just a simple http/https request client. And, yes, in 2021 that could be written over a weekend with a better CLI, mainly because most of the languages already have all important stuff implemented in a [standard] librar…

> I'm kind of surprised (again) by Daniel after 20 years of maintaining curl being so salty about "people not appreciating the achievement" You're surprised by him being salty? I can't imagine what it's like to be on the receiving end of the water torture of people continually belittling the work you have done over 20 years and given to the community.

Not to mention the guys who threatened to kidnap/kill him if he didn't fly like 10000km immediately, on his own expense of course, to solve their bug pro bono.

Re: “I Could Rewrite Curl”

#144
post #72

Earlier quoted context omitted.

Not everything on top of HTTP comes through the browser, either ;)

More and more APIs use HTTP as the transport for various reasons, but one of them is that HTTP has considered large number of edge cases and has been tested "in the fire" - rolling your own custom API with security and encryption is much more fraught with dangers.

Except HTTP 2.0/3.0 has very little to do with HTTP 1.0, and is more akin to use TCP/UDP directly anyway.

It just happens to get a free pass over port 80.

Re: “I Could Rewrite Curl”

#145
post #72

Earlier quoted context omitted.

Not everything on top of HTTP comes through the browser, either ;)

More and more APIs use HTTP as the transport for various reasons, but one of them is that HTTP has considered large number of edge cases and has been tested "in the fire" - rolling your own custom API with security and encryption is much more fraught with dangers.

But I would bet that those only use the GET or POST functionality of HTTP, and that the real reason it's used is because it's easy to quickly get something up an running.

It's a short-term gain, but quite inefficient in the long run. ZeroMQ is a perfectly valid replacement for HTTP in internal networks. There's no reason for everything to be human readable.

Re: “I Could Rewrite Curl”

#146
post #49

The problem isn't CURL, but HTTP. It's actually suprisingly complex and really needs to die. That'll never happen though.

Plenty of applications use TCP and UDP, or even custom protocols on top of raw IP. Not everything needs to come through the browser.

Micro-services, for example, are pretty much all HTTP + JSON. The only place for HTTP is the browser and purely for historical reasons. Using it for anything server-server is a waste of network and CPU.

Re: “I Could Rewrite Curl”

#147

Earlier quoted context omitted.

exactly, my point was, some people think they can build it in a day, because they see their favorite library already supports sending HTTP, they might not be aware that some of those libraries are wrappers around cURL library

Seen this comment a lot in the thread, I'd like to know which libraries do. Google didn't help (as usual nowadays).

http://pycurl.io/

PycURL is a Python interface to libcurl.

Re: “I Could Rewrite Curl”

#148
post #30

Let's be honest here. Curl to 98% of people is http/1 requests with some post params and maybe json body, with some custom headers. Most language standard libs (which might be using libcurl) can facilitate writing that relatively quickly. And probably with a more 'modern' CLI. Curl though, is a very wide breadth project. It currently supports the following protocols: DICT, FILE, FTP, FTPS, GOPHER, HTTP, HTTPS, IMAP,…

This begs the question of why all of these protocols should be in one library, instead of one each. If a project is only going to ever use http, why would I need to bundle in all of the others? Genuinely curious what the advantages are

http://begthequestion.info/

Re: “I Could Rewrite Curl”

#149
post #142
post #106

Earlier quoted context omitted.

A question that arises out of this is, should the 90% use cases be handled by a small, simple tool, or by the tip of the iceberg of a large, complex tool such as curl? I can see the advantages of standardizing on a complex, powerful tool for simple use cases. For one thing, it may be the only way to standardize: simple versions are too easy to write, and therefore you get dozens of competitors, none of whom are popul…

I would be surprised if nobody had tried to make a mini-curl that could go in busybox. The idea of having a tiny version of the program which handles the 90% cases by itself but can call out to the real-deal bigger brother when necessary is a nice one. This sort-of happens already with lots of common tools which are shadowed by shell builtins, why not curl?

Perhaps not exactly what you are looking for but there is: https://curl.se/tiny/

Re: “I Could Rewrite Curl”

#150
post #15

What's frustrating is people who think that they can radically simplify something without knowing anything about it. The same people think they can implement a text editor in 1000 lines of code because that's how the first text editors were made. Why can't we go back to those simple days? We can't because modern software handles cases that those didn't. For example, rendering and editing text seems straightforward un…

> What's frustrating is people who think that they can radically simplify something without knowing anything about it. I run into this problem with my manager a lot, who knows just enough to get himself in trouble. There's probably a term for it, but I call it the "complexity trap". He sees a problem, thinks he has an "aha!" moment, and tells me about a simple solution, and asks me to implement it. This solution, of…

I’ve found that a small amount of mathematical rigor reduces the size of most complex code by a few orders of magnitude.

It has the effect of short circuiting the complexity of the problem, because it defines all the corner cases in a compact form.

Interface with sloppy spaghetti systems can usually be hidden away at the edges of the system. The main problem is that less experienced engineers invariably come in and say, “A ha! I have a special corner case that doesn’t fit the abstraction.” and then try to add copy paste methods, boundary violations, and so on.

As long as there’s someone that diligently guards against such things, it works out OK. Results vary wildly after those people leave.

HP printer drivers are a famous example of the sort of complexity reduction I’m talking about. They used to copy the entire source tree for each printer (font rasterizer, dithering algorithms, and all), then assign a full time team to maintain it.

Eventually they had many thousands of full time engineers maintaining the result, and print quality was still terrible across the product line.

The open source drivers reverse engineered the printers, and factored out as much common logic as possible. The printer specific code was then reduced to just implementing the wire protocol and device geometry stuff. At that point, they had better output than HP, with 1% the developer resources.

Someone at HP did the same thing internally, reimplemented all the drivers in a unified way, and was promptly promoted to the executive team.

Post reply on HN