Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Why I forked httpx

tildeweb.nl

Post reply on HN
Print view View original

171–180 of 194 posts

Re: Why I forked httpx

#171

Post by Kwpolska »

xnorswap wrote: post #152
pixl97 wrote: post #135

Earlier quoted context omitted.

>"It's been around for a while" is 14 years not a while?

It is, but it's also a decade after the language was first released.

Python’s urllib2 (now urllib.request) started out in the year 2000 [0].

.NET’s WebRequest was available in .NET Framework 1.1 in 2003 [1].

But since then, Microsoft noticed the issues with WebRequest and came up with HttpClient in 2012. It has some issues and footguns, like those related to HttpClient lifetime, but it’s a solid library. On the other hand, the requests library for Python started in 2011 [2], but the stdlib library hasn’t seen many improvements.

[0] https://github.com/python/cpython/blob/6d7e47b8ea1b8cf82927d...

[1] https://learn.microsoft.com/en-us/dotnet/api/system.net.webr...

[2] https://github.com/psf/requests/blob/main/HISTORY.md#001-201...

Re: Why I forked httpx

#172

Post by paulddraper »

Kwpolska wrote: post #25

What is it about Python that makes developers love fragmentation so much? Sending HTTP requests is a basic capability in the modern world, the standard library should include a friendly, fully-featured, battle-tested, async-ready client. But not in Python, stdlib only has the ugly urllib.request, and everyone is using third party stuff like requests or httpx, which aren't always well maintained. (See also: packaging)

Web browsers -- LIKE THE THINGS THAT LIVE AND DIE ON HTTP -- didn't have an ergonomic HTTP API until 2017.

Node.js got its production version in 2023.

Rust doesn't include an HTTP client at all.

Even for stdlib that have a client, virtually none support HTTP/3, which is used for 30% of web traffic. [1]

--

HTTP (particularly 2+) is a complex protocol, with no single correct answers for high-level and low-level needs.

[1] https://radar.cloudflare.com/adoption-and-usage

Re: Why I forked httpx

#173

Post by Tostino »

ffsm8 wrote: post #160
Tostino wrote: post #120

Earlier quoted context omitted.

And those docs were likely trying to show you how to use multiple features, not the most basic implementation of it

I mean dont get me wrong, I work with Java basically 8 hours per day. I also get _why_ the API is as it is - It essentially boils down to the massive Inversion of Control fetish the Java ecosystem has. It does enable code that "hides" implementation very well, like the quoted examples authentication API lets you authenticate in any way you can imagine, as in literally any way imaginable. Its incredibly flexible. Want…

Thanks for the thoughtful response, I appreciate it.

Yeah, I remember the first time I encountered a spring project (well before boot was out) and just about lost my shit with how much magic was happening.

It is productive once you know a whole lot about it though, and I already had to make that investment so might as well reap the rewards.

Re: Why I forked httpx

#174

Post by Kwpolska »

woodruffw wrote: post #140
Kwpolska wrote: post #25

What is it about Python that makes developers love fragmentation so much? Sending HTTP requests is a basic capability in the modern world, the standard library should include a friendly, fully-featured, battle-tested, async-ready client. But not in Python, stdlib only has the ugly urllib.request, and everyone is using third party stuff like requests or httpx, which aren't always well maintained. (See also: packaging)

AFAICT, lacking a (good) standard HTTP library is kind of the norm in popular languages. Python, Ruby, Rust, etc. all either have a lackluster standard one or are missing one. I think it sits between two many decision pressures for most languages: there are a _lot_ of different RFCs both required and implied, lots of different idioms you could pick for making requests, lots of different places to draw the line on wha…

I thought Rust’s got a very small standard library, only focusing on things that must be in a standard library, mainly primitives or things which require co-operation with the underlying OS (e.g. thread and process management)? That’s completely opposite of Python’s “batteries included” approach.

Re: Why I forked httpx

#176

Post by woodruffw »

Kwpolska wrote: post #174
woodruffw wrote: post #140

Earlier quoted context omitted.

AFAICT, lacking a (good) standard HTTP library is kind of the norm in popular languages. Python, Ruby, Rust, etc. all either have a lackluster standard one or are missing one. I think it sits between two many decision pressures for most languages: there are a _lot_ of different RFCs both required and implied, lots of different idioms you could pick for making requests, lots of different places to draw the line on wha…

I thought Rust’s got a very small standard library, only focusing on things that must be in a standard library, mainly primitives or things which require co-operation with the underlying OS (e.g. thread and process management)? That’s completely opposite of Python’s “batteries included” approach.

Sure, I'm not making a categorical argument about big vs. small stdlibs. I'm just noting that "a good default HTTP library" is in fact kind of unusual, whether or not the language is batteries-included or not.

(As an outsider I had the impression that Go's net/http was good, but a lot of people in this thread are complaining about it as well. So it may be 0-4 instead of 1-3).

Re: Why I forked httpx

#177

Post by Yokohiii »

troad wrote: post #102
cachius wrote: post #100

Earlier quoted context omitted.

Right, frustration about the no strings attached sentiment for OSS devs. Of course you've no obligations for support or maintenance, but with increasing exposure responsibility grows as de facto ever more projects, people, softwares depend on you. This doesn't come over night and this is a spectrum and a choice. From purely personal side project over exotic Debian package to friggin httpx with 15k Github stars and 10…

>> Of course you've no obligations for support or maintenance, but with increasing exposure responsibility grows as de facto ever more projects, people, softwares depend on you. This is an oxymoron. Either you have obligations, or you don't. There's no such thing as having "no obligations" but also "growing responsibility". I don't understand how you can possibly conclude that just because you've chosen to become dep…

Yeah. Previous poster points out sources how a maintainer could get resources (money, support, etc). Maintainers may be exhausted or overwhelmed by the (imposed) responsibility / work. Actively acquiring those resources would just push that over the edge.

There is also the possibility that a maintainer simply doesn't care about what the community wants, it's his baby and he can do what he wants.

Forking a project is built-in by licensing. A lot of complaints, but those complainers don't fork. Why is that? Yeah right.

Side Note: Transferring projects to foundations etc with funding may be a solution for projects that are highly depended on and require active, reliable maintenance. They wont work well for innovation or experimentation. Just saying they are just a part of the equation and not the sole solution.

Re: Why I forked httpx

#178

Post by nateb2022 »

I'll plug Pyreqwest here: https://github.com/MarkusSintonen/pyreqwest

It's been a pleasure to use, has a httpx compatibility layer for gradually migrating to its API, and it's a lot more performant (right now, I think it's the most performant Python http client out there: https://github.com/MarkusSintonen/pyreqwest/blob/main/docs/b...)

Re: Why I forked httpx

#179

Post by renegat0x0 »

There are many nice http clients:

- httpx

- curl cffi

- httpmorph

- httpcloak

- stealth crawler

I wrote a framework, link below, which uses them all. You can compare each to verify crawling speed. Some sites can be cleanly crawled with a one particular framework.

Having read the article I am in a pain. I do break things while development. I rewrite stuff. Maybe some day I will find a way to develop things "stable". One thing I try to keep in good shape is 'docker' image. I update it once everything seems to be quite stable.

https://github.com/rumca-js/crawler-buddy

Re: Why I forked httpx

#180

Post by Pay08 »

woodruffw wrote: post #151
Pay08 wrote: post #150

Earlier quoted context omitted.

Is Rust popular? It's popular among HN users, and among certain other bubbles, but can it be called generally popular? Ruby sure can't be.

It's popular enough to be worth using as a datapoint. What's the point of the question?

I don't think it is worth using as a datapoint. Webdev is simply not what Rust was made for. It'd be somewhat like PHP having inline assembly.
Post reply on HN