Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Why I forked httpx

tildeweb.nl

Post reply on HN
Print view View original

161–170 of 194 posts

Re: Why I forked httpx

#161

Post by jerf »

umvi wrote: post #58
maccard wrote: post #29

Earlier quoted context omitted.

> Then I found out it was broken. I contributed a fix. The fix was ignored and there was never any release since November 2024. This seems like a pretty good reason to fork to me. > Sending HTTP requests is a basic capability in the modern world, the standard library should include a friendly, fully-featured, battle-tested, async-ready client. But not in Python, Or Javascript (well node), or golang (http/net is _wors…

What's wrong with Go's? I've never had any issues with it. Go has some of the best http batteries included of any language

Go's net/http Client is built for functionality and complete support of the protocol, including even such corner cases as support for trailer headers: https://developer.mozilla.org/en-US/docs/Web/HTTP/Reference/... Which for a lot of people reading this message is probably the first time they've heard of this.

It is not built for convenience. It has no methods for simply posting JSON, or marshaling a JSON response from a body automatically, no "fluent" interface, no automatic method for dealing with querystring parameters in a URL, no direct integration with any particular authentication/authorization scheme (other than Basic Authentication, which is part of the protocol). It only accepts streams for request bodys and only yields streams for response bodies, and while this is absolutely correct for a low-level library and any "request" library that mandates strings with no ability to stream in either direction is objectively wrong, it is a rather nice feature to have available when you know the request or response is going to be small. And so on and so on.

There's a lot of libraries you can grab that will fix this, if you care, everything from clones of the request library, to libraries designed explicitly to handle scraping cases, and so on. And that is in some sense also exactly why the net/http client is designed the way it is. It's designed to be in the standard library, where it can be indefinitely supported because it just reflects the protocol as directly as possible, and whatever whims of fate or fashion roll through the developer community as to the best way to make web requests may be now or in the future, those things can build on the solid foundation of net/http's Request and Response values.

Python is in fact a pretty good demonstration of the risks of trying to go too "high level" in such a client in the standard library.

Re: Why I forked httpx

#162

Post by zahlman »

> The fix was ignored and there was never any release since November 2024. Me, and others, asked repeatedly for a release containing my fix. I sent email to the author personally. I got response when I added that I was considering forking. The author replied “1.0 development is on course”.... I do understand about maintainer burnout, and preferring to work on ‘next’, and that there is life outside of Python, but I think not doing anything for maintenance and also not letting other people help out in maintaining, for such a high profile module, is problematic.

I feel like it's counterproductive in situations like this to mention forking. It will come across like a threat, when there isn't really anything intrinsically aggressive about it. So just do it; and when you have a decent amount of separate development, you can decide whether to make PRs back, advertise your fork, etc.

Re: Why I forked httpx

#163

Post by samset7 »

_boffin_ wrote: post #158
samset7 wrote: post #92

Earlier quoted context omitted.

We have switched to niquests in my company and yes I can confirm that it's 10x better than httpx :)

What issues do / did you have with HTTPx?

The main pain points for us were: thread-safety issues (httpx claims to be thread-safe but we hit race conditions in production), no HTTP/3 support, and the redirect behavior requiring explicit opt-in everywhere. Also the multiplexing story in httpx is quite limited compared to what niquests offers out of the box. On top of that, httpx maintenance has been slow to acknowledge valid bug reports, the thread-safety issue took over a year to even be acknowledged...

Re: Why I forked httpx

#164

Post by zahlman »

awkwardpotato wrote: post #90
ffsm8 wrote: post #74

Earlier quoted context omitted.

Ahh, java. You never change, even if you're modern HttpClient client = HttpClient.newBuilder() .version(Version.HTTP_1_1) .followRedirects(Redirect.NORMAL) .connectTimeout(Duration.ofSeconds(20)) .proxy(ProxySelector.of( new InetSocketAddress("proxy.example.com", 80) )) .authenticator(Authenticator.getDefault()) .build(); HttpResponse response = client.send(request, BodyHandlers.ofString()); System.out.println(respon…

What's the matter with this? It's a clean builder pattern, the response is returned directly from send. I've certainly seen uglier Java

> What's the matter with this? It's a clean builder pattern

I feel like you answered yourself. Java makes you do this by not supporting proper keyword arguments.

Re: Why I forked httpx

#165

Post by samset7 »

PyWoody wrote: post #129
samset7 wrote: post #92

Earlier quoted context omitted.

We have switched to niquests in my company and yes I can confirm that it's 10x better than httpx :)

Did you have any warts when switching? httpx has been "fine" for me but this thread has me seriously considering changing to niquests.

The switch was surprisingly smooth. I think there's an official migration guide in the doc. Honestly the API is closer to the classic requests library so nobody will be lost.

Re: Why I forked httpx

#166

Post by dirkc »

Pay08 wrote: post #132
dirkc wrote: post #71

Earlier quoted context omitted.

Yes, thankfully! It's amusing to read what they say about fetch on nodejs.org [1]: > Undici is an HTTP client library that powers the fetch API in Node.js. It was written from scratch and does not rely on the built-in HTTP client in Node.js. It includes a number of features that make it a good choice for high-performance applications. [1] - https://nodejs.org/en/learn/getting-started/fetch

Why is it amusing?

I say amusing because it points out that something I (and many other people) assume to be basic clearly has a lot more nuance to it.

Re: Why I forked httpx

#167

Post by denimnerd42 »

yoyohello13 wrote: post #155
denimnerd42 wrote: post #139

Earlier quoted context omitted.

the batteries included approach is the stdlib that can do everything. turns out it’s hard to maintain and make good.

Yeah that's true. Go seems to be handling the 'fat stdlib' approach pretty well though. I really don't want Python to got the path of Rust where nothing is included.

I feel like Java does it the best. Golang didn't start with generics so it's a bit odd IMO.

Re: Why I forked httpx

#168

Post by zahlman »

Spivak wrote: post #51
localuser13 wrote: post #34

Earlier quoted context omitted.

>Honestly the thing that consistently surprises me is that requests hasn't been standardised and brought into the standard library Instead, official documentation seems comfortable with recommending a third party package: https://docs.python.org/3/library/urllib.request.html#module... >The Requests package is recommended for a higher-level HTTP client interface. Which was fine when requests were the de-facto-standard…

Well, the reason for all the fragmentation is because the Python stdlib doesn't have the core building blocks for an async http or http2 client in the way requests could build on urllib. The h11, h2, httpcore stack is probably the closest thing to what the Python stdlib should look like to end the fragmentation but it would be a huge undertaking for the core devs.

> but it would be a huge undertaking for the core devs.

More importantly, it would be massively breaking to remove the existing functionality (and everyone would ignore a deprecation), and confusing not to (much like it was when 2.x had both "urllib" and "urllib2").

It'd be nice to have something high level in the standard library based on urllib primitives. Offering competition to those, not so much.

Re: Why I forked httpx

#169

Post by zahlman »

yoyohello13 wrote: post #136
BigTTYGothGF wrote: post #133

Earlier quoted context omitted.

I think the python maintainers are still feeling burnt by the consequences of the "batteries included" approach from the old times.

Most Python developers these days weren't even programming when the 2 -> 3 split happened. Unless you're referencing something else.

There are quite a few old hands among Python core devs. Certainly the culture of that burnout is in place, if you look at the responses that proposals for new standard library additions get these days. There also seems to be a lot of trauma from the loud complaints about backward compatibility breaks.

I still hear people complain about how such and such removal between "minor versions" of Python 3 (you really should be thinking of them as major versions nowadays — "Python 3 is the brand", the saying goes now), where they were warned like two years in advance about individual functions, supposedly caused a huge problem for them. It's hard for me to reconcile with the rhetoric I've heard in internal discussions; they're so worried in general about possible theoretical compatibility breaks that it seems impossible to change anything.

Re: Why I forked httpx

#170

Post by zahlman »

duskdozer wrote: post #96
matheusmoreira wrote: post #82

Earlier quoted context omitted.

Everybody's got a different idea of what it means for a library to be "friendly" and "fully-featured" though. It's probably better to keep the standard library as minimal as possible in order to avoid enshrining bad software. Programming languages could have curated "standard distributions" instead that include all the commonly used "best practice" libraries at the time.

https://xkcd.com/927/

That isn't really what was proposed, and is an unnecessarily snarky way to respond.
Post reply on HN