Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Httpx: A next-generation HTTP client for Python

python-httpx.org

Post reply on HN
Print view View original

101–110 of 119 posts

Re: Httpx: A next-generation HTTP client for Python

#101

Post by tomchristie »

hermitdev wrote: post #78
vajenetehais wrote: post #25

Earlier quoted context omitted.

I'm currently using aiohttp. There is features I don't find in butterfly docs like : limit connection, limit connection per host or per time frame. Do you know if those exist in butterfly? I'm already deep in aiohttp and it's not an easy task to learn an async client (at least in my case, but I'm no dev) so if I do switch it would be for more features (retry option on exceptions, limit requests per host and per time…

I prototyped moving to aiohttp from requests/multiprocessing. The speedup was amazing. Saw something like a 60% reduction in runtime for our use case. Only reason the code hasn't gone live: we currently use requests_negotiate_sspi for authentication, which sadly isn't supported by aiohttp. Not sure if httpx supports it. Looks like it might. Planning on giving it a shot next week.

We don’t have any third party packages for that authentication style yet, but we do have an API for supporting custom auth flows... https://www.python-httpx.org/advanced/#customizing-authentic...

If you’re interested in trying httpx you’d be very welcome to raise an issue an issue related to NTLM/Negotiate authentication - it’d be really helpful for us to work through that and figure out if our auth API is sufficient for implementing that, or if there’s anything we’re missing.

Re: Httpx: A next-generation HTTP client for Python

#102

Post by tialaramex »

dragonwriter wrote: post #71
neurostimulant wrote: post #46

Earlier quoted context omitted.

Plenty of reason why it's hard to ship in standard library. Here's some off top of my mind: - Should the library includes its own CA store, or use the system's CA store? These kind of library often include their own CA store (since they changes often), and httpx seem to use 3rd party lib to handle that (certifi). This is hard to do in a standard library for variety of reasons (users rarely update their python install…

> Should the library includes its own CA store, or use the system's CA store? The CA store should be a configurable option, and one of the supported options should be the system CA store. > The user will want tighter release schedule than python's so they can get these stuff sooner. Ruby is moving stdlib to default and bundled gems, which addresses this. There's no reason that “delivered with the interpreter” needs t…

> The CA store should be a configurable option, and one of the supported options should be the system CA store.

It's more complicated than that, especially if you aren't on Linux.

On both the other two big general purpose platforms (Mac OS, Windows) the vendor provides a library which implements their preferred trust rules as well as using their trust store.

On Linux what you usually get is the list of Mozilla trusted root CAs and you're left to your own devices. Mozilla's trusted list is IMNSHO a shorter more trustworthy list than supplied by Apple or Microsoft, but it misses nuance.

When Mozilla makes a nuanced trust decision for the Firefox browser that decision doesn't magically reflect in an OpenSSL setup on a Linux server. Say they decide that Safety Corp. CA #4 can be relied upon to put the right dates on things, but its DNS name checks are inadequate and no longer to be trusted after June 2019. Firefox can implement that rule, and distrust sites with an August 2019 cert from Safety Corp. CA #4, while still trusting say, the Safety Corp. CA #4 certificate on Italian eBay from March 2018. But there's no way for your Python code to achieve the same outcome relying on OpenSSL.

Python's key people seem to think that it's better for Python to try to mimic what a "native" web browser would do because that's least surprising. So on Windows a future Python will trust Microsoft's decisions, on macOS they'd be Apple's decisions and only on Linux will it be Mozilla decisions. Today it's Mozilla's trust store everywhere.

Hypothetically in the ideal case you'd have your own PKI and you'd have all the necessary diligence in place, hire your own auditors, maybe even have contractors red-teaming the CA you trust for you - but we don't live in a world anything like that, most people are implicitly reliant on the Web PKI and probably tools like these needs to accept that.

Re: Httpx: A next-generation HTTP client for Python

#103

Post by _bohm »

ryanmccullagh wrote: post #66

I like async support, but always, in async examples, doesn't the await keyword just block anyways? I like to see the usage of async inside of a main loop, where other things can be processed while waiting for the async response to come in. Further, not all applications have the idea of an event loop, so async may not be needed, but it's useful to have the option. I use async operations to multiplex operations in a qu…

> doesn't the await keyword just block anyways?

The await keyword only blocks the execution of the coroutine in which it is used. It releases the event loop so that other coroutines can continue processing while the result of an awaitable is being fetched.

Re: Httpx: A next-generation HTTP client for Python

#104

Post by takeda »

penagwin wrote: post #37
yegle wrote: post #20

> Fully type annotated. This is a huge win compare to requests. AFAIKT requests is too flexible (read: easier to misuse) and difficult to add type annotation now. The author of requests gave a horrible type annotation example here [0]. IMO at this time when you evaluate a new Python library before adopting, "having type annotation" should be as important as "having decent unit test coverage". [0]: https://lwn.net/Art…

> "having type annotation" should be as important as "having decent unit test coverage". I mean this is pretty spot on IMO. I've worked with many languages, and have concluded that having a powerful type system catches soooo many bugs before you even try to run the code. And they're usually "stupid" bugs too, forgetting to sanitize inputs etc. Even worse is when a language tries to be "smart", so you end up with "1"…

> And they're usually "stupid" bugs too, forgetting to sanitize inputs etc. Even worse is when a language tries to be "smart", so you end up with "1" + 2 = "12" and no errors at all.

The last one is no longer possible in Python 3 and will throw an error. That's another thing I am glad they fixed.

Re: Httpx: A next-generation HTTP client for Python

#105

Post by takeda »

anderskaseorg wrote: post #75
strbean wrote: post #73

Earlier quoted context omitted.

Some parts of that example seem pretty silly. URLs for example: he describes how just about anything can be passed in as a URL, and requests attempts to call __str__ or __unicode__ on it and then parse it. Considering (essentially?) all types in Python have __str__, this is a perfectly reasonable place to use the Any type. The issue isn't with the expressiveness of the typing system, but with the absurd flexibility o…

The right type is ‘object’, not ‘Any’. ‘object’ is the base class of all types: you can put anything into an ‘object’, and you can only do very generic operations like str() on what you get out of an ‘object’ without further checks like isinstance(). ‘Any’ is an unsound escape hatch that disables type checking: you can put anything into an ‘Any’, and you can do anything with what you get out of an ‘Any’, and the type…

Actually the proper way to handle is to use protocols. They provide already SupportsBytes type, and you can similarly create custom SupportsStr, but IMO it is silly to do that. This approach increases chance of bugs, and it's just easy to use str type and for developer to wrap such value in str(). I believe that's why SupportsStr was not introduced.

Re: Httpx: A next-generation HTTP client for Python

#106

Post by neuland »

fulafel wrote: post #98
dragonwriter wrote: post #77

Earlier quoted context omitted.

If you work in a corporate environment, you would probably notice that systems that insist on bundling their own certs without an easy to activate option of using system cert store are broken. (And even if the library has an easy to use option, if it's easy to not expose it, much software built on the library will still be broken.) People should be empowered to substitute cert stores, but the system store should be t…

A lot of corporate environments are fortunately legally bound and/or princilped about not doing mitm and are using the happy path of internet CAs.

You don't need to be doing MITM to get value out of using an internal CA.

Re: Httpx: A next-generation HTTP client for Python

#107

Post by fulafel »

neuland wrote: post #106
fulafel wrote: post #98

Earlier quoted context omitted.

A lot of corporate environments are fortunately legally bound and/or princilped about not doing mitm and are using the happy path of internet CAs.

You don't need to be doing MITM to get value out of using an internal CA.

It's a pain, sometimes it's worth it, oftentimes not.

Re: Httpx: A next-generation HTTP client for Python

#109

Post by tiangolo »

bransonf wrote: post #38
tiangolo wrote: post #22

FastAPI creator here... if you use FastAPI, HTTPX would probably be the best match for sending requests, just saying... :D

Sometimes I find it hard to justify all the time I spend on HN. But discovering FastAPI is probably going to compensate for plenty of this time. This is a really awesome library, thanks!

Thank you! :D
Post reply on HN