Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Httpx: A next-generation HTTP client for Python

python-httpx.org

Post reply on HN
Print view View original

71–80 of 119 posts

Re: Httpx: A next-generation HTTP client for Python

#71

Post by dragonwriter »

neurostimulant wrote: post #46
pbreit wrote: post #18

I don't understand why this is such a complicated category and that many platforms do not have solid http clients in standard library. On every single project I do, it's just a bunch of posting JSON and getting a response synchronously. Over and over.

Plenty of reason why it's hard to ship in standard library. Here's some off top of my mind: - Should the library includes its own CA store, or use the system's CA store? These kind of library often include their own CA store (since they changes often), and httpx seem to use 3rd party lib to handle that (certifi). This is hard to do in a standard library for variety of reasons (users rarely update their python install…

> Should the library includes its own CA store, or use the system's CA store?

The CA store should be a configurable option, and one of the supported options should be the system CA store.

> The user will want tighter release schedule than python's so they can get these stuff sooner.

Ruby is moving stdlib to default and bundled gems, which addresses this. There's no reason that “delivered with the interpreter” needs to mean “frozen with the interpreter”.

Re: Httpx: A next-generation HTTP client for Python

#72

Post by takeda »

viraptor wrote: post #67
takeda wrote: post #47

Earlier quoted context omitted.

It's simple. Python was created before HTTP even existed. Since then a lot of things had changed, and once you create an API, it is hard to rewrite it when new use patterns emerge. It's easier for a 3rd party package to come up with a better api, because they can start brand new. Also when there's a radical change it is easier for a new 3rd package to take over. Httpx is example of such evolution, although not due to…

Python was created before json existed, and yet: https://docs.python.org/3/library/json.html It's not a hard rule, sometimes things do end up in the standard library.

Of course it could be added later (there's urllib that almost no one uses), I meant that Python is older than HTTP and HTTP evolved a lot during that time.

Once you create an API, it's hard to change it. HTTP initially was very simple and evolved over time. Things like REST, JSON encoded messages, cookies, authentication (albeit rarely used), keep-alive were added incrementally. Today's HTTP is used completely different than 30 years ago. Python already had urllib, then urllib2 which was renamed back to urllib in Python 3, but its API is still behind how HTTP is used right now.

Re: Httpx: A next-generation HTTP client for Python

#73

Post by strbean »

DougBTX wrote: post #58
penagwin wrote: post #37

Earlier quoted context omitted.

> "having type annotation" should be as important as "having decent unit test coverage". I mean this is pretty spot on IMO. I've worked with many languages, and have concluded that having a powerful type system catches soooo many bugs before you even try to run the code. And they're usually "stupid" bugs too, forgetting to sanitize inputs etc. Even worse is when a language tries to be "smart", so you end up with "1"…

Type systems and static checks in general are great. What do you think of the example type in the post above? It seems like the type system might not be expressive enough to handle existing APIs, that’s something which was hard for TypeScript too, but has slowly been getting better. Perhaps that will be the case in Python too.

Some parts of that example seem pretty silly. URLs for example: he describes how just about anything can be passed in as a URL, and requests attempts to call __str__ or __unicode__ on it and then parse it. Considering (essentially?) all types in Python have __str__, this is a perfectly reasonable place to use the Any type. The issue isn't with the expressiveness of the typing system, but with the absurd flexibility of input handling by requests.

Re: Httpx: A next-generation HTTP client for Python

#75

Post by anderskaseorg »

strbean wrote: post #73
DougBTX wrote: post #58

Earlier quoted context omitted.

Type systems and static checks in general are great. What do you think of the example type in the post above? It seems like the type system might not be expressive enough to handle existing APIs, that’s something which was hard for TypeScript too, but has slowly been getting better. Perhaps that will be the case in Python too.

Some parts of that example seem pretty silly. URLs for example: he describes how just about anything can be passed in as a URL, and requests attempts to call __str__ or __unicode__ on it and then parse it. Considering (essentially?) all types in Python have __str__, this is a perfectly reasonable place to use the Any type. The issue isn't with the expressiveness of the typing system, but with the absurd flexibility o…

The right type is ‘object’, not ‘Any’.

‘object’ is the base class of all types: you can put anything into an ‘object’, and you can only do very generic operations like str() on what you get out of an ‘object’ without further checks like isinstance().

‘Any’ is an unsound escape hatch that disables type checking: you can put anything into an ‘Any’, and you can do anything with what you get out of an ‘Any’, and the type checker will make no effort to stop you from doing something wrong.

https://docs.python.org/3/library/typing.html#the-any-type

https://mypy.readthedocs.io/en/latest/dynamic_typing.html#an...

Re: Httpx: A next-generation HTTP client for Python

#76

Post by greglindahl »

neuland wrote: post #42

Looks like this does not include certifi [0] and loads system certificates by default. This is a breath of fresh air to see, because so many packages want to use their own certs and have a custom system to override it to use system certs. Edit: Well, looks like it does use certifi. But my grumble still stands, I don't understand why does everyone want to mess with your certs. [0] https://pypi.org/project/certifi/

I want to mess with certs in Python so that my web crawler can actually access the whole web. If you don't talk to a wide variety of hosts, you probably haven't noticed that it's broken for 1%.

Re: Httpx: A next-generation HTTP client for Python

#77

Post by dragonwriter »

greglindahl wrote: post #76
neuland wrote: post #42

Looks like this does not include certifi [0] and loads system certificates by default. This is a breath of fresh air to see, because so many packages want to use their own certs and have a custom system to override it to use system certs. Edit: Well, looks like it does use certifi. But my grumble still stands, I don't understand why does everyone want to mess with your certs. [0] https://pypi.org/project/certifi/

I want to mess with certs in Python so that my web crawler can actually access the whole web. If you don't talk to a wide variety of hosts, you probably haven't noticed that it's broken for 1%.

If you work in a corporate environment, you would probably notice that systems that insist on bundling their own certs without an easy to activate option of using system cert store are broken. (And even if the library has an easy to use option, if it's easy to not expose it, much software built on the library will still be broken.)

People should be empowered to substitute cert stores, but the system store should be the default.

Re: Httpx: A next-generation HTTP client for Python

#78

Post by hermitdev »

vajenetehais wrote: post #25
daze42 wrote: post #9

I've been using httpx 0.9.3 in production now for a couple months. I switched from requests when I realized I needed async support and it has been a dream to use. The only issue I've run into has been with my attempt to reuse the same AsyncClient to make multiple concurrent requests to the same remote host. It looks like this issue may have been fixed in 0.10 or 0.11 so I'll be upgrading soon to check. Also, be sure…

I'm currently using aiohttp. There is features I don't find in butterfly docs like : limit connection, limit connection per host or per time frame. Do you know if those exist in butterfly? I'm already deep in aiohttp and it's not an easy task to learn an async client (at least in my case, but I'm no dev) so if I do switch it would be for more features (retry option on exceptions, limit requests per host and per time…

I prototyped moving to aiohttp from requests/multiprocessing. The speedup was amazing. Saw something like a 60% reduction in runtime for our use case. Only reason the code hasn't gone live: we currently use requests_negotiate_sspi for authentication, which sadly isn't supported by aiohttp. Not sure if httpx supports it. Looks like it might. Planning on giving it a shot next week.

Re: Httpx: A next-generation HTTP client for Python

#79

Post by chc »

DougBTX wrote: post #58
penagwin wrote: post #37

Earlier quoted context omitted.

> "having type annotation" should be as important as "having decent unit test coverage". I mean this is pretty spot on IMO. I've worked with many languages, and have concluded that having a powerful type system catches soooo many bugs before you even try to run the code. And they're usually "stupid" bugs too, forgetting to sanitize inputs etc. Even worse is when a language tries to be "smart", so you end up with "1"…

Type systems and static checks in general are great. What do you think of the example type in the post above? It seems like the type system might not be expressive enough to handle existing APIs, that’s something which was hard for TypeScript too, but has slowly been getting better. Perhaps that will be the case in Python too.

Personally, I use type aliases in cases where I need to describe very dynamic types that make intuitive sense but are too wordy to pattern-match visually. So you might decompose that example like:

    FileSourceType = Union[basestring, file]
    FileSpecType = Union[
        # (filename, file_source)
        Tuple[basestring, Optional[FileSourceType]],
        # (filename, file_source, content_type)
        Tuple[basestring, Optional[FileSourceType], Optional[basestring]],
        # (filename, file_source, content_type, custom_headers)
        Tuple[basestring, Optional[FileSourceType], Optional[basestring], Optional[Headers]]
    ]

    ...

    files: Optional[
        Union[
            Mapping[basestring, FileSpecType], 
            Iterable[Tuple[basestring, FileSpecType]
        ]
    ]
You theoretically lose some "glance value" because now you have to look in two places for the type...but in practice, I think you can figure it out a lot easier than the original example. Obviously you don't want to do this in simple cases, but it can make pathological cases like the above a lot easier to chew on.

Re: Httpx: A next-generation HTTP client for Python

#80

Post by pensatoio »

tiangolo wrote: post #22

FastAPI creator here... if you use FastAPI, HTTPX would probably be the best match for sending requests, just saying... :D

This framework looks incredibly simple and powerful. I’ve coded a bunch of these features manually in the past. I think FastAPI is going to earn a spot in my next project!
Post reply on HN