Live data from Hacker News

Show HN: Httpdbg – A tool to trace the HTTP requests sent by your Python code

github.com

51–60 of 62 posts

Re: Show HN: Httpdbg – A tool to trace the HTTP requests sent by your Python code

#51
post #41

This looks great. I have a use case for something similar: detecting calls to the file system. Lots of code I've inherited has a habit of loading configuration from some random network share, then failing when that config got moved or the production host doesn't have the same access. I usually use strace(1) to track these down, but it's nowhere near as ergonomic as this tool. I'm wondering now if I could patch the `o…

CPython since 3.8 already has built-in audit events, including open, so you don't need to patch anything or use anything external. Just add an audit hook with sys.addaudithook().

Quick example:

    import inspect
    import pathlib
    import sys


    def callsite():
        try:
            pathlib.Path("/tmp/file").open()
        except:
            pass


    def audit_hook(event, args):
        if event == "open":
            path, mode, flags = args
            print(f"audit: open({path!r}, {mode!r}, 0o{flags:o})")
            # Not using traceback here because traceback will attempt to read the
            # source file, causing an infinite recursion of audit events.
            f = inspect.currentframe()
            while f := f.f_back:
                print(
                    f'File "{f.f_code.co_filename}", line {f.f_lineno}, in {f.f_code.co_name}'
                )


    def main():
        sys.addaudithook(audit_hook)
        callsite()


    if __name__ == "__main__":
        main()
Prints:

    audit: open('/tmp/file', 'r', 0o100000000)
    File "/path/to/python/lib/python3.12/pathlib.py", line 1013, in open
    File "/tmp/audit.py", line 10, in callsite
    File "/tmp/audit.py", line 26, in main
    File "/tmp/audit.py", line 30, in 
https://docs.python.org/3/library/audit_events.html

Re: Show HN: Httpdbg – A tool to trace the HTTP requests sent by your Python code

#52
post #46

Can recommend Opentelemetry if you need a more comprehensive tool like this. There is a whole library of so called instrumentation that can monkeypatch standard functions and produce traces of them. Traces can also propagate across process and rpc, giving you a complete picture, even in a microservice architecture.

[deleted]

Re: Show HN: Httpdbg – A tool to trace the HTTP requests sent by your Python code

#53
post #41

This looks great. I have a use case for something similar: detecting calls to the file system. Lots of code I've inherited has a habit of loading configuration from some random network share, then failing when that config got moved or the production host doesn't have the same access. I usually use strace(1) to track these down, but it's nowhere near as ergonomic as this tool. I'm wondering now if I could patch the `o…

[deleted]

Re: Show HN: Httpdbg – A tool to trace the HTTP requests sent by your Python code

#54

That's pretty cool! I was playing last night and implemented resumable downloads[0] for pip so that it could pick up where it stopped upon a network disconnect or a user interruption. It sucks when large packages, especially ML related, fail at the last second and pip has to download from scratch. This tool would have been nice to have. Thanks a bunch, - [0]: https://asciinema.org/a/1r8HmOLCfHm40nSvEZBqwm89k

It is important to check checksums (and signatures, if there are any) of downloaded packages prior to installing them; especially when resuming interrupted downloads.

Pip has a hash-checking mode, but it only works if the hashes are listed in the requirements.txt file, and they're the hashes for the target platform. Pipfile.lock supports storeing hashes for multiple platforms, but requirements.txt does not.

If the package hashes are retrieved over the same channel as the package, they can be MITM'd too.

You can store PyPi package hashes in sigstore.

There should be a way for package uploaders to sign their package before uploading. (This is what .asc signatures on PyPi were for. But if they are retrieved over the same channel, cryptographic signatures can also be MITM'd).

IMHO (1) twine should prompt to sign the package (with a DID) before uploading the package to PyPi, and (2) after uploading packages, twine should download the package(s) it has uploaded to verify the signature.

; TCP RESET and Content-Range doesn't hash resources.

Re: Show HN: Httpdbg – A tool to trace the HTTP requests sent by your Python code

#55
post #51
post #41

This looks great. I have a use case for something similar: detecting calls to the file system. Lots of code I've inherited has a habit of loading configuration from some random network share, then failing when that config got moved or the production host doesn't have the same access. I usually use strace(1) to track these down, but it's nowhere near as ergonomic as this tool. I'm wondering now if I could patch the `o…

CPython since 3.8 already has built-in audit events, including open, so you don't need to patch anything or use anything external. Just add an audit hook with sys.addaudithook(). Quick example: import inspect import pathlib import sys def callsite(): try: pathlib.Path("/tmp/file").open() except: pass def audit_hook(event, args): if event == "open": path, mode, flags = args print(f"audit: open({path!r}, {mode!r}, 0o{f…

Sounds perfect. I didn't know of this, but I think I'll start here.

Re: Show HN: Httpdbg – A tool to trace the HTTP requests sent by your Python code

#56
post #29

I've always used a proxy, like charles proxy, for this exact purpose. A neutral middle-man that gives exact timing/response data.

It's not as simple as it sounds, it requires a lot of code to capture Python traffic with charles-proxy. For example, you might modify your python code to use a Proxy and accept a self-signed Charles's certificate. If you need a 1-click solution, no dependencies, and no code's required, check out Proxyman with Auto-Setup: https://docs.proxyman.io/automatic-setup/automatic-setup Works with all popular Python libs: req…

Is that only iOS?

Re: Show HN: Httpdbg – A tool to trace the HTTP requests sent by your Python code

#57
post #46

Can recommend Opentelemetry if you need a more comprehensive tool like this. There is a whole library of so called instrumentation that can monkeypatch standard functions and produce traces of them. Traces can also propagate across process and rpc, giving you a complete picture, even in a microservice architecture.

Is there an example?

Tons; have you actually looked?

https://opentelemetry.io/docs/kubernetes/operator/automatic/ https://github.com/open-telemetry/opentelemetry-demo

Re: Show HN: Httpdbg – A tool to trace the HTTP requests sent by your Python code

#59

That's pretty cool! I was playing last night and implemented resumable downloads[0] for pip so that it could pick up where it stopped upon a network disconnect or a user interruption. It sucks when large packages, especially ML related, fail at the last second and pip has to download from scratch. This tool would have been nice to have. Thanks a bunch, - [0]: https://asciinema.org/a/1r8HmOLCfHm40nSvEZBqwm89k

It is important to check checksums (and signatures, if there are any) of downloaded packages prior to installing them; especially when resuming interrupted downloads. Pip has a hash-checking mode, but it only works if the hashes are listed in the requirements.txt file, and they're the hashes for the target platform. Pipfile.lock supports storeing hashes for multiple platforms, but requirements.txt does not. If the pa…

Thanks for the pointers. The diff is tiny and deals only with resuming downloads. i.e: everything else is left as is.
Post reply on HN