Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Httptap: View HTTP/HTTPS requests made by any Linux program

github.com

Post reply on HN
Print view View original

71–80 of 148 posts

Re: Httptap: View HTTP/HTTPS requests made by any Linux program

#71

Post by adtac »

alexflint wrote: post #29
adtac wrote: post #20

Using a TUN device for this is a really cool idea! And the "How it was made" section is one of the best things I've read in a Github README. I'm building something called Subtrace [1] but it can intercept both incoming and outgoing requests automatically. Looks like we converged on the same interface for starting the program too lol [2]. Subtrace's purpose is kinda different from httptap's though (more observability…

Super cool! Connecting what you capture to Chrome DevTools is fascinating, as is using eBPF. Great work getting the devtools to run as a standalone web app. You won't believe it but I have a half-finished attempt of the same thing for the firefox network tab - in the "networktab" dir of the repo! Very cool project, would love to learn more and happy to chat more about it.

Thanks! Subtrace uses BPF, not eBPF :) I think eBPF could be made to work with the same approach, but there's a few differences:

- eBPF requires root privileges or at least CAP_BPF. Subtrace uses seccomp_unotify [1], so it works even in unprivileged environments.

- eBPF requires using eBPF maps as the data channel + weird restrictions in the code because of the eBPF verifier. IMO these two things make it way harder to work with for the kind of networking logic that both httptap and Subtrace have in userspace. Everything is perfectly possible, just harder to reason about and debug.

>half-finished attempt of the same thing for the firefox network tab

Hahahah this is incredible. Something something great minds.

[1] https://man.archlinux.org/man/seccomp_unotify.2.en

Re: Httptap: View HTTP/HTTPS requests made by any Linux program

#72

Post by yoavm »

erdii wrote: post #11
yoavm wrote: post #4

The "How it was made" section of the README was not less interesting than the tool itself: > The way we have set things up is that we live and practice together on a bit over a hundred acres of land. In the mornings and evenings we chant and meditate together, and for about one week out of every month we run and participate in a meditation retreat. The rest of the time we work together on everything from caring for t…

To be honest: This sounds like just another of the many many other yoga/spiritual cults that currently exist all over the western world. EDIT: typos and slight wording changes

I believe I grew up in a cult myself, and one of the things I've concluded from that experience, and from leaving it, is that everywhere is a cult. Humans have a tendency towards cult-ish life, and if the cult is big enough we just refer to it as "society". People were as afraid (more or less) to leave the cult I was at, as people are around me now when they consider doing anything that is out of the norm.

By no mean am I trying to hint towards some conspiracy, or to say that all cults are equally bad (or good); Just to say that sometimes the word cult simply means "a less popular way of life than the one most people around me live by".

Re: Httptap: View HTTP/HTTPS requests made by any Linux program

#73

Post by _boffin_ »

alexflint wrote: post #16
gear54rus wrote: post #7

Earlier quoted context omitted.

can it modify requests or responses? with the current web getting increasingly user-hostile a need for tool like this was never more apparent especially if it doesn't require proxy configuration

Agreed! So there isn't any interface for modifying requests/responses at present, but it's definitely possible given the underlying approach. If you consider [this line of code]( https://github.com/monasticacademy/httptap/blob/main/http.go... ) where you have an HTTP request parsed from the that ran and are about to send it out to the public internet: you could modify the request (or the response that is received a f…

Injecting random data into telemetry requests to mess up someone’s pretty dashboard?

Re: Httptap: View HTTP/HTTPS requests made by any Linux program

#74

Post by xorcist »

alexflint wrote: post #51
wzyboy wrote: post #39

It's a genius idea to run the process in a isolated network namespace! I'm more interested in the HTTPS part. I see that it sets some common environment variables [1] to instruct the program to use the CA bundle in the temporary directory. This seems to pose a similar issue like all the variants of `http_proxy`: the program may simply choose to ignore the variable. I see it also mounts an overlay fs for `/etc/resolv.…

Thanks! But yep I agree, you're exactly right, it's ultimately... frustrating that there isn't really an agreed-upon or system-enforced way to specify CA roots to an arbitrary process. It's true that httptap mounts an overlay on /etc/resolv.conf. This is, as you'd expect, due to the also-sort-of-frustrating situation with respect to DNS resolution in which, like CA roots, there isn't a truly reliable way to tell an a…

It's a bit thin solution though, isn't it? As you say, it's dependent on both specific CA store and resolver behaviour. It's probably going to be robust enough on the most common SSL libraries, such as OpenSSL. But if we're going that route, why not just run the software against a patched SSL library which dumps the traffic?

That also doesn't require any elevated privileges (as opposed to other methods of syscall interception) and is likely much easier to do. It has the added benefit of being robust against applications either pinning certificates outright or just being particular about serial numbers, client certificates, and anything like that.

Re: Httptap: View HTTP/HTTPS requests made by any Linux program

#75

Post by adtac »

ranger_danger wrote: post #23

Why not use eBPF instead? Then you could see all http requests from all processes at once, including ones that are already running. Plus you wouldn't need to bother with TLS at all, just hook on e.g. write(2).

How would hooking on write(2) solve TLS? You'll be able to read and modify the ciphertext, but the process will never call write(2) with the plaintext bytes, so you can't actually read the HTTP request. You'll just see the encrypted bytes that go on the wire, but so does the NSA :)

You need the kind of CA certificate trick that httptap uses. It comes with its own set of caveats (e.g. certificate pinning), but it can be made to work reliably in most practical scenarios.

I've spent an unjustifiable amount of time thinking about this specific problem building Subtrace [1], so I'm genuinely very interested in a simpler / more elegant approach.

[1] https://github.com/subtrace/subtrace

Re: Httptap: View HTTP/HTTPS requests made by any Linux program

#77

Post by mhils »

alexflint wrote: post #66
q2dg wrote: post #43

Mitmproxy v11.1 can do a similar thing

Yeah mitmproxy is great. The main difference with httptap is that it's an HTTP proxy server, so you have to configure your program to use a proxy server. When I wrote httptap I wanted to be able to run `httptap ` and see the httptraces right there in standard output. There is an absolute ton of cool things that mitmproxy can do that httptap is not even close to, like interactively modifying HTTP requests and such. Ve…

> so you have to configure your program to use a proxy server.

That's not true for local capture mode: https://mitmproxy.org/posts/local-capture/linux/. :)

Re: Httptap: View HTTP/HTTPS requests made by any Linux program

#79

Post by mhils »

This is really cool, thank you for sharing! We've built a similar feature for mitmproxy lately, but with different tradeoffs. Our approach does require root and we don't have automated certificate install (yet), but we don't require apps to run in a dedicated namespace (so you can capture already-running processes). Super awesome to see this now, excited to dive into the code and see how you do TCP reassembly etc. :)

Re: Httptap: View HTTP/HTTPS requests made by any Linux program

#80

Post by adtac »

wzyboy wrote: post #39

It's a genius idea to run the process in a isolated network namespace! I'm more interested in the HTTPS part. I see that it sets some common environment variables [1] to instruct the program to use the CA bundle in the temporary directory. This seems to pose a similar issue like all the variants of `http_proxy`: the program may simply choose to ignore the variable. I see it also mounts an overlay fs for `/etc/resolv.…

IMO there's no general solution to the HTTPS part that will work for all kinds of programs and the long tail of certificate pinning implementations.

As a proof by counterexample, imagine malware that uses TLS for communication and goes to great lengths to obfuscate its compiled code. It could be a program that bundles a fixed set of CA certificates into its binary and never open any files on the filesystem. It can still create valid, secure TLS connections (at least for ~10 years or so, until most root CA certificates expire). TLS is all userspace and there's no guarantee that it uses OpenSSL (or any other common library), so you can't rely on hooking into specific OpenSSL functions either. If the server uses a self-signed certificate and the client accepts it for whatever reason, it's worse.

With that said, it's definitely possible to handle 99% of the cases reliably with some work. That's better than nothing.

Post reply on HN