Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Httptap: View HTTP/HTTPS requests made by any Linux program

github.com

Post reply on HN
Print view View original

51–60 of 148 posts

Re: Httptap: View HTTP/HTTPS requests made by any Linux program

#51

Post by alexflint »

wzyboy wrote: post #39

It's a genius idea to run the process in a isolated network namespace! I'm more interested in the HTTPS part. I see that it sets some common environment variables [1] to instruct the program to use the CA bundle in the temporary directory. This seems to pose a similar issue like all the variants of `http_proxy`: the program may simply choose to ignore the variable. I see it also mounts an overlay fs for `/etc/resolv.…

Thanks! But yep I agree, you're exactly right, it's ultimately... frustrating that there isn't really an agreed-upon or system-enforced way to specify CA roots to an arbitrary process.

It's true that httptap mounts an overlay on /etc/resolv.conf. This is, as you'd expect, due to the also-sort-of-frustrating situation with respect to DNS resolution in which, like CA roots, there isn't a truly reliable way to tell an arbitrary process what DNS server to use, but /etc/resolv.conf is a pretty good bet. As soon as you put a process into a network namespace you have to provide it with DNS resolution because it can no longer access localhost:53, which is the systemd resolver, which is the most common setup now on desktop linux systems.

I do think it might help to mount /etc/ca-certificates as an overlay. When I started looking into the structure of that directory I was kind of dismayed... it's incredibly inconsistent from one distro to the next. Still, it's doable. Interested in any knowledge you might be able to share about how to add a cert to that directory in a way that would be picked up by at least some TLS implementations.

Re: Httptap: View HTTP/HTTPS requests made by any Linux program

#52

Post by alexflint »

ranger_danger wrote: post #27

does not seem to support SOCKS proxies which I rely on

Interesting. Care to share any info about your setup? Would it be a matter of httptap reading a certain environment variable and then forwarding traffic to a SOCKS proxy?

FWIW there is also the excellent tun2socks (https://github.com/xjasonlyu/tun2socks), which was a significant inspiration for this project, and is specifically designed to forward traffic from a TUN device to a SOCKS proxy.

Re: Httptap: View HTTP/HTTPS requests made by any Linux program

#53

Post by MisterTea »

2030ai wrote: post #46
MisterTea wrote: post #37

Earlier quoted context omitted.

Did you visit their website? https://www.monasticacademy.org/ While I cannot judge them outright, their article "Cyborgs Need a Trustworthy Religion" can appear cultist as they try to intertwine technology and religion.

The fine line between futurism and cult.

I feel that spirituality does not concern itself with the material world making it immune to societal progress. Whereas organized religion becomes mired by societal progress as it's trapped in its own zeitgeist. I believe that trying to tie the two together is a misguided attempt at creating something when there is nothing that needs to be created.

Re: Httptap: View HTTP/HTTPS requests made by any Linux program

#54

Post by jshier »

maxmcd wrote: post #41
mdaniel wrote: post #36

Earlier quoted context omitted.

Does Darwin have network namespaces like the Linux kernel does? I get the impression that's an important component of this approach

Yes, good point, maybe that is the blocker.

No network namespaces, but the various Network Extension APIs might be able to do this, though it's difficult. RocketSim (I'm unaffiliated, just an example) recently added a simulator-specific network throttle (to replace the system-wide Network Link Conditioner Apple ships) using a content filter extension. Even though this is a system-wide API, it seems you can limit its impact to a single app. And it seemed to properly compose with Proxyman and Cloudflare's Warp VPN at the same time, so perhaps it could be a general solution.

Re: Httptap: View HTTP/HTTPS requests made by any Linux program

#55

Post by alexflint »

maxmcd wrote: post #18

Amazing, such a great use of the gvisor userspace tcp/ip stack!

Yeah learning about gVisor was one of the most fun parts of this project. The gVisor devs are incredibly helpful. If you look through the gvisor-users mailing list you'll see me asking them various questions about this and they really helped out a bunch with some incredibly thorough answers.

Re: Httptap: View HTTP/HTTPS requests made by any Linux program

#56

Post by alexflint »

pcpuser wrote: post #25
gear54rus wrote: post #7

Earlier quoted context omitted.

can it modify requests or responses? with the current web getting increasingly user-hostile a need for tool like this was never more apparent especially if it doesn't require proxy configuration

You might find mitmproxy useful.

Yep, mitmproxy is fantastic IMO.

Re: Httptap: View HTTP/HTTPS requests made by any Linux program

#57

Post by alexflint »

freedomben wrote: post #35

Neat! This will immediately be used by me to debug nginx configs. Currently I use curl -v and have to manually skim the output to figure out what's wrong, but this would immediately make redirect loops and other things apparent. Cool tool!

Very cool! Would love to hear how it goes, especially any features that would be useful in the context of real-world usage.

Re: Httptap: View HTTP/HTTPS requests made by any Linux program

#58

Post by lights0123 »

wutwutwat wrote: post #50
alexflint wrote: post #2

httptap is a process-scoped http tracer that you can run without root priveleges. You can run `httptap ` where is a linux program and you get a trace of http/https requests and responses in standard output: httptap -- python -c "import requests; requests.get('https://monasticacademy.org')" ---> GET https://monasticacademy.org/ GET https://www.monasticacademy.org/ It works by running in an isolated network namespace.…

Did everyone forget about wireshark, which can totally be ran as non-root? https://blog.wireshark.org/2010/02/running-wireshark-as-you/

It certainly doesn't provide automated, process-scoped HTTPS interception.

Re: Httptap: View HTTP/HTTPS requests made by any Linux program

#59

Post by boobsbr »

wutwutwat wrote: post #50
alexflint wrote: post #2

httptap is a process-scoped http tracer that you can run without root priveleges. You can run `httptap ` where is a linux program and you get a trace of http/https requests and responses in standard output: httptap -- python -c "import requests; requests.get('https://monasticacademy.org')" ---> GET https://monasticacademy.org/ GET https://www.monasticacademy.org/ It works by running in an isolated network namespace.…

Did everyone forget about wireshark, which can totally be ran as non-root? https://blog.wireshark.org/2010/02/running-wireshark-as-you/

It's still more setup than just installing this tool.

Also, can Wireshark/libpcap decrypt SSL/TLS traffic this easily?

Re: Httptap: View HTTP/HTTPS requests made by any Linux program

#60

Post by imcritic »

zamubafoo wrote: post #38
freedomben wrote: post #34

Earlier quoted context omitted.

Inside most production environments. I could use this today inside a Pod that isn't allowed root privs.

In production environments that won't give you root access, you won't be exec'ing inside of a pod if you aren't an operator or sysadmin.

No, you are wrong. I would. The pod would be mine though.
Post reply on HN