Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Show HN: Subtrace – Wireshark for Docker Containers

github.com

Post reply on HN
Print view View original

21–30 of 75 posts

Re: Show HN: Subtrace – Wireshark for Docker Containers

#21

Post by Arnavion »

adtac wrote: post #5
smw wrote: post #3

Can it decrypt tls? Perhaps by hooking the calls to common libraries?

Yes, but we've managed to do it automatically without any library/language specific hooks! It's probably one of my favourite things in Subtrace :) We generate an ephemeral TLS root CA certificate and inject it into the system store. The generated certificate is entirely in-memory and never leaves the machine. To make this work without root privileges, we intercept the open(2) syscall to see if it's /etc/ssl/certs/ca-…

This will not work with HPKP but hopefully nothing is using that any more. ( https://en.m.wikipedia.org/wiki/HTTP_Public_Key_Pinning )

It won't work with programs that defensively validate the cert chain but those are rare.

It won't work with programs that embed their own root cert store, which is also rare but I would guess less rare than the previous one. The usual reason to do this is to minimize OS deps, and in the case of Docker containers to save on container image size by only including the roots you care about.

But yes for the vast majority of programs it should work fine.

Re: Show HN: Subtrace – Wireshark for Docker Containers

#23

Post by adtac »

Arnavion wrote: post #21
adtac wrote: post #5

Earlier quoted context omitted.

Yes, but we've managed to do it automatically without any library/language specific hooks! It's probably one of my favourite things in Subtrace :) We generate an ephemeral TLS root CA certificate and inject it into the system store. The generated certificate is entirely in-memory and never leaves the machine. To make this work without root privileges, we intercept the open(2) syscall to see if it's /etc/ssl/certs/ca-…

This will not work with HPKP but hopefully nothing is using that any more. ( https://en.m.wikipedia.org/wiki/HTTP_Public_Key_Pinning ) It won't work with programs that defensively validate the cert chain but those are rare. It won't work with programs that embed their own root cert store, which is also rare but I would guess less rare than the previous one. The usual reason to do this is to minimize OS deps, and in t…

Yep, certificate pinning is the one scenario Subtrace can't handle in my experience, but thankfully, it's fairly rare like you said. And IMO there is no general solution to the problem [1], but it's one of those very interesting problems to daydream thinking about when you're stuck in traffic or whatever :)

We still try our best by handling as much of the long tail of environments with some library/framework specific workaround (e.g. Deno bundles all TLS certs in its binary so we set the DENO_CERT env var when applicable).

[1] https://news.ycombinator.com/item?id=42923998

Re: Show HN: Subtrace – Wireshark for Docker Containers

#24

Post by adtac »

polithrow22 wrote: post #22

anything similar for k8s?

Subtrace already works great on Kubernetes (https://docs.subtrace.dev/kubernetes)! Add a single line to your image's Dockerfile and that's it.

I'm working on an even simpler way where you can just `kubectl apply` a DaemonSet or a Helm chart to get automatic tracing for all pods in your cluster instantly without any code-level changes. If anyone is interested in beta testing this, email me at adtac@subtrace.dev, I'd love to understand your usecase!

Re: Show HN: Subtrace – Wireshark for Docker Containers

#25

Post by sophacles »

adtac wrote: post #20
jgauth wrote: post #18

Looks like it is for http requests only? If so, wireshark is not an apt comparison.

For now, yes :) Since we operate at the TCP level, we can actually handle pretty much any protocol. I have an implementation of a postgres handler in my git stash that intercepts and shows the SQL queries executed + the resulting rows alongside the HTTP request that triggered it (I still need to do some robustness and correctness testing before it's ready to merge). With a handful of other protocols like MySQL, Mongo…

It's a pretty cool looking product. It's not wireshark, it's not close to wireshark just because it can capture some tcp pcaps, and there are more protocols relevant to container networking than a handful of TCP app-level protocols.

When I came in I was hoping to see a product that actually was for container networking, not just app data flows. Again - this is a neat tool, and probably incredibly useful for people developing way up the stack like that, but a lot of us live below the bottom of a "full-stack developer's" stack. Some features I would expect in a "wireshark for Docker containers":

* Ability to inspect DNS traffic

* Ability to trace packets the enter the conainter network stack (e.g. the packet(s) generated when the server calls send() ) into the virtual interface for the namespace and through the host machine. Ideally with multiple observation points and correlation of packets in the overlay/underlay contexts (decrypting from local keys whenever possible).

* Ability to see where a packet dies between the app and exit NIC on the host. Including things like "packets delivered to this container even though the dest IP/subnet isn't in this container"

* Similar to the previous point: ability to track packets through all the NAT steps containers introduce.

* See arp traffic on virtual interfaces.

* Ability to observe the TLS handshake and gather all the parameters of the connection.

* Packet dissection and protocol session tracing for all the tunnels.

* Bonus points if you can capture weird teleports caused by ebpf programs in the network path.

I expect this because that's how I use wireshark_+ bpftrace in container environments for the most part. I've also used it to debug while implementing protocols, but that's a less common use case of packet dissection and tracing in wireshark.

What you've built is cool, and I can see it expanding in a lot of directions very, very, usefully. I just really dislike something calling itself a wireshark while not really helping with networking (and in fact - the networking has to work reasonably well for this tool to be effective).

Re: Show HN: Subtrace – Wireshark for Docker Containers

#26

Post by TachyonicBytes »

adtac wrote: post #5
smw wrote: post #3

Can it decrypt tls? Perhaps by hooking the calls to common libraries?

Yes, but we've managed to do it automatically without any library/language specific hooks! It's probably one of my favourite things in Subtrace :) We generate an ephemeral TLS root CA certificate and inject it into the system store. The generated certificate is entirely in-memory and never leaves the machine. To make this work without root privileges, we intercept the open(2) syscall to see if it's /etc/ssl/certs/ca-…

Is this a different method from the httptap [1] that was on hackernews a few weeks ago? Somebody in that post seemed to say that it also generates CA certificates on the fly.

[1] https://news.ycombinator.com/item?id=42919909

Re: Show HN: Subtrace – Wireshark for Docker Containers

#27

Post by robinhoodexe »

adtac wrote: post #24
polithrow22 wrote: post #22

anything similar for k8s?

Subtrace already works great on Kubernetes ( https://docs.subtrace.dev/kubernetes )! Add a single line to your image's Dockerfile and that's it. I'm working on an even simpler way where you can just `kubectl apply` a DaemonSet or a Helm chart to get automatic tracing for all pods in your cluster instantly without any code-level changes. If anyone is interested in beta testing this, email me at adtac@subtrace.dev, I'd…

It’d be neat to use subtrace in an ephemeral pod for debugging purposes, that just runs alongside the regular pod.

For monitoring the network traffic for the whole cluster, the CNI and/or whatever ebpf-based runtime security stuff you’re using (falco, tetragon, tracee) is usually enough, but I can definitely see the usefulness of subtract for more specific debugging purposes. If run as a DaemonSet make sure to add some pod filtering such as namespace and label selectors (but I’m sure you’ve already thought about that).

Re: Show HN: Subtrace – Wireshark for Docker Containers

#29

Post by adtac »

sophacles wrote: post #25
adtac wrote: post #20

Earlier quoted context omitted.

For now, yes :) Since we operate at the TCP level, we can actually handle pretty much any protocol. I have an implementation of a postgres handler in my git stash that intercepts and shows the SQL queries executed + the resulting rows alongside the HTTP request that triggered it (I still need to do some robustness and correctness testing before it's ready to merge). With a handful of other protocols like MySQL, Mongo…

It's a pretty cool looking product. It's not wireshark, it's not close to wireshark just because it can capture some tcp pcaps, and there are more protocols relevant to container networking than a handful of TCP app-level protocols. When I came in I was hoping to see a product that actually was for container networking, not just app data flows. Again - this is a neat tool, and probably incredibly useful for people de…

That's totally fair, I can see why Wireshark wouldn't be the most accurate description to someone working on those kinds of problems. And fwiw, I wish my problems (before Subtrace) were cool enough to need to whip out Wireshark for packet-level inspection :)

Re: Show HN: Subtrace – Wireshark for Docker Containers

#30

Post by adtac »

TachyonicBytes wrote: post #26
adtac wrote: post #5

Earlier quoted context omitted.

Yes, but we've managed to do it automatically without any library/language specific hooks! It's probably one of my favourite things in Subtrace :) We generate an ephemeral TLS root CA certificate and inject it into the system store. The generated certificate is entirely in-memory and never leaves the machine. To make this work without root privileges, we intercept the open(2) syscall to see if it's /etc/ssl/certs/ca-…

Is this a different method from the httptap [1] that was on hackernews a few weeks ago? Somebody in that post seemed to say that it also generates CA certificates on the fly. [1] https://news.ycombinator.com/item?id=42919909

httptap is really cool! Their technique is different (they do a filesystem mount instead of intercepting syscalls like Subtrace does) but both tools effectively reach the same goal using different routes.
Post reply on HN