Live data from Hacker News

The Sisyphean Task of DNS Client Config on Linux

tailscale.com

71–80 of 86 posts

Re: The Sisyphean Task of DNS Client Config on Linux

#71
post #12
post #8

I recommend the series "Anatomy of a Linux DNS lookup"[0]. Everything you never wanted to know. https://zwischenzugs.com/2018/06/08/anatomy-of-a-linux-dns-l...

That appears to have been written prior to systemd-resolved becoming enabled by default on many distros.

Which systemd is still unable to do DNS split horizon with DNSSEC.

- ISC Bind9 is still king.

Re: The Sisyphean Task of DNS Client Config on Linux

#72
post #29
post #13

Tailscale is awesome, you should use it for everything. This Linux DNS stuff drives us batty at Fly.io. We run user containers as Firecracker VMs; users belong to "organizations", and organizations share a private IPv6 network. We do DNS for that private network under the fake "internal" TLD, so if you have an app "phoenix-frontend" and another app "rabbitmq-cluster", they can see each other at "phoenix-frontend.inte…

... I realize I'm a monster, but aren't y'all already extensively using BPF for things? Sticking in a BPF egress filter on the VM that rewrites outbound DNS packets with .internal in the question to point at your DNS server seems like it would be lighter weight than just handling all queries recursively.

The main problem with this is that you need to handle TCP connections, in which the queries do not have to match up to packet boundaries, and the client can multiplex internal and external queries on the same connection.

The interface for DNS from an arbitrary container is the DNS protocol, and since you can’t control the software that is making the queries on the client side of the protocol, you have to do the special logic on the server side.

Re: The Sisyphean Task of DNS Client Config on Linux

#74
post #13

Tailscale is awesome, you should use it for everything. This Linux DNS stuff drives us batty at Fly.io. We run user containers as Firecracker VMs; users belong to "organizations", and organizations share a private IPv6 network. We do DNS for that private network under the fake "internal" TLD, so if you have an app "phoenix-frontend" and another app "rabbitmq-cluster", they can see each other at "phoenix-frontend.inte…

Funnily enough, that is the _exact_ same problem I'm facing right now, down to Firecracker and a custom internal TLD. I'm excited to see the solutions. I think the only difference is that I need to run this DNS service on the same host as my VMs, so I will need to use a different port than systemd-resolve.

So apparently systemd-resolve does support redirecting custom TLDs to a specific nameserver [0], as mentioned by dnr. However, support for custom ports was only added in version 246 and my Ubuntu 18.04 ships with 237.

As far as I can tell, my way forward is to use v246, then set my TLD to my custom nameserver with my custom port and add a stub listener on localhost (instead of 127.0.0.53, which is the default). Then I can offer my VMs to use their tap gateway (which is the Ubuntu host) as a DNS server if they want, and that will resolve my custom TLD and forward all other stuff to its own nameserver. Or, if the user wants, they can also do their own stuff.

It seems that a lot has happened with systemd-resolve in the last few months.

Re: The Sisyphean Task of DNS Client Config on Linux

#75
post #72
post #29

Earlier quoted context omitted.

... I realize I'm a monster, but aren't y'all already extensively using BPF for things? Sticking in a BPF egress filter on the VM that rewrites outbound DNS packets with .internal in the question to point at your DNS server seems like it would be lighter weight than just handling all queries recursively.

The main problem with this is that you need to handle TCP connections, in which the queries do not have to match up to packet boundaries, and the client can multiplex internal and external queries on the same connection. The interface for DNS from an arbitrary container is the DNS protocol, and since you can’t control the software that is making the queries on the client side of the protocol, you have to do the speci…

Do you, realistically? The standard behavior for most things afaik on looking up a domain name is to send out an A query and (simultaneously ideally) A queries with searchdomains appended. That's not gonna get anywhere near 512 bytes for a fly .internal service name, I'd expect.

Re: The Sisyphean Task of DNS Client Config on Linux

#76
post #26

Earlier quoted context omitted.

On macOS, you could do this trivially by creating /etc/resolver/internal with the name servers in it. Linux should copy that.

Systemd-resolved does much more than MacOS resolver. It can make the additional resolvers active conditional on whether the link is up, for example (think VPN). MacOS resolver can't do that.

I thought tptacek was looking for something simpler than systemd-resolved. I was just pointing out the simplicity of /etc/resolver/ on macOS that's been available since, I think, the first version of macOS. It's a nice feature.

In any case, on macOS, you use scutil to adjust resolvers when you change networking state as when bringing up a VPN.

Re: The Sisyphean Task of DNS Client Config on Linux

#77
post #64

Earlier quoted context omitted.

I guess you've heard of dnsmasq which I'm pretty sure can do all you want. Perhaps it doesn't count as bare. [EDIT: not only do many other mention this, but the article does too! my bad]

Yes, we built our own DNS server in Rust, we're aware of dnsmasq. The problem we'd like to resolve is not having to have a SPOF DNS server we're responsible for in between our users VMs and the Internet. The prospect of running dnsmasq on every VM we launch is to aversive to consider. At that point, we'd just put a DNS recursor in our init.

Multiple redundant servers? I'm way out of my depth here, but if a spof is your main problem, perhaps you just need some redundancy?

Re: The Sisyphean Task of DNS Client Config on Linux

#78

Wait till you see what happens when Zscaler gets added into the mix for corporate laptops. I now have to scenario for when Zscaler intercepts/rewrites DNS requests along with when the Cisco VPN is active. Given that a large amount of folks are still WFH, it’s enough to make me want to tear the remainder of my hair out.

Can't agree more. They have 'clever' way doing a lot of things, which make us suffering to troubleshooting

Re: The Sisyphean Task of DNS Client Config on Linux

#79
post #75
post #72

Earlier quoted context omitted.

The main problem with this is that you need to handle TCP connections, in which the queries do not have to match up to packet boundaries, and the client can multiplex internal and external queries on the same connection. The interface for DNS from an arbitrary container is the DNS protocol, and since you can’t control the software that is making the queries on the client side of the protocol, you have to do the speci…

Do you, realistically? The standard behavior for most things afaik on looking up a domain name is to send out an A query and (simultaneously ideally) A queries with searchdomains appended. That's not gonna get anywhere near 512 bytes for a fly .internal service name, I'd expect.

The requirement is to support arbitrary containers, so it should still work if the container contains stub resolver software that pipelines requests over TCP.

Re: The Sisyphean Task of DNS Client Config on Linux

#80
post #76

Earlier quoted context omitted.

Systemd-resolved does much more than MacOS resolver. It can make the additional resolvers active conditional on whether the link is up, for example (think VPN). MacOS resolver can't do that.

I thought tptacek was looking for something simpler than systemd-resolved. I was just pointing out the simplicity of /etc/resolver/ on macOS that's been available since, I think, the first version of macOS. It's a nice feature. In any case, on macOS, you use scutil to adjust resolvers when you change networking state as when bringing up a VPN.

Conceptually they are similar; one has config files in /etc/resolver/*, other in /etc/systemd/resolved.conf.d/*.conf; one has mDNSResponder process (yes, it is also unicast DNS resolver and cache, since 10.6, despite the name) and the other systemd-resolved. With systemd-resolved, you can achieve exactly the same thing as with MacOS resolver, for approximately the same cost.

With systemd-resolved, you don't have to adjust resolvers manually as links come up and down, NetworkManager's VPN connections play nice with it, while DNS resolver handling in Mac VPN software (including Wireguard and Tunnelblick) is a nuasance :(.

Post reply on HN