Live data from Hacker News

The Sisyphean Task of DNS Client Config on Linux

tailscale.com

61–70 of 86 posts

Re: The Sisyphean Task of DNS Client Config on Linux

#61

I’m confused, this article seems to assert that systemd-resolved can handle split DNS based on the domain name and even claims the docs for it are fantastic. But last time I went down this rabbit hole I finally gave up and combined it with dnsmasq. The docs are verbose but useless, and I ultimately determined it was impossible. systemd-resolved would failover to alternate DNS servers but you’d end up making a ton of…

It depends on how long ago you checked, but it can do it now without dnsmasq.

https://gist.github.com/brasey/fa2277a6d7242cdf4e4b7c720d42b...

Re: The Sisyphean Task of DNS Client Config on Linux

#63
post #13

Tailscale is awesome, you should use it for everything. This Linux DNS stuff drives us batty at Fly.io. We run user containers as Firecracker VMs; users belong to "organizations", and organizations share a private IPv6 network. We do DNS for that private network under the fake "internal" TLD, so if you have an app "phoenix-frontend" and another app "rabbitmq-cluster", they can see each other at "phoenix-frontend.inte…

I guess you've heard of dnsmasq which I'm pretty sure can do all you want.

Perhaps it doesn't count as bare.

[EDIT: not only do many other mention this, but the article does too! my bad]

Re: The Sisyphean Task of DNS Client Config on Linux

#64
post #13

Tailscale is awesome, you should use it for everything. This Linux DNS stuff drives us batty at Fly.io. We run user containers as Firecracker VMs; users belong to "organizations", and organizations share a private IPv6 network. We do DNS for that private network under the fake "internal" TLD, so if you have an app "phoenix-frontend" and another app "rabbitmq-cluster", they can see each other at "phoenix-frontend.inte…

I guess you've heard of dnsmasq which I'm pretty sure can do all you want. Perhaps it doesn't count as bare. [EDIT: not only do many other mention this, but the article does too! my bad]

Yes, we built our own DNS server in Rust, we're aware of dnsmasq. The problem we'd like to resolve is not having to have a SPOF DNS server we're responsible for in between our users VMs and the Internet.

The prospect of running dnsmasq on every VM we launch is to aversive to consider. At that point, we'd just put a DNS recursor in our init.

Re: The Sisyphean Task of DNS Client Config on Linux

#65
post #29
post #13

Tailscale is awesome, you should use it for everything. This Linux DNS stuff drives us batty at Fly.io. We run user containers as Firecracker VMs; users belong to "organizations", and organizations share a private IPv6 network. We do DNS for that private network under the fake "internal" TLD, so if you have an app "phoenix-frontend" and another app "rabbitmq-cluster", they can see each other at "phoenix-frontend.inte…

... I realize I'm a monster, but aren't y'all already extensively using BPF for things? Sticking in a BPF egress filter on the VM that rewrites outbound DNS packets with .internal in the question to point at your DNS server seems like it would be lighter weight than just handling all queries recursively.

This is good praxis. I have received some thoughts about how to do this without parsing DNS in eBPF (which would be a disaster).

Re: The Sisyphean Task of DNS Client Config on Linux

#66
post #13

Tailscale is awesome, you should use it for everything. This Linux DNS stuff drives us batty at Fly.io. We run user containers as Firecracker VMs; users belong to "organizations", and organizations share a private IPv6 network. We do DNS for that private network under the fake "internal" TLD, so if you have an app "phoenix-frontend" and another app "rabbitmq-cluster", they can see each other at "phoenix-frontend.inte…

But shouldn't this be solved with Multicast DNS [1] and DNS based service discovery [2]?

Sounds to me that your .internal TLD is causing a lot of headache and sticking to avahi and .local would be easier?

[1] https://tools.ietf.org/html/rfc6762

[2] http://dns-sd.org

Re: The Sisyphean Task of DNS Client Config on Linux

#67
Wait till you see what happens when Zscaler gets added into the mix for corporate laptops.

I now have to scenario for when Zscaler intercepts/rewrites DNS requests along with when the Cisco VPN is active. Given that a large amount of folks are still WFH, it’s enough to make me want to tear the remainder of my hair out.

Re: The Sisyphean Task of DNS Client Config on Linux

#68
post #13

Tailscale is awesome, you should use it for everything. This Linux DNS stuff drives us batty at Fly.io. We run user containers as Firecracker VMs; users belong to "organizations", and organizations share a private IPv6 network. We do DNS for that private network under the fake "internal" TLD, so if you have an app "phoenix-frontend" and another app "rabbitmq-cluster", they can see each other at "phoenix-frontend.inte…

How about generating a custom resolv.conf per instance, protecting it from systemd via `chattr +i`, then run any other services you need to inside?

Re: The Sisyphean Task of DNS Client Config on Linux

#69
post #13

Tailscale is awesome, you should use it for everything. This Linux DNS stuff drives us batty at Fly.io. We run user containers as Firecracker VMs; users belong to "organizations", and organizations share a private IPv6 network. We do DNS for that private network under the fake "internal" TLD, so if you have an app "phoenix-frontend" and another app "rabbitmq-cluster", they can see each other at "phoenix-frontend.inte…

This is called "Split Horizon DNS". The problem is, there is no standard way to communicate this information to a machine. e.g., afaik there is no DHCP extension to say "for .internal, please see this server, for the rest, see whatever". So you pretty much have to run a server in the middle.

It -is- possible, as others have commented, to do this within the VM or container but they need a way to get that information. systemd-resolved does this. From what I understand of fly.io, the container is setup by the user, so you're not going to have control over their networking configuration or environment - is that right?

Now there is an alternative, which is instead of using .internal, to use a public facing internet hosting that is delegated to your internal DNS servers. That way they use their normal DNS, and it resolves to your nameserver. But the DNS needs to be public, and critically, if the internet DNS goes down, your .internal resolution will also stop working. Which is highly likely to cause problems for peoples applications. And also adds un-predictable latency to their local service connections which is almost certainly going to cause problems - it's not a good idea.

Having said all of that, generally speaking, it makes sense that -you- provide the DNS. It's pretty much the standard way that your 'ISP', 'Hosting Provider', etc, provides the DNS rather than relying on the public internet DNS. Only a few years ago there was no such public DNS.

The other possible alternative would be to leverage multicast DNS (libnss-mdns/avahi-daemon) or LLMNR but I'm not sure those are really ideal for this use case either. And in both cases requires resolver support that you dont get from the basic glibc resolver. So for your use case, may still not be workable.

So I think you're stuck, unless you can mandate that all of your VM images use systemd-resolved and somehow communicate the config into them.

Re: The Sisyphean Task of DNS Client Config on Linux

#70
post #69
post #13

Tailscale is awesome, you should use it for everything. This Linux DNS stuff drives us batty at Fly.io. We run user containers as Firecracker VMs; users belong to "organizations", and organizations share a private IPv6 network. We do DNS for that private network under the fake "internal" TLD, so if you have an app "phoenix-frontend" and another app "rabbitmq-cluster", they can see each other at "phoenix-frontend.inte…

This is called "Split Horizon DNS". The problem is, there is no standard way to communicate this information to a machine. e.g., afaik there is no DHCP extension to say "for .internal, please see this server, for the rest, see whatever". So you pretty much have to run a server in the middle. It -is- possible, as others have commented, to do this within the VM or container but they need a way to get that information.…

We do and we don't control the user environment. We can run anything we want in a container, and we can tamper with the runtime environment of a container entrypoint any way we want. But what we come up with has to be compatible with any container someone throws at us.

Additionally: the basic problem of routing DNS requests based on domains is trivial. It's probably less than 100 lines of Rust to send `.internal` to our servers and anything else to a public recursor. But the goal is to get our own code out of the path of people's DNS lookups, even if it's just 100 lines of Rust.

So running dnsmasq or some systemd resolver server thingy doesn't really solve anything for us. Before we did that, we'd just make our `init` a DNS server. But it'd be neat to come up with a way to to solve the problem without having a SPOF server dependency. It might could be possible! You can already see that if you assume a specific libc, you can just preload a libc stub resolver that routes requests. But, of course, we can't assume a particular libc.

Most of my point here is just that "standard" Linux DNS is bad.

Post reply on HN