Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

The Sisyphean Task of DNS Client Config on Linux

tailscale.com

Post reply on HN
Print view View original

31–40 of 86 posts

Re: The Sisyphean Task of DNS Client Config on Linux

#31

Post by YarickR2 »

ptomato wrote: post #29
tptacek wrote: post #13

Tailscale is awesome, you should use it for everything. This Linux DNS stuff drives us batty at Fly.io. We run user containers as Firecracker VMs; users belong to "organizations", and organizations share a private IPv6 network. We do DNS for that private network under the fake "internal" TLD, so if you have an app "phoenix-frontend" and another app "rabbitmq-cluster", they can see each other at "phoenix-frontend.inte…

... I realize I'm a monster, but aren't y'all already extensively using BPF for things? Sticking in a BPF egress filter on the VM that rewrites outbound DNS packets with .internal in the question to point at your DNS server seems like it would be lighter weight than just handling all queries recursively.

Why do you need kernel-mode parser for that ? iptables -t nat -I POSTROUTING -p udp --dport 53 -j DNAT --to

Re: The Sisyphean Task of DNS Client Config on Linux

#32

Post by ptomato »

YarickR2 wrote: post #31
ptomato wrote: post #29

Earlier quoted context omitted.

... I realize I'm a monster, but aren't y'all already extensively using BPF for things? Sticking in a BPF egress filter on the VM that rewrites outbound DNS packets with .internal in the question to point at your DNS server seems like it would be lighter weight than just handling all queries recursively.

Why do you need kernel-mode parser for that ? iptables -t nat -I POSTROUTING -p udp --dport 53 -j DNAT --to

Well, because he doesn't want to be inline for non-.internal DNS queries.

Re: The Sisyphean Task of DNS Client Config on Linux

#33

Post by mdlowman »

tptacek wrote: post #13

Tailscale is awesome, you should use it for everything. This Linux DNS stuff drives us batty at Fly.io. We run user containers as Firecracker VMs; users belong to "organizations", and organizations share a private IPv6 network. We do DNS for that private network under the fake "internal" TLD, so if you have an app "phoenix-frontend" and another app "rabbitmq-cluster", they can see each other at "phoenix-frontend.inte…

It depends on what you're using for the resolver. I'm assuming you only care about gethostbyname(3) and friends. With glibc that means nss; generally you're also looking at libnss_dns.so, which uses glibc's resolv (copied from BIND). This doesn't include enough configuration to do what you suggest; it pretty much just points everything towards a server.

So you have two options: use a different NSS module (maybe write your own?) or have a proxy DNS resolver that sends different requests to different places.

systemd-resolved actually handles the first option pretty well (although it would prefer that you use the dbus interface over gai). It can handle multiple interfaces with separate domains and split DNS fairly well! (Not so good with reverse DNS, unfortunately. But I get it, reverse DNS is pretty hacky anyways.)

If you prefer the forwarder route, dnsmasq seems to be fairly popular these days in the embedded world and elsewhere.

If I were you, I think I'd write a short NSS module or use dnsmasq, depending upon your needs.

Re: The Sisyphean Task of DNS Client Config on Linux

#34

Post by kelnos »

denysvitali wrote: post #15
jiqiren wrote: post #7

If you are just trying to use curl or some other straightforward small application - this is enough. But running Kubernetes, docker, or some other container system? This mess is much more deep... For example: there are known issues with systemd-resolved & Kubernetes (at least on Ubuntu that defaults to systemd-resolved) https://kubernetes.io/docs/tasks/administer-cluster/dns-debu...

FYI, if you don't bother opening the link, systemd-resolv sets the /etc/resolv.conf to have only 127.0.0.1 as a DNS server. You can see how things get bad when the CoreDNS pod tries to get the "upstream" DNS servers from the host /etc/resolv.conf

I guess it shouldn't be doing that? I mean, that also breaks in the case where you're running something like dnsmasq as your local resolver, which will also require 127.0.0.1 in your resolv.conf.

As much as I'm not always comfortable with the larger complexity, the CoreDNS pod really needs to be doing something like the flowchart described in the article. If systemd-resolved or NetworkManager are in play, it should be using D-Bus to talk to them to get the information it needs.

resolv.conf is an old idea that is too inflexible for today's DNS needs; a more complex solution with more complex interface is unfortunately warranted here.

Re: The Sisyphean Task of DNS Client Config on Linux

#35

Post by Koffiepoeder »

To be honest I hate all of the aforementioned programs trying to battle over /etc/resvolv.conf. That's why by default I have the file marked as immutable, and pointing to 127.0.0.1. This way I cannot have accidental DNS traffic leaking. If I need local network DNS I send a manual dhcp command to get the wifi's DNS and add it temporarily to my dnscrypt config. Similar thing for VPN DNS.

Re: The Sisyphean Task of DNS Client Config on Linux

#36

Post by dijit »

tptacek wrote: post #13

Tailscale is awesome, you should use it for everything. This Linux DNS stuff drives us batty at Fly.io. We run user containers as Firecracker VMs; users belong to "organizations", and organizations share a private IPv6 network. We do DNS for that private network under the fake "internal" TLD, so if you have an app "phoenix-frontend" and another app "rabbitmq-cluster", they can see each other at "phoenix-frontend.inte…

I used to have a very similar problem (didn’t want my company vpn to handle all dns traffic, only what was directed at the company), I managed to solve it with a local dnsmasq on my machine.

Unlikely to fit your usecase. But it’s not impossible for us plebs.

Re: The Sisyphean Task of DNS Client Config on Linux

#37

Post by dijit »

jiqiren wrote: post #7

If you are just trying to use curl or some other straightforward small application - this is enough. But running Kubernetes, docker, or some other container system? This mess is much more deep... For example: there are known issues with systemd-resolved & Kubernetes (at least on Ubuntu that defaults to systemd-resolved) https://kubernetes.io/docs/tasks/administer-cluster/dns-debu...

It also messes up docker-compose, probably in the same way.

I have to disable systemd-resolved and manually configure resolv.conf to get my containers to resolve each other properly. (Also Ubuntu)

Re: The Sisyphean Task of DNS Client Config on Linux

#38

Post by geofft »

tptacek wrote: post #13

Tailscale is awesome, you should use it for everything. This Linux DNS stuff drives us batty at Fly.io. We run user containers as Firecracker VMs; users belong to "organizations", and organizations share a private IPv6 network. We do DNS for that private network under the fake "internal" TLD, so if you have an app "phoenix-frontend" and another app "rabbitmq-cluster", they can see each other at "phoenix-frontend.inte…

If you're willing to assume glibc, a reasonably clean approach is to write your own nss_fly module that returns results for .internal.

The downside is that Go code will drop to cgo for all resolutions (because it will see something it can't handle in pure Go in resolv.conf) and non-glibc code (like Alpine containers using musl, or non-libc resolvers like ares) won't get anywhere at all.

But you're kind of doomed in that latter case, anyway, because since there currently isn't a standard for how resolv.conf should express the rules you want, the effort to get it adopted by glibc, musl, Go, Chrome, ares, and all the other various interpreters of resolv.conf will take way too long to make a practical impact on a startup's product.

Re: The Sisyphean Task of DNS Client Config on Linux

#39

Post by vetinari »

js2 wrote: post #26
tptacek wrote: post #13

Tailscale is awesome, you should use it for everything. This Linux DNS stuff drives us batty at Fly.io. We run user containers as Firecracker VMs; users belong to "organizations", and organizations share a private IPv6 network. We do DNS for that private network under the fake "internal" TLD, so if you have an app "phoenix-frontend" and another app "rabbitmq-cluster", they can see each other at "phoenix-frontend.inte…

On macOS, you could do this trivially by creating /etc/resolver/internal with the name servers in it. Linux should copy that.

Systemd-resolved does much more than MacOS resolver. It can make the additional resolvers active conditional on whether the link is up, for example (think VPN). MacOS resolver can't do that.

Re: The Sisyphean Task of DNS Client Config on Linux

#40

Post by 4rtyui9xd »

tptacek wrote: post #13

Tailscale is awesome, you should use it for everything. This Linux DNS stuff drives us batty at Fly.io. We run user containers as Firecracker VMs; users belong to "organizations", and organizations share a private IPv6 network. We do DNS for that private network under the fake "internal" TLD, so if you have an app "phoenix-frontend" and another app "rabbitmq-cluster", they can see each other at "phoenix-frontend.inte…

I can think of a variety of ways. Some involve running some code on the VM. For example, dnscache or dqcache would work. This can also be complished using tinydns. Tiny programs that use very little memory. But it sounds like you are trying to avoid asking the user to install anything other than flyctl on the "bare Linux VM". What is not clear is what programs are installed by default in the "bare Linux VM". The other question is how many ".internal" domains the VM will need to resolve. The simplest solution that comes to mind is when the user provisions an IP, flyctl writes the ".internal" domain(s) for that IP to /etc/hosts. /etc/resolv.conf can then point to whatever the user prefers. IMO, users should be running their own DNS servers, not setting /etc/resolv.conf to point to third party DNS addresses like 1.1.1.1 or 8.8.8.8 or whatever. In the case they are running their own DNS server on the VM, then it becomes trivial segregate internal from external domains. The configuration for dnscache is so easy that it could be done by the flyctl program, requiring no user interaction. (Something like tinydns-config.) One could even have configurations for a variety of DNS servers in flyctl, in case the user prefers unbound, etc.
Post reply on HN