Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

The Sisyphean Task of DNS Client Config on Linux

tailscale.com

Post reply on HN
Print view View original

81–86 of 86 posts

Re: The Sisyphean Task of DNS Client Config on Linux

#81

Post by tptacek »

cookiengineer wrote: post #66
tptacek wrote: post #13

Tailscale is awesome, you should use it for everything. This Linux DNS stuff drives us batty at Fly.io. We run user containers as Firecracker VMs; users belong to "organizations", and organizations share a private IPv6 network. We do DNS for that private network under the fake "internal" TLD, so if you have an app "phoenix-frontend" and another app "rabbitmq-cluster", they can see each other at "phoenix-frontend.inte…

But shouldn't this be solved with Multicast DNS [1] and DNS based service discovery [2]? Sounds to me that your .internal TLD is causing a lot of headache and sticking to avahi and .local would be easier? [1] https://tools.ietf.org/html/rfc6762 [2] http://dns-sd.org

What we deploy has to work with any reasonable application; we build to the lowest common denominator interface.

People on this thread have a lot of ideas about what they'd do if they got to be the sysadmin for every container running on our fleet.

Re: The Sisyphean Task of DNS Client Config on Linux

#82

Post by tptacek »

throwaway823882 wrote: post #54
tptacek wrote: post #13

Tailscale is awesome, you should use it for everything. This Linux DNS stuff drives us batty at Fly.io. We run user containers as Firecracker VMs; users belong to "organizations", and organizations share a private IPv6 network. We do DNS for that private network under the fake "internal" TLD, so if you have an app "phoenix-frontend" and another app "rabbitmq-cluster", they can see each other at "phoenix-frontend.inte…

You're trying to do something DNS wasn't built for, so, there is no good solution for what you want ("only serve .internal and nothing else"). As I see it, you have 3 options: 1. Use ".internal.someotherdomain.io" and the regular old public DNS. (this probably only does half of what you want) 2. Use dnsmasq at the edge of each .internal subnet, and maybe require the customer to configure your DNS in their container.…

(2) is what we do, modulo that we don't run dnsmasq. (We slip our DNS configuration in via our init).

I'm not persuaded by appeals to what DNS was or wasn't built for (unfortunately neither is the IETF; to wit: DNSSEC). I agree with you that the ask here is big! But I'm not convinced there isn't a cleaner solution that works within our constraints.

Re: The Sisyphean Task of DNS Client Config on Linux

#83

Post by indigodaddy »

tptacek wrote: post #70
lathiat wrote: post #69

Earlier quoted context omitted.

This is called "Split Horizon DNS". The problem is, there is no standard way to communicate this information to a machine. e.g., afaik there is no DHCP extension to say "for .internal, please see this server, for the rest, see whatever". So you pretty much have to run a server in the middle. It -is- possible, as others have commented, to do this within the VM or container but they need a way to get that information.…

We do and we don't control the user environment. We can run anything we want in a container, and we can tamper with the runtime environment of a container entrypoint any way we want. But what we come up with has to be compatible with any container someone throws at us. Additionally: the basic problem of routing DNS requests based on domains is trivial. It's probably less than 100 lines of Rust to send `.internal` to…

Bad compared to what? Is there any OS that has a better DNS resolution process/methodology? Windows? BSD? Solaris? Serious question. I mean do they all suck or?

Re: The Sisyphean Task of DNS Client Config on Linux

#84

Post by 4rtyui9xd »

4rtyui9xd wrote: post #40
tptacek wrote: post #13

Tailscale is awesome, you should use it for everything. This Linux DNS stuff drives us batty at Fly.io. We run user containers as Firecracker VMs; users belong to "organizations", and organizations share a private IPv6 network. We do DNS for that private network under the fake "internal" TLD, so if you have an app "phoenix-frontend" and another app "rabbitmq-cluster", they can see each other at "phoenix-frontend.inte…

I can think of a variety of ways. Some involve running some code on the VM. For example, dnscache or dqcache would work. This can also be complished using tinydns. Tiny programs that use very little memory. But it sounds like you are trying to avoid asking the user to install anything other than flyctl on the "bare Linux VM". What is not clear is what programs are installed by default in the "bare Linux VM". The othe…

If it is flyctl doing the resolution then another idea is to have flyctl do the resolution in a chroot with its own resolv.conf.

Re: The Sisyphean Task of DNS Client Config on Linux

#85

Post by lmz »

BeefWellington wrote: post #60
tptacek wrote: post #13

Tailscale is awesome, you should use it for everything. This Linux DNS stuff drives us batty at Fly.io. We run user containers as Firecracker VMs; users belong to "organizations", and organizations share a private IPv6 network. We do DNS for that private network under the fake "internal" TLD, so if you have an app "phoenix-frontend" and another app "rabbitmq-cluster", they can see each other at "phoenix-frontend.inte…

> If there's a clean way to resolve this, so that the VM itself can just send `.internal` queries to us, and everything else to `1.1.1.1` or `8.8.8.8` or whatever the customer's container had, I would _love_ to hear it. I've come pretty close to breaking out preload in anger over this problem. If you're at that point, rather than preload, why not simply add the code to glibc and contribute it back? Has there been som…

This just adds more complexity to a universally used user space library. And also breaks when some programs don't use the same resolver. Just implement your own dns server that does it for you (or reuse dnsmasq).

Re: The Sisyphean Task of DNS Client Config on Linux

#86

Post by throwaway823882 »

tptacek wrote: post #82
throwaway823882 wrote: post #54

Earlier quoted context omitted.

You're trying to do something DNS wasn't built for, so, there is no good solution for what you want ("only serve .internal and nothing else"). As I see it, you have 3 options: 1. Use ".internal.someotherdomain.io" and the regular old public DNS. (this probably only does half of what you want) 2. Use dnsmasq at the edge of each .internal subnet, and maybe require the customer to configure your DNS in their container.…

(2) is what we do, modulo that we don't run dnsmasq. (We slip our DNS configuration in via our init). I'm not persuaded by appeals to what DNS was or wasn't built for (unfortunately neither is the IETF; to wit: DNSSEC). I agree with you that the ask here is big! But I'm not convinced there isn't a cleaner solution that works within our constraints.

Then another option is to implement all 3. This gives you what you want most of the time, but also supports all customer use cases. The downside is the customer now needs a flow chart to understand how their DNS is resolving.

To start with, first implement (1).

Assuming .internal points to internal-only IPs, and you are reusing internal IPs (each customer VPC can have arbitrary private IPs), you can have ni-172-16-0-5.customer.internal.somedomain.io return A record to "172.16.0.5". (You don't have to add .customer, but you might gain some useful analytics)

A customer needing a hostname for 172.16.0.5 can resolve ni-172-16-0-5.customer.internal.somedomain.io. By default it'll use whatever DNS server/resolver they have configured (or yours with option (2)). The request goes through public DNS until it gets back to somedomain.io (your domain) and you return the result.

Upside: You're not hosting all the customer's DNS resolution. Downside: The customer will see a longer round-trip for .internal requests, and their VPC needs a NAT gateway.

Next implement (3).

Intercept all DNS traffic that matches ".internal.somedomain.io" and spoof responses. Write this tight enough that it will only return responses when you are sure they won't break clients, and pass the rest to the public DNS using option (1), which prevents from having to mess with TCP connections (they'll be slower though the public internet, but it's TCP, what did they expect?).

Upside: You are only returning results for .internal, everything else can use somebody else's DNS, and the customer doesn't need a NAT gateway. Downside: Requires custom engineering which is subtly complex, and is hidden magic the customer doesn't expect, so could cause troubleshooting headaches.

Next, implement (2) as an opt-in.

Have your VPC DHCP server return your internal dns server. You resolve .internal locally and handle the rest as a big hulking resolver. You can also slip this DNS server into their containers, but as a customer I'd want a giant red flag in the docs that advertises this.

Upside: Works in every customer scenario. Bonus: now you can return reverse records. Downside: Now you really are hosting the customer's DNS resolver.

Post reply on HN