DNS Infrastructure at GitHub

#1

Post by logicalstack » Wed, May 31, 2017, 4:47 PM UTC

DNS Infrastructure at GitHub
githubengineering.com

Re: DNS Infrastructure at GitHub

#2

Post by chrxr » Wed, May 31, 2017, 5:21 PM UTC

I'm always interested to know how long these infrastructure changes took from project initiation to launch, and whether it was a dedicated project team or something completed alongside BAU tasks.

These details are almost never included in these write-ups. Anyone have any guesses?

Re: DNS Infrastructure at GitHub

#3

Post by logicalstack » Wed, May 31, 2017, 6:01 PM UTC

chrxr wrote: post #2

I'm always interested to know how long these infrastructure changes took from project initiation to launch, and whether it was a dedicated project team or something completed alongside BAU tasks. These details are almost never included in these write-ups. Anyone have any guesses?

This was a roughly six month project for a single engineer working around 75% of the time on it, with help from other folks along the way for code reviews and etc. The first three months was research, planning, implementation, etc and the latter three months was a very careful roll out and migration from the old system to the new and finally decommissioning the old system.

Re: DNS Infrastructure at GitHub

#4

Post by bogomipz » Wed, May 31, 2017, 6:02 PM UTC

logicalstack wrote: post #3

chrxr wrote: post #2

I'm always interested to know how long these infrastructure changes took from project initiation to launch, and whether it was a dedicated project team or something completed alongside BAU tasks. These details are almost never included in these write-ups. Anyone have any guesses?

This was a roughly six month project for a single engineer working around 75% of the time on it, with help from other folks along the way for code reviews and etc. The first three months was research, planning, implementation, etc and the latter three months was a very careful roll out and migration from the old system to the new and finally decommissioning the old system.

I noticed PowerDNS in the mix, can you say what backend you are using with PowerDNS and how that has been?

Re: DNS Infrastructure at GitHub

#5

Post by chrxr » Wed, May 31, 2017, 6:04 PM UTC

logicalstack wrote: post #3

chrxr wrote: post #2

I'm always interested to know how long these infrastructure changes took from project initiation to launch, and whether it was a dedicated project team or something completed alongside BAU tasks. These details are almost never included in these write-ups. Anyone have any guesses?

This was a roughly six month project for a single engineer working around 75% of the time on it, with help from other folks along the way for code reviews and etc. The first three months was research, planning, implementation, etc and the latter three months was a very careful roll out and migration from the old system to the new and finally decommissioning the old system.

Thanks very much for the reply. Very useful. I think these kinds of details really help people in other organizations who might want to undertake similar projects.

Re: DNS Infrastructure at GitHub

#6

Post by logicalstack » Wed, May 31, 2017, 6:26 PM UTC

bogomipz wrote: post #4

logicalstack wrote: post #3

Earlier quoted context omitted.

This was a roughly six month project for a single engineer working around 75% of the time on it, with help from other folks along the way for code reviews and etc. The first three months was research, planning, implementation, etc and the latter three months was a very careful roll out and migration from the old system to the new and finally decommissioning the old system.

I noticed PowerDNS in the mix, can you say what backend you are using with PowerDNS and how that has been?

We use the mysql backend and http API, a few small nits but for our purposes it has worked very well thus far. Note that our authorities never see production traffic outside of AXFRs from our "edge" hosts so I can't say how well it works for other use cases.

Re: DNS Infrastructure at GitHub

#7

Post by bogomipz » Wed, May 31, 2017, 6:35 PM UTC

logicalstack wrote: post #6

bogomipz wrote: post #4

Earlier quoted context omitted.

I noticed PowerDNS in the mix, can you say what backend you are using with PowerDNS and how that has been?

We use the mysql backend and http API, a few small nits but for our purposes it has worked very well thus far. Note that our authorities never see production traffic outside of AXFRs from our "edge" hosts so I can't say how well it works for other use cases.

Thanks!

Re: DNS Infrastructure at GitHub

#8

Post by pbarnes_1 » Wed, May 31, 2017, 7:28 PM UTC

logicalstack wrote: post #3

chrxr wrote: post #2

I'm always interested to know how long these infrastructure changes took from project initiation to launch, and whether it was a dedicated project team or something completed alongside BAU tasks. These details are almost never included in these write-ups. Anyone have any guesses?

This was a roughly six month project for a single engineer working around 75% of the time on it, with help from other folks along the way for code reviews and etc. The first three months was research, planning, implementation, etc and the latter three months was a very careful roll out and migration from the old system to the new and finally decommissioning the old system.

Do queries to github.net stay internal or do you also sync github.net zones to Route53/Dynect ... just in case?

We have a similar setup with unbound and nsd (no need for powerdns for us). Even then it took a while to get it right because JVM apps especially love to hang for no reason doing NS lookups. You also need to specify -Dnetworkaddress.cache.ttl= etc since they don't listen to TTLs.

Running unbound on every single machine has saved us a lot of downtime.

Re: DNS Infrastructure at GitHub

#9

Post by logicalstack » Wed, May 31, 2017, 7:50 PM UTC

pbarnes_1 wrote: post #8

logicalstack wrote: post #3

Earlier quoted context omitted.

This was a roughly six month project for a single engineer working around 75% of the time on it, with help from other folks along the way for code reviews and etc. The first three months was research, planning, implementation, etc and the latter three months was a very careful roll out and migration from the old system to the new and finally decommissioning the old system.

Do queries to github.net stay internal or do you also sync github.net zones to Route53/Dynect ... just in case? We have a similar setup with unbound and nsd (no need for powerdns for us). Even then it took a while to get it right because JVM apps especially love to hang for no reason doing NS lookups. You also need to specify -Dnetworkaddress.cache.ttl= etc since they don't listen to TTLs. Running unbound on every si…

Nearly all of our internal zones are internal and not sync'd to an external provider. In a few cases we need to perform lookups of internal zones external to our network and those zones live both internal and external.

Re: DNS Infrastructure at GitHub

#10

Post by antoncohen » Wed, May 31, 2017, 8:05 PM UTC

logicalstack wrote: post #3

chrxr wrote: post #2

I'm always interested to know how long these infrastructure changes took from project initiation to launch, and whether it was a dedicated project team or something completed alongside BAU tasks. These details are almost never included in these write-ups. Anyone have any guesses?

This was a roughly six month project for a single engineer working around 75% of the time on it, with help from other folks along the way for code reviews and etc. The first three months was research, planning, implementation, etc and the latter three months was a very careful roll out and migration from the old system to the new and finally decommissioning the old system.

Do you still run a local caching DNS daemon on every server? If not, why the change?