Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Why is my NTP server costing $500 per year? Part 1 (2014)

blog.pivotal.io

Post reply on HN
Print view View original

51–60 of 113 posts

Re: Why is my NTP server costing $500 per year? Part 1 (2014)

#51

Post by raverbashing »

spydum wrote: post #26
rincebrain wrote: post #8

Earlier quoted context omitted.

I wish he'd explained somewhere how they leapt to examining virtualized NTP clients, or what they ultimately did (since there's no part 3 that I can find).

Virtualization and time sync have had notorious problems. One ugly work around was frequent NTP polling and adjustments. NTP has a min and max poll interval, and it determines how frequently it should poll automatically based on how far it sees drift happening. If it drifts pretty fast, it will quickly gravitate to the minpoll value, which is exactly what they show in their first graph: tons of polling at the minfreq…

I wonder, should this be solved with local NTP servers that resolve from the name pool.ntp.org?

Re: Why is my NTP server costing $500 per year? Part 1 (2014)

#52

Post by brian_cunnie »

fxlv wrote: post #30

I'm surprised there are no comments about the fact that these guys decided to run NTPD on a VM.

[author]

NTP runs fairly decently in a VM. Don't take my word for it — look at the graphs of my servers:

Here's my Google VM, notice the jitter is within +/- 5 milliseconds:

http://www.pool.ntp.org/scores/104.155.144.4

Here's my Hetzner VM (Germany). +/- 10 milliseconds, though I can't help but suspect the distance from the monitoring station (Los Angeles) may have more to do with it than being a VM:

http://www.pool.ntp.org/scores/78.46.204.247

Here's my AWS VM. Much worse than Google in that it's +/- 50 milliseconds, but still good enough to pass muster with pool.ntp.org:

http://www.pool.ntp.org/scores/52.0.56.137

Here's my Azure VM. It's in Singapore, and I re-deployed it last night, so the numbers are still coming in, but it has a pretty tight distribution:

http://www.pool.ntp.org/scores/52.187.42.158

Re: Why is my NTP server costing $500 per year? Part 1 (2014)

#53

Post by brian_cunnie »

lucb1e wrote: post #29

I'm not sure if I've missed it, but is the question (from the title) ever answered? The discrepancy between expected traffic volume and actual traffic volume is huge and seemingly unexplained.

[author]

My bad — I never wrapped it up. Thanks to the HN interest, I'll try to write Part 3 over the winter break.

The short version is this: it's gonna cost a couple of hundred dollars to run a 1Gbe NTP server in pool.ntp.org, but you can tweak the ntp.conf to save ~$100.

Re: Why is my NTP server costing $500 per year? Part 1 (2014)

#54

Post by brian_cunnie »

Faaak wrote: post #5

It was mainly due to the poorly coded snapshat program: https://news.ntppool.org/2016/12/load/ EDIT: this post was indeed from 2014. My bad then. however the same issue started again two weeks ago (~17 dec 2016).

Here is a visual representation of the affect of the snapchat broken-ness on my NTP server:

https://cloud.githubusercontent.com/assets/1020675/21468123/...

Note that inbound traffic which was steady at ~4k packets/sec spikes as high as five times as much. Also note that the snapchat traffic followed a circadian rhythm (much higher traffic during the daytime).

Re: Why is my NTP server costing $500 per year? Part 1 (2014)

#55

Post by ChuckMcM »

I think this is great look at walking through the analysis. I too experienced a huge spike in NTP traffic in 2014 but it was because of people exploiting NTP for reflection attacks to DDOS other parties. The forced me to use a GPS module and a Beaglebone Black as an internal time server (which has been great)

Re: Why is my NTP server costing $500 per year? Part 1 (2014)

#56

Post by brian_cunnie »

late2part wrote: post #36
fxlv wrote: post #30

I'm surprised there are no comments about the fact that these guys decided to run NTPD on a VM.

t1.micro to boot - with more than the usual nondeterministic statmuxing and arbitrary "fairness" policy.

[author]

Yeah, maybe that's why my AWS instance is the most jittery of my 4 timeservers (Google, Hetzner, Azure).

Re: Why is my NTP server costing $500 per year? Part 1 (2014)

#57

Post by fxlv »

brian_cunnie wrote: post #52
fxlv wrote: post #30

I'm surprised there are no comments about the fact that these guys decided to run NTPD on a VM.

[author] NTP runs fairly decently in a VM. Don't take my word for it — look at the graphs of my servers: Here's my Google VM, notice the jitter is within +/- 5 milliseconds: http://www.pool.ntp.org/scores/104.155.144.4 Here's my Hetzner VM (Germany). +/- 10 milliseconds, though I can't help but suspect the distance from the monitoring station (Los Angeles) may have more to do with it than being a VM: http://www.pool.…

Interesting stats. Thanks!

Re: Why is my NTP server costing $500 per year? Part 1 (2014)

#58

Post by djsumdog »

johansch wrote: post #23

Because you use AWS and they charge insane fees for outgoing bandwidth.

Literally any of the major providers like Linode or Digital Ocean give you much more reasonably priced bandwidth. You don't get all the nice AWS tools, but if all you use in EC2 or S3, you can still use Vagrant or Terraform plus some type of configuration management (Ansible, Puppet, etc.) to provision servers programmatically.

Re: Why is my NTP server costing $500 per year? Part 1 (2014)

#59

Post by rhizome »

raverbashing wrote: post #51
spydum wrote: post #26

Earlier quoted context omitted.

Virtualization and time sync have had notorious problems. One ugly work around was frequent NTP polling and adjustments. NTP has a min and max poll interval, and it determines how frequently it should poll automatically based on how far it sees drift happening. If it drifts pretty fast, it will quickly gravitate to the minpoll value, which is exactly what they show in their first graph: tons of polling at the minfreq…

I wonder, should this be solved with local NTP servers that resolve from the name pool.ntp.org?

I've found it to be good practice to run a single/few stratum 2 node(s) to serve local resources. To whatever degree, it's usually more important that these resources be more in sync with each other than with a satellite, which is fostered by having as few nodes as possible trying their luck over the internet to bogged down public stratum 1 NTP sources, instead configuring them to use a single source of time from a box more in its vicinity.

Re: Why is my NTP server costing $500 per year? Part 1 (2014)

#60

Post by rincebrain »

spydum wrote: post #26
rincebrain wrote: post #8

Earlier quoted context omitted.

I wish he'd explained somewhere how they leapt to examining virtualized NTP clients, or what they ultimately did (since there's no part 3 that I can find).

Virtualization and time sync have had notorious problems. One ugly work around was frequent NTP polling and adjustments. NTP has a min and max poll interval, and it determines how frequently it should poll automatically based on how far it sees drift happening. If it drifts pretty fast, it will quickly gravitate to the minpoll value, which is exactly what they show in their first graph: tons of polling at the minfreq…

I knew that was once an issue, I'm surprised to find it's still an issue these days.

In addition, even with minpoll, that's still a hell of a lot of clients abruptly polling him abusively.

Post reply on HN