Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Why is my NTP server costing $500 per year? Part 1 (2014)

blog.pivotal.io

Post reply on HN
Print view View original

91–100 of 113 posts

Re: Why is my NTP server costing $500 per year? Part 1 (2014)

#91

Post by duaneb »

njharman wrote: post #90
duaneb wrote: post #66

Earlier quoted context omitted.

Wow, I never realized operators couldn't push fixes to their routers without permission. The internet is indeed a tragedy of the commons: trivial to ruin, but a Sisyphean task to fix.

Most admins would consider having network infrastructure's firmware change outside of their control a bug/misfeature. Not to mention most devices would require reboot to apply change. And to be able to remotely change the code running a HUGE security issue.

Sure. It's also why the internet is super vulnerable to 0-days.

Re: Why is my NTP server costing $500 per year? Part 1 (2014)

#92

Post by zanchey »

Spooky23 wrote: post #3

Hats off to everyone contributing to public services like this. My then company wanted to give back ny doing this many years ago and it was an eye opening experience. We had troubles almost immediately with utilization and script kiddies. The company ended up only doing it for a relatively short period and ended up making contributions to projects instead

Our student-run computing club added a machine to the pool and melted the University's firewall. Oops.

Re: Why is my NTP server costing $500 per year? Part 1 (2014)

#93

Post by rincebrain »

dfc wrote: post #63
brian_cunnie wrote: post #47

Earlier quoted context omitted.

[author] > I wish he'd explained somewhere how they leapt to examining virtualized NTP clients... I had a hunch [wrongly] that the traffic was caused by a particular operating system. I didn't have enough machines to run the tests on bare-metal, so I virtualized them. And I suspected that virtualization would provide a worst-case scenario (the virtualized clocks would be jittery). My big surprise was that Windows was…

I have never understood why people think kod is a useful setting. Why do you think a misbehaving/improperly configured client is going to honor the kod packet? The kod packet helps with some clients but I have never seen it change the behavior of the most egregious abusers. Just ignore future requests from misbehaving clients, there is not a lot of benefit in saying "please stop misbehaving" to a client that does not…

It's not strictly harmful, I think, to presume that some percentage of misbehaving clients might just be misconfigured and honor a KoD, as long as you have other measures, unless I'm overlooking something?

Re: Why is my NTP server costing $500 per year? Part 1 (2014)

#94

Post by jeffbarr »

Animats wrote: post #68

This is the Snapchat bug reported yesterday, right? Incidentally, how is AWS dealing with the leap second next week? Google is going to have their time servers start to run fast around 20 minutes in advance of the leap second, so they're back in sync at 00:00:60 UTC.

Details on AWS at https://aws.amazon.com/blogs/aws/look-before-you-leap-decemb...

Re: Why is my NTP server costing $500 per year? Part 1 (2014)

#95

Post by mitchty »

foota wrote: post #67
mitchty wrote: post #65

Earlier quoted context omitted.

Retry yes, immediately/very quickly no. Ntp is designed to handle network issues transparently.

This was one of the issues wth netgears client, it retried every second until it worked.

Incorrect clients aside, it should at worst be another 64 seconds (from memory so I might be wrong) before a client retries a poll.

Re: Why is my NTP server costing $500 per year? Part 1 (2014)

#96

Post by jlgaddis »

_delirium wrote: post #82
duaneb wrote: post #66

Earlier quoted context omitted.

Wow, I never realized operators couldn't push fixes to their routers without permission. The internet is indeed a tragedy of the commons: trivial to ruin, but a Sisyphean task to fix.

Some operators do, mostly ISPs that lease routers to customers and retain a way to push firmware updates to them (for example, Comcast does this). But router manufacturers typically don't touch the device once it's out of their hands.

Note that cable modems (all of them, not just from Comcast) download their configuration from the provider every time they boot up. Ironically (since it uses TFTP, for one), this is called "secure provisioning".

They might give you a web interface where you can configure certain settings (e.g. integrated Wi-Fi) but the ISP ultimately has at least some control over any cable modem connected to it.

Re: Why is my NTP server costing $500 per year? Part 1 (2014)

#97

Post by jlgaddis »

AstroJetson wrote: post #76
dfc wrote: post #63

Earlier quoted context omitted.

I have never understood why people think kod is a useful setting. Why do you think a misbehaving/improperly configured client is going to honor the kod packet? The kod packet helps with some clients but I have never seen it change the behavior of the most egregious abusers. Just ignore future requests from misbehaving clients, there is not a lot of benefit in saying "please stop misbehaving" to a client that does not…

Can you post a few more lines about this? I'm in the process of standing up a server and would like to know more about controlling the load. Thanks!

If an NTP client is already misbehaving and/or misconfigured to the point where it's considered "abusive", what are the chances the client will do The Right Thing(TM) when it receives a "kiss of death" ("kod") packet from the NTP server?

Enable KOD, by all means, but you may also consider putting in some (high) per-IP rate limiting for 123/UDP in your firewall rules as a backup plan (for if/when clients ignore kod).

Re: Why is my NTP server costing $500 per year? Part 1 (2014)

#98

Post by jlgaddis »

daveguy wrote: post #77
ChuckMcM wrote: post #55

I think this is great look at walking through the analysis. I too experienced a huge spike in NTP traffic in 2014 but it was because of people exploiting NTP for reflection attacks to DDOS other parties. The forced me to use a GPS module and a Beaglebone Black as an internal time server (which has been great)

I have a few questions about that if you have a minute: What GPS module did you go with and is it still available? Did you have problems getting signal inside (need to be by a window, run an antenna, etc)?

I've also used the Adafruit module, although with a Raspberry Pi. FWIW, the PPS signal seemed to be a little off from another receiver though (Garmin GPS 18x LVC). I never tried to pinpoint the issue but I strongly suspected the Pi. No noticeable issues w/ the antenna inside on a window sill (should be fine as long as you can see four satellites).

I haven't tried it myself but I've heard of several other good experiences w/ the BeagleBone Black. The Garmin seemed to work the best for me, although it is a little more expensive. I was strongly considering putting a few of them in $work's (private) facilities as a fun, nerdy project but I never got around to actually doing it. The Garmin with a BBB might very well be a great combination for that.

One other thing: make sure you use a "real" serial (or parallel) port -- not a USB to serial adapter!

Re: Why is my NTP server costing $500 per year? Part 1 (2014)

#99

Post by jlgaddis »

brian_cunnie wrote: post #64
z92 wrote: post #32

Earlier quoted context omitted.

This $500 would have been $60, on a Digital Ocean box. DO has 1 TB/month limit. Their usage was 300 GB/month.

[author] Digital Ocean is a great deal! Thanks for pointing that out. The reason I use {aws,azure,google} to host my NTP servers is that my day job is developing a VM orchestrator (BOSH) for Cloud Foundry, and BOSH doesn't support Digital Ocean yet (AFAIK). But that's a personal choice, and an admittedly expensive one.

Why not just use the public pool? A virtualized NTP server isn't the ideal scenario and there are plenty of "not quite 100% public" NTP servers you could use as well (i.e. you first have to send an e-mail to get access).

Re: Why is my NTP server costing $500 per year? Part 1 (2014)

#100

Post by jlgaddis »

zanchey wrote: post #92
Spooky23 wrote: post #3

Hats off to everyone contributing to public services like this. My then company wanted to give back ny doing this many years ago and it was an eye opening experience. We had troubles almost immediately with utilization and script kiddies. The company ended up only doing it for a relatively short period and ended up making contributions to projects instead

Our student-run computing club added a machine to the pool and melted the University's firewall. Oops.

Honestly, that's the University's fault then. Properly configured, it should've had very little noticeable effect on the firewall (i.e. "permit udp any host 10.11.12.13 eq 123") as there's no need to do any inspection or tracking state ...

... unless they saturated the available bandwidth but, really, that's a different issue (although also preventable!).

Post reply on HN