And to think that SC's engineering is praised among college kids is laughable.
Excessive load on NTP servers
121–130 of 157 posts
Re: Excessive load on NTP servers
#122Earlier quoted context omitted.
I got told off for diagnosing issues in the past. The IT director is a megalomaniac and interprets it as a challenge to his power. The only time I offer suggestions now is when one of his employees specifically asks me for help. Edit: I realize "got told off" didn't really capture what happened. I came in early one day and noticed we were having a dns issue. I manually refreshed my DNS cache and it started to work. I…
What kind of firm are you in (and perhaps more important: country/culture)? I've actively been a friendly pita wherever I worked and always found people to help search for answers, even if the question was not really in the line of work. Have a discussion related to tax and payrolling going on right now. Couldn't make sense of company policy even when reading / quoting the law (ianal). Still the surroundings stay fri…
Non-profit foundation.
I'm definitely a friendly PITA. Most of my coworkers get it, though some definitely think that I'm just creating more work for myself. It's really only the IT director who is so defensive, and unfortunately he has the power to back it up. HR goes out of their way to find answers to questions, and my immediate coworkers are really driven.
Re: Excessive load on NTP servers
#123Why on earth would you do that? If you want to prevent users from altering their time use your server and do a time compare with your server. NTP can be easily intercepted and altered so it would make a lot more sense to do this via a encrypted certificate pinned communication path increasing my work load drastically to alter the time. I snapchat going to pay for the DDOS they created?
Stupid decision by SC for sure, but is there a reason there is no rate-limiting on the NTP servers? I'm not up to date on their structure. Maybe it's just not feasible because they don't synchronize clients?
Rate limiting can only control number of responses issued by the server, it won't help reduce number of incoming UDP requests.
NTP has KOD (Kiss of Death) which was meant for that situation, but it is often ignored/not handled properly by applications :/
Re: Excessive load on NTP servers
#124FWIW my teenage daughter has been complaining about this latest Snapchat update for iOS the past couple days. It constantly crashes and causes the phone to reboot itself. Looking at Twitter, there's tons and tons of people reporting the same issue, so it seems pretty widespread. Wonder if it's related to this NTP issue.
Can apps cause iOS to reboot ? That's a bit shocking. I don't own an iPhone.
Yes, I believe they can. I haven't had my iPhone (6S) crash more than 5 times that I can recall in the 14 months I have had it. Every time it has rebooted I have been in a third party app. The phone begins acting strange, gestures may stop for a few moments then it goes black and the Apple logo shows up on the screen as it reboots.
Two of them have been in the last month or so. Unfortunately the phone doesn't just reboot. It shuts down completely and refuses to boot back up with a low battery warning. I believe this is actually hardware related but I have had it crash when trying to call an Uber. I think Facebook may have caused a crash too.
Uber was especially painful because hailing taxis in my city (Seattle) is essentially impossible and pay phones are not a thing that exist in 2016. Luckily I was near a bus stop and know how the lines work without my phone.
It was a shocking illustration of how dependent I have become on my smartphone.
Re: Excessive load on NTP servers
#125It is interesting to read through the whole thread in a chronological order starting from the first message: http://mailman.nanog.org/pipermail/nanog/2016-December/08952...
It took 4 days, to zero on the root cause. As is usual in a complex scenario like this there are a few false positives, some suspects abusing the protocol and alas final redemption. Amazing work by a dedicated group of technical folks in coordinating (just via emails, I suppose) and tracing the root cause.
Re: Excessive load on NTP servers
#126Earlier quoted context omitted.
I encountered a similar IT manager in high school. I kept telling him that netsend wasn't locked down and that any user could run a .bat. He told me I was wrong. So, I wrote a .bat with a netsend command and emailed it to all staff. Multiple staff clicked on the attachment. Once they figured out it was me, they made me start a computer club with the IT manager as the supervisor of the club. First order of business wa…
Im endlessly glad that my high school was extremely tolerant of us exploring the system and messing with things as long as we didn't try to cause harm (eg, deleting one file is okay, trashing a whole network drive is not) and reported what we found to the school IT manager. Most of us ended up as techs for the school and district as a student job. Several of those students went on to be whitehats. (Who knows where ca…
Me and my buddy in like 2010 ended up being questioned by police, then he caught some charges. No fun. All we did was spoof some emails.
Re: Excessive load on NTP servers
#127Earlier quoted context omitted.
As a user, I'd rather you build your code and sign it for me. Enabling a third-party to alter your programs taking away the trust I would have in you.
But you are already trusting Apple to manufacture your device and the operating system the app runs on top of.
Hypothetically, signing power could be useful if an organization wanted to attribute malicious code to somebody else.
Re: Excessive load on NTP servers
#128Earlier quoted context omitted.
Im endlessly glad that my high school was extremely tolerant of us exploring the system and messing with things as long as we didn't try to cause harm (eg, deleting one file is okay, trashing a whole network drive is not) and reported what we found to the school IT manager. Most of us ended up as techs for the school and district as a student job. Several of those students went on to be whitehats. (Who knows where ca…
Me and my buddy in like 2010 ended up being questioned by police, then he caught some charges. No fun. All we did was spoof some emails.
[deleted]
Re: Excessive load on NTP servers
#129Earlier quoted context omitted.
If a cached DNS record expired it would not be a problem. If a bad or old record was cached before TTL, clearing your cache works to update it. Maybe ITs ego didn't like your terminology of DNS cache being expired, and telling them theirs is- by definition expired records get looked up again because they no longer exist locally. As former IT it was mildly irritating to get unsolicited advice, But still it sounds like…
Right, the DNS records weren't expired, they were no longer accurate. I'm not sure what the word for that is. The Windows DNS Cache on the local machines was inaccurate. Basically, I couldn't access any websites I had accessed before, but if I tried accessing a new site it worked fine. If I forced a DNS request for a site I had already visited, that site would start working but all the other sites would still be brok…
It's called the resolver cache, and Linux does it too. Most commonly with nscd (use "nscd -i hosts" to clear it, keep in mind this is not specific to DNS requests). In the specific situation of a web browser it's more complicated and you'll find that the web browser caches these things too. If you run Chrome check out chrome://net-internals to get a glimpse of what's going on there.
Re: Excessive load on NTP servers
#130Earlier quoted context omitted.
I don't know why an org would be blocking NTP. But ideally, IT would run their own NTP service (if the shop is big enough), block NTP to any other org, and sync their server with upstream. Similar to package management, DNS, etc. - self-host it all.
Blocking UDP 123 upstream is a common tactic to avoid participating in NTP DDoS botnets. You're best off running your own NTP infrastructure that's isolated from the public Internet NTP servers if you want stable time sources. Put a CDMA NTP server in two locations, let them sync, and go. You can get cheap EndRun servers on eBay for a few hundred bucks.
It's tempting to believe telecom has got their act together, but I've found the public Internet NTP infrastructure to be more stable than CDMA/GSM refclocks. YMMV.