Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Excessive load on NTP servers

news.ntppool.org

Post reply on HN
Print view View original

141–150 of 157 posts

Re: Excessive load on NTP servers

#141

Post by acqq »

brazzledazzle wrote: post #138
acqq wrote: post #98

Earlier quoted context omitted.

> I'm guessing of cause, (...) they used the ntp.org pool, because that's what a quick Google search would tell you to do You're wrong, see my upper comment in this very thread, I've wrote already: they used the third-party library which has as default in the call without any arguments the huge number of the ntp.org servers. The servers hit and the servers in the library sources match exactly. They used that default,…

Looks like the library author removed the unsafe default after someone opened an issue several hours ago.

Thanks! The whole thread there is worth reading:

https://github.com/jbenet/ios-ntp/issues/47

The original author writes:

" I wrote this library a long time ago for the iPhone 3GS and earlier. At that time, as best I could tell, the iPhone obtained its time from the phone company and, for me in South-East Michigan, it was not unusual for the iPhone time to be at up to two minutes variance from true time. Since my app was predicting the location of the International Space Station and a two minute error represented about 600 miles, I wrote this library so people wouldn't be staring at the wrong side of the sky!"

"Since then, the world has changed. First, iOS devices use the NTP protocol as a time source -- I stopped using my own library years ago because iPhones were delivering sub-second accurate times natively."

"I've thought for a while that this library had passed its "sell-by" date and was of minimal value; this Snapchat incident (about which I've heard only a little, and that obliquely), makes me want to remove it. I've been involved in network software, on and off, for about forty years and regard myself as a good net citizen."

The defaults up to now were really, really bad. His correction now is to remove the list. But if then the users just use any list, they are doing it wrong again. For effectively all the scenarios in practice, just time.apple.com instead of any other list and the limited number of IP addresses used from there, by default 1, should be enough for the iOS apps, or even better that they use their own servers. The swift library with the similar goals, mentioned elsewhere, has that IP-limiting loop, this library probably not.

Re: Excessive load on NTP servers

#142

Post by uremog »

Declanomous wrote: post #32
Twirrim wrote: post #26

Earlier quoted context omitted.

> I guess it will be entertaining to see how long it takes for IT to figure this one out. Why not just tell them. What have you got to lose? Hell, blame your charitable spirit on the holiday season if you must.

I got told off for diagnosing issues in the past. The IT director is a megalomaniac and interprets it as a challenge to his power. The only time I offer suggestions now is when one of his employees specifically asks me for help. Edit: I realize "got told off" didn't really capture what happened. I came in early one day and noticed we were having a dns issue. I manually refreshed my DNS cache and it started to work. I…

So he forbids anyone from considering possibilities of why any IT might be broken? LOL

Re: Excessive load on NTP servers

#143

Post by brainfire »

jrockway wrote: post #135
brainfire wrote: post #62

Earlier quoted context omitted.

I've never used Snapchat but I believe one of its features is time-expiring photos. If they do the expiration in the client then this may be a way to check if a user is getting around it by setting the system clock backwards.

NTP doesn't cryptographically verify the time. All you do is have your router redirect these NTP requests to your own server, which is set to serve the wrong time. Ironically, part of an HTTPS handshake involves sharing the server time in a cryptographically-verifiable manner. I am not sure why they don't use that! https://github.com/ioerror/tlsdate

Well, I didn't say it was a good way ;)

Re: Excessive load on NTP servers

#144

Post by nxtrafalgar »

_RPM wrote: post #121

And to think that SC's engineering is praised among college kids is laughable.

On my device, at various points, Snapchat and Uber have both been completely nonfunctional for days on end. I didn't think building phone apps could be so difficult for these large companies.

Re: Excessive load on NTP servers

#145

Post by Declanomous »

brazzledazzle wrote: post #136
Declanomous wrote: post #88

Earlier quoted context omitted.

We do have our own time source. Our NTP source is locked to that server via GPO. NTP requests aren't blocked, because I can still set NTP on things like my printer to other NTP sources and receive updates. I'm willing to bet they don't realize that they've blocked their server. What I want to know is how the drift got to be this bad in a matter of days. It's like whatever computer they are using as the NTP server doe…

"It's like whatever computer they are using as the NTP server doesn't have a real-time clock." It sounds like their domain controller holding the PDCE FSMO role is a virtual machine. This advice might be outdated but last time I checked it was better to host it on a physical machine. If they're restoring DBs from backups without notifications and managing their Active Directory environment like this I can only imagin…

>It sounds like their domain controller holding the PDCE FSMO role is a virtual machine.

Called it. I didn't even think about the fact that it was a virtual machine until you mentioned it. Everything makes so much sense now. The randomly disappearing files, random issues with file locking. I just assumed the problem was they had no idea what rules to use for file replication.

> I can only imagine how fragile your infrastructure really is.

Yeah, it's a nightmare. Ignoring the "files disappear and are never found" issue, files get overwritten with earlier versions all the time. I'm not entirely sure what their hosting situation is, but I suspect they are running virtual machines in virtual machines.

On top of that, most applications are only available on VMs through a Citrix connection. This isn't inherently a problem, but they have it set up so everyone is logged on to the same machine. This causes issues with most MS Office applications, because they assume one user per computer is accessing the files. So file locking is completely broken, because Windows looks at the file and says "Hey, that's me, I have that open!"

It's the biggest nightmare with Access applications, (which we have a ton of), because Access assumes that everyone has their own copy of the front-end (client) database, and each independently accesses the back-end (server) Access database. Optimistic locking doesn't work at all with the "everyone uses the same file from the same computer" approach, and pessimistic locking barely works. I've set up the client databases to automatically create a copy for each user, but as you can imagine it's a bit of a nightmare to keep everything updated.

Re: Excessive load on NTP servers

#146

Post by acqq »

conradev wrote: post #132
easytiger wrote: post #2

Wait.. they are saying the app itself is making NTP requests? > Confirmed - starting up the iOS Snapchat app does a lookup to the domains you listed, and then sends NTP to every unique IP. Around 35-60 different IPs. Hmm. Is that a fraud prevention thing or something? No way on earth a user app should be getting its own time

> No way on earth a user app should be getting its own time This practice is becoming increasingly common for "time-sensitive applications": https://eng.lyft.com/freezing-time-6ebe8ffe3321

Just checked, at least they use "time.apple.com" as default, and the default "maximum of servers" is 5.

https://github.com/lyft/Kronos/blob/master/Sources/NTPClient...

The third-party library used by Snapchat didn't have any "maximum of servers" (using 30 at once(!)) and defaulted to many in the ntp.org pool, across all the continents!

Re: Excessive load on NTP servers

#147

Post by viraptor »

jrockway wrote: post #135
brainfire wrote: post #62

Earlier quoted context omitted.

I've never used Snapchat but I believe one of its features is time-expiring photos. If they do the expiration in the client then this may be a way to check if a user is getting around it by setting the system clock backwards.

NTP doesn't cryptographically verify the time. All you do is have your router redirect these NTP requests to your own server, which is set to serve the wrong time. Ironically, part of an HTTPS handshake involves sharing the server time in a cryptographically-verifiable manner. I am not sure why they don't use that! https://github.com/ioerror/tlsdate

That only works for current TLS. Version 1.3 makes sending the server date optional.

Re: Excessive load on NTP servers

#148

Post by Dylan16807 »

viraptor wrote: post #147
jrockway wrote: post #135

Earlier quoted context omitted.

NTP doesn't cryptographically verify the time. All you do is have your router redirect these NTP requests to your own server, which is set to serve the wrong time. Ironically, part of an HTTPS handshake involves sharing the server time in a cryptographically-verifiable manner. I am not sure why they don't use that! https://github.com/ioerror/tlsdate

That only works for current TLS. Version 1.3 makes sending the server date optional.

Optional, but does anyone turn it off? They control their own servers, anyway.

Re: Excessive load on NTP servers

#150

Post by viraptor »

Dylan16807 wrote: post #148
viraptor wrote: post #147

Earlier quoted context omitted.

That only works for current TLS. Version 1.3 makes sending the server date optional.

Optional, but does anyone turn it off? They control their own servers, anyway.

Optional for the implementations. So basically your ssl library is unlikely to do it. And at that point having a "getTime()" API call in your service is simpler than having a custom patched SSL implementation.
Post reply on HN