Question for people who read the link- Do you like the writing style and inclusion of gifs?
I liked both. It kept things light and breezy, a comedy of errors.
The Internet of Unprofitable Things
41–50 of 129 posts
Re: The Internet of Unprofitable Things
#42Good on the author for playing nice with them. Hardcoding IPs for any purpose is a bad idea, this is literally the reason DNS exists.
I don't think that was really the main problem in this case. Hardcoding any address (DNS or IP) that you do not control is always a horrible idea.
Does anyone really control their IP addresses? I thought that ultimately those were controlled by ARIN, RIPE, et.al.
Re: The Internet of Unprofitable Things
#43In short, if you are ever going to make an embedded device, or an operating system distribution, or anything with NTP default on, please make sure,
1. NEVER, ever hardcode an individual NTP server (in form of a IP or domain). DO NOT just go to a list of NTP servers, then copy a few into your code. DON'T ping pool.ntp.org and get its IP address written down. DON'T DO ANY OF THESE! PLEASE!
2. DO NOT use Stratum 0 and Stratum 1 servers. Please use Tier-2 and lower. Practically, if you follow Rule No.1, then you are always following this rule.
3. If the scale of your system is small, in hundreds, or in a few thousands, PLEASE USE pool.ntp.org, this is the NTP community cluster backed by DNS load balancer. Always request the DNS, and make sure the IP is not cached locally for too long. If you need more than one servers, use 0.pool.ntp.org, 1.pool.ntp.org, 2.pool.ntp.org, etc (3 is often enough).
4. If the scale of your system is large, such as tens of thousand, or you are making a new system, you SHOULD request a customized prefix from pool.ntp.org, such as debian.pool.ntp.org, it helps the community to manage the traffic. If your system is a large commercial one, you ARE REQUIRED to donate some servers to the NTP Pool to compensate the community. Another option is running your own private NTP cluster. The policy is here: https://www.ntppool.org/en/vendors.html
5. If possible run a standard NTP implementation, like NTPd, chrony, or something else as long as it's written professionally. Nowadays even lightbulbs run Linux, then why don't you run a standard NTPd?
But If you can't, then make sure...
(a) implement NTPv4, DO NOT use NTPv1.
(b) Read the new SNTP RFC if you are implementing an SNTP client. http://www.faqs.org/rfc/rfc4330.txt
(c) DO NOT synchronize time on the beginning of an hour, or 00:00 UTC! Select a minute in a hour randomly for synchronization.
(d) Use an exponentially-increase retrial interval, DO NOT keep retrying when the server is unreachable, you are launching a DDoS attack!
(e) Support Kiss of Death packet, your client should immediately stop requesting a server, cease and desist, once a KoD packet is received.
(f) Make sure the client will stop requesting the builtin list of servers, once an alternative server is set by the user.
These should have been written in all textbooks related to practical networking lectures, but apparently, there aren't. People don't even realize that their actions are harmful, and we have all the problems...
--------------------------------
The NTP community is a complete tragedy of the commons. Even many government institutions cannot keep up with the abusive traffic, and stopped providing public NTP servers.
Today, if we don't count Microsoft and Apple's NTP, almost all public NTP servers are provided by the volunteers from https://www.pool.ntp.org. By using DNS, it forms a NTP cluster to distribute the load. These people provide time for the entire Internet, and they are the people who withstand all the abuses day by day.
People just assume they are some random super servers that always work, without being responsible for their actions, such as hardcoding IP addresses, writing abusive retry code (without exponential increment of timeout), and making a cronjob that initialize a synchronization exactly at midnight (without randomization), effectively a DDoS.
Usually, if a device comes with hardcoded NTP addresses, it, in fact, usually indicates their program is poorly-written, and the manufacturers are irresponsible. Those devices have the worst homebrew NTP implementation on the planet,
1. They send ancient NTPv1 packets, while the latest version is NTPv4.
2. They synchronize their time on the beginning of an hour, effectively making a flooding attack. Another larger flooding attack starts at 00:00 UTC.
3. They retry interval is around 3 minutes, if fails to reach the server, make even more traffic to the server, rather than an exponentially-increase interval.
4. They still try to talk to the default hardcoded servers, even if an alternative server list is set.
5. They don't support the Kiss of Death packet, nothing can stop them if they became wild.
Stratum 0/1 servers are the most vulnerable: they have highest accuracy, with reference clock. Despite the acceptable usage of ST-1 is only passing time to downstream, or for scientific purposes, since there's only a handful of them and often listed publicly, they are often spotted by those manufacturers, and put in their devices by default.
Stratum 0/1 are usually provided by universities, or unpaid volunteers for the public good of the Internet. If a single server got hardcoded in those mass-manufactured devices, serious consequences can happen, the volunteer may literally bankrupt: your whole institute/school will be kicked out from the Internet [0]; when you came to the manufacture asking to pay the damage they are responsible for, you are threatened by a lawyer from California. [1] The whole Internet community should honor the spirit of self-sacrifice of these NTP volunteers.
The NTP community pool is Stratum 2+, suitable for general use. It has similar issues of abuses - once you're in and became well-known on the net, there's no way out and you keep receiving bad-traffic, because some clueless people have hardcoded your IP address, or has put it in a cache that never expires. Fortunately given a reasonable bandwidth, it is often a negligible issue and safe to ignore. But there are exceptions. [2] One of my NTP server became DDoSed one day, because an ISP cached the IP address for pool.ntp.org with a large TTL, and the IP address happened to be mine! The traffic was 40 Mbps...
In contrast, NTPd has proper rate-limit mechanism built-in, such as KoD and good pooling interval, blocking NTP does NOT causes more user traffic. What increased is the abuser traffic. The damage caused by a standard NTPd and silly sysadmin is much less significant and is negligible compared to the Internet of Scary Things.
By the way, not only hardware devices can contains dangerous NTP code, but also software.
As long as manufactures still write broken code and unaware of the proper way to use NTP, nothing can be done to solve this issue. Many involved in these misuses and abuses are totally unaware what they are doing. The proper way to use NTP should have been written in all textbooks related to practical networking lectures.
[0]: Flawed Routers Flood University of Wisconsin Internet Time Server http://pages.cs.wisc.edu/~plonka/netgear-sntp/
[1]: Open Letter to D-Link about their NTP vandalism https://web.archive.org/web/20060423012837/http://people.fre...
[2]: Recent NTP pool traffic increase https://mailman.nanog.org/pipermail/nanog/2016-December/0895...
Re: The Internet of Unprofitable Things
#44Question for people who read the link- Do you like the writing style and inclusion of gifs?
... There were gifs?? I guess I'm so used to useless ads and graphics in blog posts, I honestly, scout's honour, did not notice and have no recollection whatsoever! A damning statement on the Internet of today perhaps, but it neither enhanced nor detrimented my reading of the article ️
same! I had to go back and look because I would have sworn there were no images at all! Now I realize my brain had classified them as ads and simply edited them out of my conscious awareness all together.
Re: The Internet of Unprofitable Things
#45Device fleets are incredibly hard to get right, and if you have no updateability, you have to nail them the first time. For someone coming from a cloud background, it's a jungle out there, where all sorts of "easy" or "solved" problems are nothing of the sort. As a sort of PSA, I want to plug the work that we have done at balena.io (formerly known as resin.io) to solve problems like these for everyone deploying Linux…
What kinds of algorigthms you guys use to coordinate fleets?
Also, your story is quite fascinating, can you tell us more?
I am very, and I mean VERY interested in working on something like that, but not sure how to transition out of the generic CRUD web programmer to your area... :(
Re: The Internet of Unprofitable Things
#46In short, if you are ever going to make an embedded device, or an operating system distribution, or anything with NTP default on, please make sure, 1. NEVER, ever hardcode an individual NTP server (in form of a IP or domain). DO NOT just go to a list of NTP servers, then copy a few into your code. DON'T ping pool.ntp.org and get its IP address written down. DON'T DO ANY OF THESE! PLEASE! 2. DO NOT use Stratum 0 and S…
On the one hand, I completely appreciate this. When I set up NTP, I'm careful to think through consequences and tread lightly. After all, I learned my sysadmin skills at a university before the September that Never Ended, so it really was a collegial place back then.
On the the other hand, the internet is a much bigger place. Things are orders of magnitude more complex. The feedback loops that made NTP work well in a 1990s university environment are mostly absent. When a problem happened then, I'd see something in the logs or in packet captures, figure out what was happening, and get the responsible person quickly on the phone. That's not even hazily possible these days.
As much as I'd love to think putting a stern warning in textbooks would fix this, I doubt that would matter at all. What we really need is a major increase in observability or traceability. And failing that, what we'll get is common resources getting sliced up so they fit within domains of accountability.
Re: The Internet of Unprofitable Things
#47Earlier quoted context omitted.
I don't think that was really the main problem in this case. Hardcoding any address (DNS or IP) that you do not control is always a horrible idea.
Does anyone really control their IP addresses? I thought that ultimately those were controlled by ARIN, RIPE, et.al.
If you have your own dedicated IP range and you are advertising routes correctly, then you're as close to ownership of your IP addresses as Google or Facebook are of theirs.
ARIN and RIPE control allocations, but do not control routing to allocated ranges.
Re: The Internet of Unprofitable Things
#48I'm trying to coin "Postel decentralisation" for things like this: people assume NTP is a distributed robust system, but in practice it turns out to be run by one very overwhelmed guy in a basement somewhere. (It could have been set up properly to be distributed here, but they didn't do it)
David L. Mills, the inventor of NTP, probably looks exactly how you picture him.
https://en.wikipedia.org/wiki/Network_Time_Protocol#/media/F...
Re: The Internet of Unprofitable Things
#49Earlier quoted context omitted.
But it's not an "article". It's a blog post. An informal post, written informally, for an obviously technical audience. I very, very rarely explain what "AWS" is when I'm casually writing about cloud stuff. It's table stakes. You should know, or you aren't gonna appreciate reading it anyway.
I agree with you but if you take the time to just maybe write out an acronym once its much easier for us who aren't quite there yet technically to understand the context instead of having to search for 4-5 terms. This gives us a chance to enjoy the write up and gives it maybe a broader audience.
This is a technical person blogging for a technical audience. If you are not tall enough to ride this ride and aren't interested in growing, maybe find another ride.
If this were a project blog post explaining their latest features, I'd agree with you. If the point is outreach, then yes, they should make it accessible. But he's telling a story. A story that requires a relatively deep understanding of the history and practice of operating internet services. Him writing "NTP (Network Time Protocol)" will not make the story much clearer. And if he explains the whole background, then it's no longer a story, it's a general-audience essay. That's a lot of work for you to expect from somebody that you aren't paying.
Re: The Internet of Unprofitable Things
#50Device fleets are incredibly hard to get right, and if you have no updateability, you have to nail them the first time. For someone coming from a cloud background, it's a jungle out there, where all sorts of "easy" or "solved" problems are nothing of the sort. As a sort of PSA, I want to plug the work that we have done at balena.io (formerly known as resin.io) to solve problems like these for everyone deploying Linux…
What kinds of algorigthms you guys use to coordinate fleets? Also, your story is quite fascinating, can you tell us more? I am very, and I mean VERY interested in working on something like that, but not sure how to transition out of the generic CRUD web programmer to your area... :(
To be honest, with fleets in the wild, simple works. We do deploy containers to these devices, but rather than kubernetes-ing the whole thing, it's more like "every device in the fleet must run this container". In IoT/Edge affinity is a big deal, so the payloads are pretty uniform. You don't want your container to migrate away to another device while you're trying to use the device to e.g. fly a drone or open a lock.
The difficulty comes in configuration, where things are much more complex than the cloud, since every device is "special". Different customers, keys, settings of all sorts. So we allow individual environment variable settings for each device.
The real complexity is in ensuring you can reach every device as long as possible, and that the devices behave well when they can't reach the cloud. We set up a VPN to ensure the former, and have a pull-based architecture where the device is responsible for "catching up" to the fleet when it comes back online to handle the latter.
There's a bunch of other things we've solved, like container deltas to improve download times by something like 10-70 times (and reduce bandwidth costs) but honestly the biggest difficulty has been in integrating cloud/hardware/developer/network/OS etc seamlessly, so people can succeed at building fleets, and fast. It's like there's 10,000 papercuts moreso than one or two big problems to solve. It's a long chain of things that all have to work right to make the system work, and each one has to be done in a way so that it doesn't break after some time passes (see the OP for an example).
To answer your last question with (yet another) plug, you can probably run through our getting started guide in an hour and have your first device that you can deploy a js project to pretty fast. If you spend a few days, you can start to have some real accomplishments. It's not that hard to make the jump these days if you have a web background (hell, most of the founders didn't have IoT experience when we started) but there are things you need to learn that relate to hardware and linux if you want to do something more advanced. You can learn at your pace though, and I thoroughly recommend starting with a project you enjoy, something like, but not necessarily https://www.balena.io/blog/make-a-web-frame-with-raspberry-p...