Xfinity outage leaves tens of thousands in the Bay Area without internet

https://media.ccc.de/v/32c3-7133-beyond_your_cable_modem

A email I sent just weeks ago..

________________________

>> ------- Forwarded Message -------- >> Subject: Serious DOCSIS maintenance network issue >> Date: Sat, 23 Oct 2021 21:09:16 -0700 >> From: Admin >> To: xxxx@cablelabs.com, xxxx@cablelabs.com, xxx@cablelabs.com, xxx@cablelabs.com, xxx@cablelabs.com >> >> >> >> Hi all.. >> >> You guys hate me :( First it was Puma and the badmodems.com list and now this.. >> >> I am sorry to directly email you. No need to respond. Its OK, I understand its a legal thing. I wont email again. Sorry for this hassle. Sorry for a long read. >> >> There appears to be a very serious gap in your security best practices and policies that could result in a very widespread serious incident that could effect all DOCSIS systems worldwide and result in a worldwide incident. >> >> This appears to be from MSOs deploying horrendously bad security on the maintenance network. >> >> The issues are being discussed publicly. This thread begins with discussion of firmware and then turns to the maintenance network which appears to have little if any security implemented possibly because there is no modern published best practices for the maintenance network beyond something from the 1990's. https://www.dslreports.com/forum/r31122204-SB6190-Puma6-TCP-... >> >> The maintenance network, which controls all the devices on a DOCSIS network, is susceptible to attack. In fact its nearly criminally negligent in its lack of security and appears to be based on 1990's security protocols of mostly security thru obscurity. .. A subscriber on the LAN side can determine his address on the maintenance network and can ping ANY CPE on the network as long as they are on the same ISP. The CPE are not walled off from each other in any way. This could result in a VAST compromise of the entire MSO network nationwide from a 0-day worm that self spreads via the wide open maintenance network connecting all devices. . . ALL susceptible devices on your network, 10's of millions, could be taken over in hours with a self spreading worm with a nearly impossible task of clean up and maybe a week of complete ISP downtime. This would also result in the largest loss of subscribers in history for cable as people flee to DSL and 5G that day trying to get internet. You would need new firmware for every device that addresses the issue, and getting new firmware will take weeks. All the susceptible CPE might be bricked with no hope of recovery once taken over. The current security practices are inadequacy. The news coverage would be devastating. Each modem/router could attack the subscriber side and scrape data and files. On the ISP side it would lock out all maintenance access and recovery of the devices, and the whole network, nearly impossible. It would setup a serious botnet - possibly the largest ever created when combined with the other top world wide ISPs. It might even result in a Ransom ware attack on a massive scale with all the CPE locked out from the ISP. A silent malware could spread stealthy and then sit on CPE and attack the subscribers quietly by doing fake DNS and even MiM attacks. This could already be the case. A botnet of CPE would be incredibly powerful >> >> This wide open gap appears to exist in most ISPs. So it is a CableLabs lack of proper security vision to keep up with modern threats by doing best practices for the maintenance network seems to be the main issue. 10G offers micronets and SDN containment of LAN devices,,, yet the ISP has nothing like it to protect its own network and its subscribers. >> >> Each ISP will need to do a 3rd party security audit and pentest of all the MSO's maintenance networks and secure them. The kinda emergency level, possible fairly easy temp fix is simple. Isolate each piece of CPE. Right now all CPE can see each other and spread worms. Simply doing a config change could wall off each device with NO downside. This might be able to be implemented maybe in a day. This alone would reduce the issue to nearly zero. BLocking access to the maintenance network from the subscriber is also key and most likely easy. MSOs REALLY need to do this and because these discussions are going on now, badguys could be reading, so RIGHT NOW is the time to secure MSO networks BEFORE a incident occurs. >> >> There may be simple quick solutions to avoid this doomsday scenario.. Make sure you read up to the current postings. https://www.dslreports.com/forum/r31122204-SB6190-Puma6-TCP-... >> >> I will be following up to be sure you got this message. >> >> You can contact me for any further details or respond to this email. >> >> I am the guy who found the Puma issue. So you guys know I can be persistent and noisy. I would really like to hear that CableLabs is going to pursue a whole new approach to device security on the maintenance network including RAPID firmware deployment. EVERYBODY wins.. >> >> Sorry for blasting the email. Sorry to start your monday kinda ruff. Think of it as a cool new feature. >> >> I have contacted all the top 10 MSOs and sent reports to the security teams. They are the guys who made this mess, but, they need a good best practice to follow and that does not seem to be there. >> >> Gone are the days of junk boxes with poor CPUs. MSOs are dropping POWERFUL devices with lots of RAM and Flash. They run Microsoft or Linux. They are connected to a massive bandwidth pipe. It looks possible to take over whole ISPs. These are prime targets no one has noticed yet apparently. Gone are the days of old.. These are high value targets and a bot net of incredible scale... Its time for a top down new approach to firmware and device security.. >> >> Of course none of my doomsday scenarios most likely will ever happen.. And most likely everything is fine.. BUT MSO's can't just keep these maintenance networks so 1990s sloppy. >> >> IMHO.. >> >> xxxxxx xxxxxxxxxx >> >> >> >