I am 100% not surprised. I spent a year working for Ubiquiti, running the Network Controller team.
Trust me, this whistle-blower "Adam" (I have a few suspicions of who it actually is), toned it down.
The reality is much much worse.
Whistleblower: Ubiquiti Breach “Catastrophic”
91–100 of 815 posts
Re: Whistleblower: Ubiquiti Breach “Catastrophic”
#92It really doesn't get worse than this. But isn't Ubiquiti more of a prosumer company, like MikroTik? MikroTik does get a lot of heat when they have a security vulnerability and get downranked for it as if it were far, far away from Ubiquiti's security profile (something like "US vs. some east EU country"), but this event tells a lot about Ubiquiti's upper management and their internal security practices.
Re: Whistleblower: Ubiquiti Breach “Catastrophic”
#93> “The breach was massive, customer data was at risk, access to customers’ devices deployed in corporations and homes around the world was at risk.” > “They were able to get cryptographic secrets for single sign-on cookies and remote access, full source code control contents, and signing keys exfiltration,” Maybe putting your network control plane in 'the cloud' isn't such a good idea after all... Edit: Just re-read…
Was shopping for alternatives to my Ubiquiti last night. Seems like there is nothing good out there. Engenius has shit hardware and a cloud controller. Aruba has a cloud controller AND you have to pay for a license. Cisco makes you pay for a license. TP-Link is cloud-based. WTF. Does anyone have a decent WAP where I can use PoE, deploy like 5 of them and have them support roaming between APs, all managed locally? Is…
> Does anyone have a decent WAP where I can use PoE, deploy like 5 of them and have them support roaming between APs, all managed locally? Is that too much to ask?
Not as comprehensive as Ubiquiti’s management interface but the CAPsMAN feature on Mikrotik routers and APs does cover this use case.
Re: Whistleblower: Ubiquiti Breach “Catastrophic”
#94Earlier quoted context omitted.
What is the right way store credentials to something like this? Hardware keys?
The root account credentials should be used to create a privileged IAM user and then physically locked away in a box after setting up a hardware MFA device (plus a backup MFA) for the root account: https://docs.aws.amazon.com/IAM/latest/UserGuide/best-practi... The privileged IAM user should then be used to administer other IAM users and roles. All IAM users should be required to have hardware security keys like Yubi…
But how fast a determined attacker will be able to utilize acquired physical key?
Is something like kidnapping in the threat model for companies like ubiquiti?
Re: Whistleblower: Ubiquiti Breach “Catastrophic”
#95I wonder why their legal department would PREVENT them from saving their users. What legal reason would exist for that? I thought legal would instead force them to save their users, since otherwise they would risk getting sued by all of them by all the damages caused or something.
> a source who participated in the response to that breach alleges Ubiquiti massively downplayed a “catastrophic” incident to minimize the hit to its stock price, and that the third-party cloud provider claim was a fabrication.
I'm sure their lawyers don't know anything about tech or forensics, but they know how buy shareholders time in a way that minimizes anyone's chances of going to prison or facing serious civil liability. If you ask someone in charge of hiring corporate counsel what they look for in a lawyer, they will flat out tell you "a good risk manager who understands discretion" which just means "someone who's going to tell us what we can get away with".
The regulatory system in the US is sufficiently dysfunctional that there is zero incentive for corporate counsel to even consider what's in the best interest of consumers.
Re: Whistleblower: Ubiquiti Breach “Catastrophic”
#96Ubiquiti is another one of these companies where if you did nothing but read about them on HN, Reddit, et al, you would think they're filing for bankruptcy tomorrow, set orphanages on fire, kill puppies, etc. The negative hyperbole around this company is something else, hack or not. And yet, all they do is thrive...
The hardware is very cheap and the market for their products is thriving. In fact it's possible to put custom software on it actually without using their cloud. > if you did nothing but read about them on HN, Reddit, et al, you would think they're filing for bankruptcy tomorrow, set orphanages on fire, kill puppies, etc. I need to check these posts ;)
Seriously I'm just tired of it. Do you know how many tech geeks over the last few years have proudly proclaimed online that the company is "going downhill" and they'll never buy any more Ubiquiti products? 50 billion, that's how many. How many follow through? Evidently zero. It's comical. The hack obviously not good, but GMAFB.
Re: Whistleblower: Ubiquiti Breach “Catastrophic”
#97I wonder why their legal department would PREVENT them from saving their users. What legal reason would exist for that? I thought legal would instead force them to save their users, since otherwise they would risk getting sued by all of them by all the damages caused or something.
> I wonder why their legal department would PREVENT them from saving their users.
Good legal departments understand that the company is there to serve the users and make them happy and operate within those constraints (even trading off possibly liability when it makes the products sell better).
Horrible legal departments will block anything that has even a smell of liability, even when it comes to sabotaging the product itself and hiding serious issues from users and employees.
I've met way too many ones from the second group.
Re: Whistleblower: Ubiquiti Breach “Catastrophic”
#98> “The breach was massive, customer data was at risk, access to customers’ devices deployed in corporations and homes around the world was at risk.” > “They were able to get cryptographic secrets for single sign-on cookies and remote access, full source code control contents, and signing keys exfiltration,” Maybe putting your network control plane in 'the cloud' isn't such a good idea after all... Edit: Just re-read…
Was shopping for alternatives to my Ubiquiti last night. Seems like there is nothing good out there. Engenius has shit hardware and a cloud controller. Aruba has a cloud controller AND you have to pay for a license. Cisco makes you pay for a license. TP-Link is cloud-based. WTF. Does anyone have a decent WAP where I can use PoE, deploy like 5 of them and have them support roaming between APs, all managed locally? Is…
Mikrotik, but unfortunately getting reasonable throughput for wireless clients is a serious challenge (I always have better results with openwrt on the same hardware). Still, nice to have local control and not have to rely on some cloud service just to use the hardware I bought.
Re: Whistleblower: Ubiquiti Breach “Catastrophic”
#99Earlier quoted context omitted.
For AWS root account? Generate a long random password, print it out and then lock it in a safe without allowing anyone to see it. Turn on 2FA and then lock the second factor in a different safe. There’s virtually never a need for the root account and it’s impossible to attenuate (by design).
This is a lot harder to do if you have lots of AWS accounts and create new ones over time on-demand (e.g. AWS account per team).
Use Organizations. If you’re creating new standalone independent accounts for teams you’re just seeking yourself up for some kind of billing/security/governance catastrophe down the road.
Re: Whistleblower: Ubiquiti Breach “Catastrophic”
#100Cloud managed anything has a giant red target painted on it. Especially infrastructure equipment. I'm still surprised anyone think's it's ok to use their ISP provided router and wifi, let alone having it be managed remotely by the manufacturer.
The problem is that on-prem isn't much better in many cases. Only the largest organizations have the capability to operate deep defenses against these threats whether it's the cloud, or the on-prem.
If you and your team have the skills you can operate fairly effectively on a small scale, but that's a pretty luxurious situation. Most home users can't tell the difference between a router and cable modem hence it's in the interest of cable providers to lower support costs by providing a managed offering. It's terrible from a security perspective, but customers have signed that away.
The common theme running through these breaches is that the organization isn't necessarily small, but they aren't Google/Apple/Microsoft-size either. Those companies have multiple layers of expertise and the cash flow to hold up development of anything in order to make sure things are secure. It's hard to wing stuff once the bureaucracy understands security is needed. They even start pushing their product security initiatives outside of product development to mundane departments because they get attacked by very smart actors. You can see from the news it's still far from perfect.
Once you get to companies the size of Ubiquiti, you start having challenges with implementing close to the same degree of security because you don't have float in the system to allow for additional costs, delays, etc. on top of the lack of expertise. Apparently Ubiquiti have been hemorrhaging expertise in other areas due to opportunistic cost-cutting, so it isn't a surprise that they suffer and respond in this way given that culture. A bad security decision by one exec in companies of this size can cut across many departments which doesn't happen in the behemoths.