While it's ultimately going to help to shame vendors regarding their poor security practices, it's really irritating and unfortunate this is all being framed as a Supermicro issue. How about the other companies in the same market space, like Tyan, that I'm sure are no better? For that matter how about the "Tier 1" OEMs like Dell and HP - how well-written are their BMC firmwares?
Supermicro boards were so bug ridden, why would hackers ever need implants?
71–80 of 81 posts
Re: Supermicro boards were so bug ridden, why would hackers ever need implants?
#72Earlier quoted context omitted.
Good point, I agree with that thinking. But the actual execution of such a hardware-based attack would surely be discovered at some point anyway, and risk the same negative outcome. So then that would leave the only possible conclusion that the story just isn't true at all. In the end, none of it makes clear sense...
The difference is two-fold: actively planting a fake story means that first, the espionage is fake and thus no real intelligence can be gathered, so the only benefit is the hypothetical respect you suggested; second, the story will definitely get out, thus the potential for the negative effect is innately 100%. However, as a real intelligence operation the cost/benefit analysis is inverted, because there is a real, t…
I would say that given the amount of motherboard variants, even gene rationally that have varying differences, especially in component supplies, it was pretty unlikely to see the issue. I mean, while some may take a MB out and inspect it thoroughly, most that I'm aware of, will plug it in and if it works, leave it there.
Re: Supermicro boards were so bug ridden, why would hackers ever need implants?
#73Earlier quoted context omitted.
Maybe for smaller companies, but Apple is very paranoid and AIUI does indeed inspect the hardware to make sure it hasn't been tampered with. I know less about Amazon in this regard but I would expect Amazon to do at least some level of hardware inspection to detect tampering as well.
Forgive my ignorance, but what hardware does Amazon sell?
We're talking about server hardware used by Amazon and Apple, not what they sell.
Re: Supermicro boards were so bug ridden, why would hackers ever need implants?
#74I feel like this article reflects some significant technical confusion. The BMC is supposed to be on a trusted network inaccessible from the outside. I've always viewed authentication on the BMC as being like the numeric lock on luggage--it's designed to keep honest people honest, not for real security. Being able to bypass the BMC security is really not a big deal. What the Bloomberg article says about the hardware…
It's quite incompetent and negligent to put network connectors on servers that cause spectacular failures when exposed to a network. A valid conclusion might indeed be to stop them being accessible from the outside, by installing such server boards in a network-topologically secure location such as a landfill.
It's quite incompetent and negligent to put network connectors on servers that cause spectacular failures when exposed to a network.
This isn't a fair assessment of the situation. Networks that aren't entirely trusted and controlled can cause spectacular failures. By knowing this, administrators can use BMCs safely. My Poweredge server even came with a warning sticker that had to be removed before the DRAC port could be used.
In general, tools can have "pointy parts" with which the user could harm themselves so long as the risks and proper uses are documented and explained adequately.
Re: Supermicro boards were so bug ridden, why would hackers ever need implants?
#75I feel like this article reflects some significant technical confusion. The BMC is supposed to be on a trusted network inaccessible from the outside. I've always viewed authentication on the BMC as being like the numeric lock on luggage--it's designed to keep honest people honest, not for real security. Being able to bypass the BMC security is really not a big deal. What the Bloomberg article says about the hardware…
Unfortunately, some servers piggyback the BMC onto one of the regular network ports.
Re: Supermicro boards were so bug ridden, why would hackers ever need implants?
#76Earlier quoted context omitted.
I can't cite this case specifically, but normally it would be incredibly difficult to impersonate a government official as a source. In my experience verifying a source means weeding out that possibility before publishing... e.g, cross-checking data from a third party (background checks, employment history, social media accounts, public records), then photos of credentials, video chats, etc. Then you cross-reference…
What has truly surprised me in all of this is the skepticism expressed about this being plausible. Most nerd sites are rife with thoughts on how insecure things are and hypothetical ideas on how something could be compromised but all of a sudden this one isn't possible? We know the US Gov't has done it in transit but it's ridiculous to think a state owned manufacturer wouldn't do it on the factory line? We know this…
Yes! I am amazed at the general attitude of skepticism expressed in response to the Bloomberg article.
BTW, Amazon doesn't know anything about security. Every day I observe examples of people who work there, wittingly or unwittingly, doing things to erode any security that might happen to be in place. It's almost entirely run by below average people scrapped up and recruited from the dregs of third world countries.
Re: Supermicro boards were so bug ridden, why would hackers ever need implants?
#77I feel like this article reflects some significant technical confusion. The BMC is supposed to be on a trusted network inaccessible from the outside. I've always viewed authentication on the BMC as being like the numeric lock on luggage--it's designed to keep honest people honest, not for real security. Being able to bypass the BMC security is really not a big deal. What the Bloomberg article says about the hardware…
BMCs like DRAC or iLO are invaluable when you have hundreds or thousands of fresh servers with no OS. The BMC lets you mount an OS or hypervisor ISO in a way reminiscent of DaemonTools et al. , and update bios and other firmware from a shared network folder. I'm pretty sure there's even an API to develop against.
You can bootstrap fresh servers using PXE. The problem with BMC is that it never goes away even after you've booted your system. BMC owns your system and there's no way to completely disable it. Options for disabling it simply control the BMC's software interfaces, and they only work to the extent that the software is bug free. It's like the infamous cPanel, but for hardware--it's a juicy target that you're stuck with.
Re: Supermicro boards were so bug ridden, why would hackers ever need implants?
#78Not saying I believe in one side or the other, but from a standpoint of avoiding detection I think firmware hacking goes out the window. A deep-pocketed attacker isn't going to risk flashing the firmware with a non-oem one on a brand new board leaving the factory. That probably gets quality inspected somehow later on anyway whereas a visual inspection is just a rubber stamp (IE: OK if the box isn't crushed or wet). N…
Maybe for smaller companies, but Apple is very paranoid and AIUI does indeed inspect the hardware to make sure it hasn't been tampered with. I know less about Amazon in this regard but I would expect Amazon to do at least some level of hardware inspection to detect tampering as well.
They also do extensive code reviews, including of imported open source, yet their software is hardly bug free. Bugs can linger for years, often fixed only because someone stumbled upon odd behavior.
Spotting a tiny chip sitting on the SPI bus that looks identical to a bunch of other chips? That doesn't do anything unless it's tickled in just the right way? If you believe Amazon and Apple are even remotely capable of protecting against that....
The solution to these problems is to put critical code and critical secrets on discrete, simple SoCs where you actually have a chance of defending both hardware and software attacks. Apple and Amazon understand this because they already do it. The difficulty is building your software systems (firmware, kernel, etc) to make use of these secure elements, not to mention making them available for ad hoc application software. It's an extremely difficult integration problem, and even when you succeed you haven't.
For example, AFAIU Amazon's servers have secure elements to perform attestation of the box; it's utilized by their hypervisors to authenticate VMs for things like KMS. But it can't actually protect the data in the VM itself, such as the secrets obtained by dint of the attestation. It can't even prevent taking control of the hypervisor. All it does is help Amazon define a fixed security parameter--that you can't impersonate their hardware nodes on the network. That's extremely useful, but ultimately extremely limited.
Re: Supermicro boards were so bug ridden, why would hackers ever need implants?
#79Earlier quoted context omitted.
> a trusted network inaccessible from the outside There is no such practical network which remains such a network for long. All networks must be assumed to be byzantine as they certainly will be compromised at some point, if they're not already.
I think this is the first time I've seen byzantine used like this - I assume you are referring to the Byzantine Generals problem.
Byzantine is a synonym to "convoluted": https://www.thesaurus.com/browse/byzantine?s=t
Re: Supermicro boards were so bug ridden, why would hackers ever need implants?
#80Earlier quoted context omitted.
Maybe for smaller companies, but Apple is very paranoid and AIUI does indeed inspect the hardware to make sure it hasn't been tampered with. I know less about Amazon in this regard but I would expect Amazon to do at least some level of hardware inspection to detect tampering as well.
They also do extensive code reviews, including of imported open source, yet their software is hardly bug free. Bugs can linger for years, often fixed only because someone stumbled upon odd behavior. Spotting a tiny chip sitting on the SPI bus that looks identical to a bunch of other chips? That doesn't do anything unless it's tickled in just the right way? If you believe Amazon and Apple are even remotely capable of…
Code reviews for bugs is very different than security reviews to look for malicious tampering. Apple may have bugs that linger for years, but I'm not aware of any documented case of someone managing to slip a backdoor into Apple software via an open source package.
> Spotting a tiny chip sitting on the SPI bus that looks identical to a bunch of other chips? That doesn't do anything unless it's tickled in just the right way? If you believe Amazon and Apple are even remotely capable of protecting against that....
Why not? X-raying every board, inspecting every single component, making sure it matches up with the documented specs and perhaps with a proven-good board... if you're replacing a component with a different one, or adding a component that wasn't there before (which is the case in this alleged attack), even if the component looks harmless and even if it's tiny, it can still be revealed by a detailed comparison of the board against specs. A component that's the size of a grain of rice is still a component that can be detected.