71–80 of 119 posts
Earlier quoted context omitted.
Isn't the EEPROM "patch" with ethtool simply sending the inoculating packet (the one with the "40" value) through the network stack?
That's not the impression I'm getting from the article or the author's comments.
Earlier quoted context omitted.
> the disbelief many show when hearing a software message can bring down a NIC. Shouldn't be a surprise to anyone. Firmware is just software, and it necessarily deals with raw bytes. Not really surprising that it can contain bugs that are triggered by certain byte patterns.
It would be interesting to see what the actual error in the NIC firmware source was. This invalidates my assumption that a shop like Intel probably uses formal verification in firmware development. Its also scary to consider how many very important (nuclear/damn control, etc) systems while themselves might be formally verified are dependent on services of lower level software (OS, drivers) and hardware (firmware) tha…
PS You should be very frightened. The industrial control systems industry are slow to adopt new tech/new best practices/anything else new; and even more reticent to touch anything that "works." Tons of vulnerable equipment controlling dangerous things with default passwords (if any at all). I could go on for days.
You'd think the odds of getting a packet with 0x32 in position 0x47f is almost 1/256 per packet? So why aren't these network cards falling over everywhere every few seconds?
Earlier quoted context omitted.
It would be interesting to see what the actual error in the NIC firmware source was. This invalidates my assumption that a shop like Intel probably uses formal verification in firmware development. Its also scary to consider how many very important (nuclear/damn control, etc) systems while themselves might be formally verified are dependent on services of lower level software (OS, drivers) and hardware (firmware) tha…
Dam control, or damn control? I teach a control systems course, so I know about both kinds. PS You should be very frightened. The industrial control systems industry are slow to adopt new tech/new best practices/anything else new; and even more reticent to touch anything that "works." Tons of vulnerable equipment controlling dangerous things with default passwords (if any at all). I could go on for days.
That is great HN content! Debugging deep down the rabbit hole, until you find a bug in the NIC EEPROM - and the disbelief many show when hearing a software message can bring down a NIC. I for one would enjoy reading more content like this on HN that what qualifies as best as a friday-night hack
> the disbelief many show when hearing a software message can bring down a NIC. Shouldn't be a surprise to anyone. Firmware is just software, and it necessarily deals with raw bytes. Not really surprising that it can contain bugs that are triggered by certain byte patterns.
Very good detective work. However, a small suggestion, given: I’ve been working with networks for over 15 years and I’ve never seen anything like this. I doubt I’ll ever see anything like it again. This is a very excellent case for fuzz testing. My thinking is that you want to whip up your Ruby and your EventMachine and Redis going and run a constant fuzz with all sorts of packets in your pre-shipping lab. The idea i…
Fuzz testing would have been very unlikely to help since any byte value at that position besides ASCII 1, 2, or 3 "inoculated" the NIC from the bug. There is a very excellent case to be made for fuzz testing, but this isn't it. Given that, for all we know the relevant parties did conduct extensive fuzz testing and your condescension is misplaced.
It wouldn't surprise me at all if there are other issues that don't depend on these exact circumstances to get something to fall over.
Earlier quoted context omitted.
It would be interesting to see what the actual error in the NIC firmware source was. This invalidates my assumption that a shop like Intel probably uses formal verification in firmware development. Its also scary to consider how many very important (nuclear/damn control, etc) systems while themselves might be formally verified are dependent on services of lower level software (OS, drivers) and hardware (firmware) tha…
Dam control, or damn control? I teach a control systems course, so I know about both kinds. PS You should be very frightened. The industrial control systems industry are slow to adopt new tech/new best practices/anything else new; and even more reticent to touch anything that "works." Tons of vulnerable equipment controlling dangerous things with default passwords (if any at all). I could go on for days.
Earlier quoted context omitted.
In this light, the surprising thing is that we don't hear about firmware bugs more often. Have manufacturers have been doing a great job of keeping things simple and exhaustively testing, driven by the fear of the losses an incident and recall would cause?
Rare bugs on released hardware are almost by nature ignored in favor of new products. Its also why fuzzing is such as great method in finding security bugs.
It appears to work if you send the packet to the network broadcast address. Quick way to detect if any of the machines are vulnerable(they won't respond to the second ping).
you conveniently omitted the part in which you walk to the racks and reboot them all.
I ran into a similar problem with an Intel motherboard about 10 years ago. We had problems when some NFS traffic would end up getting stalled. Our NFS server would use UDP packets larger than the MTU and they would end up getting fragmented. Turns out the NIC would not look at the fragmentation headers of the IP packet and always assume a UDP header was present. From time to time, the payload of the NFS packet would…
Reading about the OP's issue got me to a doc from Intel ( http://www.intel.com/content/dam/doc/application-note/sideba... ) re: the "NC Sideband Interface", which sounds like the place where the bug that bit you "lives". Reading over that doc made me shudder a few times, thinking about the complexity and, thus, potential bugs that could be lurking there. I wonder if the OP's bug was related, too. Having the NIC inspe…