101–110 of 119 posts
Earlier quoted context omitted.
Probably because there is a 2/256 chance of getting sent the inoculation value. But it's a good question.
Later in the article it states that any value other than 0x31, 0x32 or 0x33 acts as an "inoculation value", so that would be a 253/256 chance for each packet of at least 1151 bytes.
Earlier quoted context omitted.
Later in the article it states that any value other than 0x31, 0x32 or 0x33 acts as an "inoculation value", so that would be a 253/256 chance for each packet of at least 1151 bytes.
is 1151 an unusually large packet? Because otherwise how are these NICs not getting inoculated as soon as they're online?
Earlier quoted context omitted.
> the disbelief many show when hearing a software message can bring down a NIC. Shouldn't be a surprise to anyone. Firmware is just software, and it necessarily deals with raw bytes. Not really surprising that it can contain bugs that are triggered by certain byte patterns.
It would be interesting to see what the actual error in the NIC firmware source was. This invalidates my assumption that a shop like Intel probably uses formal verification in firmware development. Its also scary to consider how many very important (nuclear/damn control, etc) systems while themselves might be formally verified are dependent on services of lower level software (OS, drivers) and hardware (firmware) tha…
Earlier quoted context omitted.
Be very afraid of PCI firmwares. You can insert rootkits there that have full access to RAM. An IOMMU can mitigate this threat.
It sounds like, in this case, the OP is talking about the EEPROM holding code executed by the embedded coprocessor on the NIC (or, at least, lookup tables that the coprocessor uses) rather than a PCI option ROM that will be executed by the host computer's CPU. Depending on how the access to the EEPROM is performed (i.e. if such access is facilitated by the co-processor versus being read out directly from the EEPROM)…
Yes, but this coprocessor has access to the host's PCI bus. That is enough to totally pwn the machine, since it gives read/write access to all memory (well, all memory below 4GB, IIRC, but that's enough.)
There are subtle differences between the offsets I get when I run "ethtool -e interface" versus those in the article that indicate an affected card (but they're quite close).
Mine are:
0x0010: ff ff ff ff 6b 02 69 83 43 10 d3 10 ff ff 58 a5
0x0030: c9 6c 50 31 3e 07 0b 46 84 2d 40 01 00 f0 06 07
0x0060: 00 01 00 40 48 13 13 40 ff ff ff ff ff ff ff ff
Output of "ethtool -i interface" (in case anyone wants to compare notes):
driver: e1000e version: 1.5.1-k firmware-version: 1.8-0
I tested both packet replays by broadcasting to all attached devices on a simple Gbit switch and no links dropped.
Basically one linux box with NVidia embedded gigabit controller could take down the whole segment. It would only happen after a random period, like after days when the box was busy. No two machines connected to the same switch would be able to ping each other any more after that. I suspected the switch, bad cables, etc. In the end I successfully circumvented the problem by buying a discrete gigabit ethernet card for the server in question.
My servers all have the affected cards (two per machine - yikes!) but so far I can't reproduce the bug (yay). There are subtle differences between the offsets I get when I run "ethtool -e interface" versus those in the article that indicate an affected card (but they're quite close). Mine are: 0x0010: ff ff ff ff 6b 02 69 83 43 10 d3 10 ff ff 58 a5 0x0030: c9 6c 50 31 3e 07 0b 46 84 2d 40 01 00 f0 06 07 0x0060: 00 01…
Persistent bugger. "With a modified HTTP server configured to generate the data at byte value (based on headers, host, etc) you could easily configure an HTTP 200 response to contain the packet of death - and kill client machines behind firewalls!" That's worrisome, I'll bet there are lots of not-so-nice guys trying to figure out a way to do just that. There must be tons of server hardware out there with these cards…
$ wget http://zorinaq.com/pub/intel-packet-of-death.txt
It has 0x32 at offset 0x47f regardless of the size of the IP and TCP headers. Try to run the wget AS SOON AFTER HAVING COLD BOOTED the machine (it is the very first packet of 1152+ bytes that determines if the NIC will crash or be inoculated until the next cold boot; well... unless it is the "no-op" packet).
Edit: fixed link.
My servers all have the affected cards (two per machine - yikes!) but so far I can't reproduce the bug (yay). There are subtle differences between the offsets I get when I run "ethtool -e interface" versus those in the article that indicate an affected card (but they're quite close). Mine are: 0x0010: ff ff ff ff 6b 02 69 83 43 10 d3 10 ff ff 58 a5 0x0030: c9 6c 50 31 3e 07 0b 46 84 2d 40 01 00 f0 06 07 0x0060: 00 01…
You need to shut down, boot up the server, and do a test right away. The very first packet of 1152 bytes or more that it receives after a cold boot determines if the NIC is going to be affected or "inoculated" (until next cold boot).
EDIT: it's difficult to tell definitively doing it remotely but I still can't re-produce the bug after a cold boot.
I'm currently working on an open source project where we are chasing "hang really hard and need a reboot to come back" issues with exactly this same ethernet controller, the Intel 82574L. I wonder if it's related!
Our Github issue: https://github.com/SnabbCo/snabbswitch/issues/39