Packets of Death
blog.krisk.org
Packets of Death
1–10 of 119 posts
Re: Packets of Death
#2I'm not surprised - firmware for ethernet controllers have grown quite complex, with the addition of new features that allow the hardware to do more work on behalf of the kernel.
Could this be a bug in the code of the EEPROM that handles TCP offloading, or one of the other hardware features that are now becoming more common? (https://en.wikipedia.org/wiki/TCP_offload_engine)
Re: Packets of Death
#3Intruiging.
Intel 82574L ethernet controller looks to be popular too. Intel, Supermicro, Tyan and Asus use it on multiple current motherboards and Asus notably on their WS (Workstation) variants of consumer motherboards, e.g. the Asus P8Z77 WS (socket LGA 1155) and Asus Z9PE-D8 WS (dual CPU, socket LGA 2011).
Re: Packets of Death
#4Intruiging. Intel 82574L ethernet controller looks to be popular too. Intel, Supermicro, Tyan and Asus use it on multiple current motherboards and Asus notably on their WS (Workstation) variants of consumer motherboards, e.g. the Asus P8Z77 WS (socket LGA 1155) and Asus Z9PE-D8 WS (dual CPU, socket LGA 2011).
[deleted]
Re: Packets of Death
#5I've always had mixed emotions about NICs that have hardware assisted offload features. I welcome the decrease in CPU utilization and increased throughput, but the NIC ends up being a complex system that very subtle bugs can lurk inside versus being a simple I/O device that a kernel driver controls.
If there's denial of service hiding in there I wonder about what other security bugs might be lurking. It's scary stuff, and pretty much impossible to audit yourself.
Edit:
Also, I'm a little freaked-out that the EEPROM on the NIC can be modified easily with ethtool. I would have hoped for some signature verification. I guess I'm hoping for too much.
Edit 2:
I wonder if this isn't the same issue described here: https://bugzilla.redhat.com/show_bug.cgi?id=632650
Re: Packets of Death
#6As someone who works with FPGAs/ASICs, this isn't that weird.
Everything gets serialized/deserialized these days, so there's all kinds of boundary conditions where you can flip just the right bit and get the data to be deserialized the wrong way.
What's more interesting is that it bypasses all of the checks to prevent this from happening.
Here is the wiki page on the INVITE OF DEATH which sounds like the problem you hit:
Re: Packets of Death
#7As someone who works with FPGAs/ASICs, this isn't that weird. Everything gets serialized/deserialized these days, so there's all kinds of boundary conditions where you can flip just the right bit and get the data to be deserialized the wrong way. What's more interesting is that it bypasses all of the checks to prevent this from happening. Here is the wiki page on the INVITE OF DEATH which sounds like the problem you…
The INVITE of death was discovered on Feb 16th, 2009.
Re: Packets of Death
#8As someone who works with FPGAs/ASICs, this isn't that weird. Everything gets serialized/deserialized these days, so there's all kinds of boundary conditions where you can flip just the right bit and get the data to be deserialized the wrong way. What's more interesting is that it bypasses all of the checks to prevent this from happening. Here is the wiki page on the INVITE OF DEATH which sounds like the problem you…
The INVITE of death was discovered on Feb 16th, 2009. http://ims-bisf.nexginrc.org/OpenSBC-vul.html
[deleted]
Re: Packets of Death
#9Well this hurts. I have a critical machine with a dual NIC Intel motherboard. I had to abandon the 82579LM port because of unresolved bugs in the Linux drivers, and the other one is a 82574L, the one documented in this post.
I suppose I can send just the right ICMP echo packet to router to make it send me back an innoculating frame.
Re: Packets of Death
#10Very good detective work. However, a small suggestion, given:
I’ve been working with networks for over 15 years and I’ve never seen anything like this. I doubt I’ll ever see anything like it again.
This is a very excellent case for fuzz testing. My thinking is that you want to whip up your Ruby and your EventMachine and Redis going and run a constant fuzz with all sorts of packets in your pre-shipping lab.
The idea is that you want to create a condition where you do see it, and the other handful of lockups that are there that you haven't yet seen.