Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Packets of Death

blog.krisk.org

Post reply on HN
Print view View original

21–30 of 119 posts

Re: Packets of Death

#21

Post by Jabbles »

wglb wrote: post #10

Very good detective work. However, a small suggestion, given: I’ve been working with networks for over 15 years and I’ve never seen anything like this. I doubt I’ll ever see anything like it again. This is a very excellent case for fuzz testing. My thinking is that you want to whip up your Ruby and your EventMachine and Redis going and run a constant fuzz with all sorts of packets in your pre-shipping lab. The idea i…

Surely that's the manufacturer's job.

Since it's caused by a specific byte at a specific place, surely you'd only need to fuzz an average of 256 packets (of the required length) to find it... which suggests it wasn't done at all... zero...

Re: Packets of Death

#22

Post by thechut »

jacquesm wrote: post #20

Persistent bugger. "With a modified HTTP server configured to generate the data at byte value (based on headers, host, etc) you could easily configure an HTTP 200 response to contain the packet of death - and kill client machines behind firewalls!" That's worrisome, I'll bet there are lots of not-so-nice guys trying to figure out a way to do just that. There must be tons of server hardware out there with these cards…

I read the whole thing and that is the line that stuck out most to me. This could very scary. It could be used to bring down a webserver

Re: Packets of Death

#23

Post by mikeash »

engtech wrote: post #6

As someone who works with FPGAs/ASICs, this isn't that weird. Everything gets serialized/deserialized these days, so there's all kinds of boundary conditions where you can flip just the right bit and get the data to be deserialized the wrong way. What's more interesting is that it bypasses all of the checks to prevent this from happening. Here is the wiki page on the INVITE OF DEATH which sounds like the problem you…

INVITE of Death looks completely different. There, malformed packets would cause trouble in the VoIP software that was trying to parse them. Here, well-formed packets would sometimes cause trouble in an ethernet controller that shouldn't even be trying to parse them.

Re: Packets of Death

#24

Post by jerdfelt »

I ran into a similar problem with an Intel motherboard about 10 years ago.

We had problems when some NFS traffic would end up getting stalled. Our NFS server would use UDP packets larger than the MTU and they would end up getting fragmented.

Turns out the NIC would not look at the fragmentation headers of the IP packet and always assume a UDP header was present. From time to time, the payload of the NFS packet would have user data that matched the UDP port number the NIC would scan for to determine if the packet should be forwarded to the BMC. This motherboard had no BMC but it was configured as if it did have one.

It would time out after a second or so but in the meantime drop a bunch of packets. The NFS server would retransmit the packet but since the payload didn't change, the NIC would reliably drop the rest of the fragments of the packet.

Of course Intel claimed it wasn't their bug ("it's a bug in the Linux NFS implementation") but they quickly changed their tune when I coded up a sample program that would send one packet a second and reliably cause the NIC to drop 99% of packets received.

While it turned out to be a fairly lame implementation problem on Intel's part (both by ignoring the fragmentation headers and the poor implementation of the motherboard) I have to say it was very satisfying to solve the mystery.

Re: Packets of Death

#25

Post by EvanAnderson »

thechut wrote: post #22
jacquesm wrote: post #20

Persistent bugger. "With a modified HTTP server configured to generate the data at byte value (based on headers, host, etc) you could easily configure an HTTP 200 response to contain the packet of death - and kill client machines behind firewalls!" That's worrisome, I'll bet there are lots of not-so-nice guys trying to figure out a way to do just that. There must be tons of server hardware out there with these cards…

I read the whole thing and that is the line that stuck out most to me. This could very scary. It could be used to bring down a webserver

It sounds like the vulnerability could be used to bring down any machine you can send an arbitrary Ethernet frame to. (I immediately wonder if it works for broadcast frames? Sounds like a way to take down a LAN full of machines quickly if it does.)

Edit: Per http://www.kriskinc.com/intel-pod it does work on broadcast frames. Yikes!

Re: Packets of Death

#28

Post by EvanAnderson »

jerdfelt wrote: post #24

I ran into a similar problem with an Intel motherboard about 10 years ago. We had problems when some NFS traffic would end up getting stalled. Our NFS server would use UDP packets larger than the MTU and they would end up getting fragmented. Turns out the NIC would not look at the fragmentation headers of the IP packet and always assume a UDP header was present. From time to time, the payload of the NFS packet would…

Reading about the OP's issue got me to a doc from Intel (http://www.intel.com/content/dam/doc/application-note/sideba...) re: the "NC Sideband Interface", which sounds like the place where the bug that bit you "lives". Reading over that doc made me shudder a few times, thinking about the complexity and, thus, potential bugs that could be lurking there. I wonder if the OP's bug was related, too.

Having the NIC inspecting incoming frames and potentially diverting them to the management controller sounds like a scary proposition. I'd almost rather just have dedicated Ethernet hardware for the management controller. The decrease in switch ports needed is certainly seductive, but I wonder if it's worth the risk.

(Do you happen to recall which Intel motherboard this bit you on? I was just getting out of whitebox Intel motherboard-based server builds about the time you're describing, but I'm just curious if only for the nostalgia.)

Re: Packets of Death

#29

Post by quentusrex »

It appears to work if you send the packet to the network broadcast address. Quick way to detect if any of the machines are vulnerable(they won't respond to the second ping).

Re: Packets of Death

#30

Post by fusiongyro »

Jabbles wrote: post #21
wglb wrote: post #10

Very good detective work. However, a small suggestion, given: I’ve been working with networks for over 15 years and I’ve never seen anything like this. I doubt I’ll ever see anything like it again. This is a very excellent case for fuzz testing. My thinking is that you want to whip up your Ruby and your EventMachine and Redis going and run a constant fuzz with all sorts of packets in your pre-shipping lab. The idea i…

Surely that's the manufacturer's job. Since it's caused by a specific byte at a specific place, surely you'd only need to fuzz an average of 256 packets (of the required length) to find it... which suggests it wasn't done at all... zero...

It is, but your clients are going to complain to you.
Post reply on HN