Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Packets of Death

blog.krisk.org

Post reply on HN
Print view View original

31–40 of 119 posts

Re: Packets of Death

#31

Post by jerdfelt »

meshko wrote: post #18

I have mixed feelings about the write up. I think it gets clear pretty early on that the issue is in the NIC hardware at which point it is time to stop wasting your time investigating problem you can't fix and start contacting the vendor.

In my experience dealing with a similar bug (see my other post in the thread), the vendors will immediately assume it's not their problem.

They spent a long time "showing" us that a different version of the Linux kernel didn't exhibit the problem so it must be a Linux kernel bug. Turned out the different version just sent data differently so it didn't trigger the same bug with the same data. Other data would have triggered it.

I wouldn't be surprised if the majority of "bugs" they receive reports on turn out to not be bugs in their hardware. There's probably parallels with the reports of compiler bugs, most end up not being bugs in the compiler.

The unfortunate truth is that responsibility of proving it's the vendors bug falls on the customer.

I had to write a proof-of-concept "exploit" to show the problem was with their hardware, effectively troubleshooting most of the problem for them.

Re: Packets of Death

#32

Post by [deleted] »

Jabbles wrote: post #21
wglb wrote: post #10

Very good detective work. However, a small suggestion, given: I’ve been working with networks for over 15 years and I’ve never seen anything like this. I doubt I’ll ever see anything like it again. This is a very excellent case for fuzz testing. My thinking is that you want to whip up your Ruby and your EventMachine and Redis going and run a constant fuzz with all sorts of packets in your pre-shipping lab. The idea i…

Surely that's the manufacturer's job. Since it's caused by a specific byte at a specific place, surely you'd only need to fuzz an average of 256 packets (of the required length) to find it... which suggests it wasn't done at all... zero...

[deleted]

Re: Packets of Death

#33

Post by cdvonstinkpot »

jws wrote: post #9

Well this hurts. I have a critical machine with a dual NIC Intel motherboard. I had to abandon the 82579LM port because of unresolved bugs in the Linux drivers, and the other one is a 82574L, the one documented in this post. I suppose I can send just the right ICMP echo packet to router to make it send me back an innoculating frame.

Good luck

Re: Packets of Death

#34

Post by EvanAnderson »

meshko wrote: post #18

I have mixed feelings about the write up. I think it gets clear pretty early on that the issue is in the NIC hardware at which point it is time to stop wasting your time investigating problem you can't fix and start contacting the vendor.

The write-up gave me the feeling that the OP did just that. My experience has been that manufacturers will blow you off unless you can provide a reproducible test case. Certainly, his company doesn't sound like it has the volume necessary to threaten Intel w/ moving to a different NIC manufacturer unless the problem is resolved. With that in mind, I think the amount of work he did was just the right amount.

Re: Packets of Death

#35

Post by wglb »

Jabbles wrote: post #21
wglb wrote: post #10

Very good detective work. However, a small suggestion, given: I’ve been working with networks for over 15 years and I’ve never seen anything like this. I doubt I’ll ever see anything like it again. This is a very excellent case for fuzz testing. My thinking is that you want to whip up your Ruby and your EventMachine and Redis going and run a constant fuzz with all sorts of packets in your pre-shipping lab. The idea i…

Surely that's the manufacturer's job. Since it's caused by a specific byte at a specific place, surely you'd only need to fuzz an average of 256 packets (of the required length) to find it... which suggests it wasn't done at all... zero...

Well, my feeling is that if you are going to ship it, you better fuzz it. And if you are assembling something with this card, you are the manufacturer.

A true fuzzing run for something like this ought to run in the tens of millions of packets.

Re: Packets of Death

#36

Post by brazzy »

guylhem wrote: post #15

That is great HN content! Debugging deep down the rabbit hole, until you find a bug in the NIC EEPROM - and the disbelief many show when hearing a software message can bring down a NIC. I for one would enjoy reading more content like this on HN that what qualifies as best as a friday-night hack

> the disbelief many show when hearing a software message can bring down a NIC.

Shouldn't be a surprise to anyone. Firmware is just software, and it necessarily deals with raw bytes. Not really surprising that it can contain bugs that are triggered by certain byte patterns.

Re: Packets of Death

#37

Post by samstave »

EvanAnderson wrote: post #25
thechut wrote: post #22

Earlier quoted context omitted.

I read the whole thing and that is the line that stuck out most to me. This could very scary. It could be used to bring down a webserver

It sounds like the vulnerability could be used to bring down any machine you can send an arbitrary Ethernet frame to. (I immediately wonder if it works for broadcast frames? Sounds like a way to take down a LAN full of machines quickly if it does.) Edit: Per http://www.kriskinc.com/intel-pod it does work on broadcast frames. Yikes!

Heh, yeah and he pretty much gives all the info on how to create the attack.

The test would be to find out how common that particular NIC is out there - and grab a few and test out his method.

Looks like it would be fairly trivial to setup duplication given the author did all the heavy lifting in finding this. Putting a small AWS bot-net up that just sweeps massive IP blocks would be easy - heck you could do it really easily from a single machine, it would seem.

If you can take out a machine with a single packet....

Re: Packets of Death

#38

Post by arnsholt »

Jabbles wrote: post #21
wglb wrote: post #10

Very good detective work. However, a small suggestion, given: I’ve been working with networks for over 15 years and I’ve never seen anything like this. I doubt I’ll ever see anything like it again. This is a very excellent case for fuzz testing. My thinking is that you want to whip up your Ruby and your EventMachine and Redis going and run a constant fuzz with all sorts of packets in your pre-shipping lab. The idea i…

Surely that's the manufacturer's job. Since it's caused by a specific byte at a specific place, surely you'd only need to fuzz an average of 256 packets (of the required length) to find it... which suggests it wasn't done at all... zero...

That's assuming you know the magical position. If you need to test all positions, it's 256 to the power of the number of bytes in the message.

Re: Packets of Death

#39

Post by moconnor »

brazzy wrote: post #36
guylhem wrote: post #15

That is great HN content! Debugging deep down the rabbit hole, until you find a bug in the NIC EEPROM - and the disbelief many show when hearing a software message can bring down a NIC. I for one would enjoy reading more content like this on HN that what qualifies as best as a friday-night hack

> the disbelief many show when hearing a software message can bring down a NIC. Shouldn't be a surprise to anyone. Firmware is just software, and it necessarily deals with raw bytes. Not really surprising that it can contain bugs that are triggered by certain byte patterns.

In this light, the surprising thing is that we don't hear about firmware bugs more often. Have manufacturers have been doing a great job of keeping things simple and exhaustively testing, driven by the fear of the losses an incident and recall would cause?

Re: Packets of Death

#40

Post by voidlogic »

brazzy wrote: post #36
guylhem wrote: post #15

That is great HN content! Debugging deep down the rabbit hole, until you find a bug in the NIC EEPROM - and the disbelief many show when hearing a software message can bring down a NIC. I for one would enjoy reading more content like this on HN that what qualifies as best as a friday-night hack

> the disbelief many show when hearing a software message can bring down a NIC. Shouldn't be a surprise to anyone. Firmware is just software, and it necessarily deals with raw bytes. Not really surprising that it can contain bugs that are triggered by certain byte patterns.

It would be interesting to see what the actual error in the NIC firmware source was.

This invalidates my assumption that a shop like Intel probably uses formal verification in firmware development.

Its also scary to consider how many very important (nuclear/damn control, etc) systems while themselves might be formally verified are dependent on services of lower level software (OS, drivers) and hardware (firmware) that are not...

Post reply on HN