Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

"Intel Packet of Death" not Intel's problem

h-online.com

Post reply on HN
Print view View original

21–30 of 33 posts

Re: "Intel Packet of Death" not Intel's problem

#21

Post by kkielhofner »

GiHe wrote: post #19

Cross-posted at h-online.com: I have a plurality of systems with Intel motherboards which demonstrate the same kind of problems. The motherboards in question have two Intel ethernet controllers, one of which is an 82574L. The systems connect to two different networks. When the systems attach to one of the networks (but not the other) using the 82574L interface (but not the other), that interface dies after some unpre…

I've been thinking about your situation and I'm reminded of my own words: "there is always a reason".

Would you be able to e-mail me (CAPTCHA here):

http://tinyurl.com/66srzt

I'd like to discuss your issue further. Thanks!

Re: "Intel Packet of Death" not Intel's problem

#22

Post by ajross »

ersii wrote: post #5
brownbat wrote: post #3

Company from Taipei flashes some Intel equipment, then it appears to function correctly, but can be bricked remotely with a specially crafted incoming packet. Company has US branch that's a government contractor: http://government-contractor.bizdirlib.com/ceo/Synertron_Tec... Charming.

I think you're reading way more into this, than there is to it. Taiwan (Republic of China) is by the way, basically it's own country with it's own leadership and currency. I find it somewhat hard to put China (People's Republic of China) and Taiwan (Republic of China) together.

Well, I suppose the argument would go that a Taiwanese engineer would, by dint of shared language and culture, be more susceptible to coercion and bribes than someone from europe, south asia, etc...

And I agree that you don't want to be too paranoid about this stuff. But at the same time, if you were expecting and looking for an "illicit backdoor" in hardware, this is exactly the kind of thing you'd expect. Firmware in a place virtually no one knows about gets modified on a per-product basis to do nefarious things. And this is exactly how you'd expect such a modification to be discovered, by accidentally introducing a bug that distinguishes itself from the clean parent.

I mean, I'm not screaming "spy" here, but if I were to have read this story in a techno-thriller novel I'd be writing a post applauding the author for her excellently researched and eminently plausible plot hook.

Re: "Intel Packet of Death" not Intel's problem

#23

Post by noonespecial »

kkielhofner wrote: post #18
jessaustin wrote: post #14

Earlier quoted context omitted.

I am impressed by his original troubleshooting, but this followup seems impractical. Of his three suggestions, only the third (Intel providing improved board testing tools) even seems like it could possibly prevent this sort of problem. Asking for hardware-enforced "sane" behavior is like asking, "why doesn't my computer know I don't want my program to deadlock, segfault, or loop indefinitely?" That is, if the contro…

Kristian Kielhofner here - While I understand your analogy I don't think it's an accurate one. In fact, with the release of the successor to the 82574 Intel has already implemented some of the things I suggested: http://communities.intel.com/community/wired/blog/2012/10/18... Clearly they have learned from the various EEPROM issues on previous controllers (including the 82574) and implemented (among other things) EEP…

Hey Kris. Welcome to HN. I remember you from Astricons of years past and used astlinux on alix a lot over the years. Good stuff.

I didn't realize you guys were so close. I live just off Televast, about a mile and a half from Star2Star.

Re: "Intel Packet of Death" not Intel's problem

#24

Post by sliverstorm »

ajross wrote: post #22
ersii wrote: post #5

Earlier quoted context omitted.

I think you're reading way more into this, than there is to it. Taiwan (Republic of China) is by the way, basically it's own country with it's own leadership and currency. I find it somewhat hard to put China (People's Republic of China) and Taiwan (Republic of China) together.

Well, I suppose the argument would go that a Taiwanese engineer would, by dint of shared language and culture, be more susceptible to coercion and bribes than someone from europe, south asia, etc... And I agree that you don't want to be too paranoid about this stuff. But at the same time, if you were expecting and looking for an "illicit backdoor" in hardware, this is exactly the kind of thing you'd expect. Firmware…

Well, I suppose the argument would go that a Taiwanese engineer would, by dint of shared language and culture, be more susceptible to coercion and bribes than someone from europe, south asia, etc...

You could try to make that argument, but I've always figured it would go kind of the other direction. Every Taiwanese national I know is a fierce supporter of Taiwan's independence of China, and China certainly does all it can to foster that every time it tries to annex the country.

Re: "Intel Packet of Death" not Intel's problem

#25

Post by GiHe »

kkielhofner wrote: post #20
GiHe wrote: post #19

Cross-posted at h-online.com: I have a plurality of systems with Intel motherboards which demonstrate the same kind of problems. The motherboards in question have two Intel ethernet controllers, one of which is an 82574L. The systems connect to two different networks. When the systems attach to one of the networks (but not the other) using the 82574L interface (but not the other), that interface dies after some unpre…

This. As I say in my updated post, this is a complex issue with clear combinatorial factors. More than likely it's not limited to one chip, one packet, or one EEPROM configuration. A quick reading of the web shows various unexplained issues with this family of Intel ethernet controllers randomly exhibiting the exact behavior I've described. Different controllers, different mobo OEMs, different EEPROM settings. Are al…

I have another interesting (and reproducible) manifestation on Supermicro motherboards with two 82574L controllers. In this case, it is again true that we only experience problems on the first (as ordered by ascending MAC address) of the two interfaces.

That was the case, though I did not clearly state so above, on the Intel motherboard with one 82574L and one 82579LM.

Re: "Intel Packet of Death" not Intel's problem

#26

Post by samstave »

ersii wrote: post #5
brownbat wrote: post #3

Company from Taipei flashes some Intel equipment, then it appears to function correctly, but can be bricked remotely with a specially crafted incoming packet. Company has US branch that's a government contractor: http://government-contractor.bizdirlib.com/ceo/Synertron_Tec... Charming.

I think you're reading way more into this, than there is to it. Taiwan (Republic of China) is by the way, basically it's own country with it's own leadership and currency. I find it somewhat hard to put China (People's Republic of China) and Taiwan (Republic of China) together.

I have commented about this in the past; when I was at lockheed, in 2006, we had security debriefings about chinese hacking attempts.

There were trojans on the network that were sending little data packets back to china... but more interestingly:

Lockheed employees were not allowed to connect their macines to any foreign network. Even of those which were suppliers.

There was a supplier in Taiwan where employees would go and would transfer some files via sneakernet (USB keys) - the supplier had been hacked and the chinese were using the Taiwanese suppliers machines to attack the lockheed employees via the transfer of the USB sticks.

The point is that don't underestimate Chinese hackers and the potntial vectors they are willing to exploit.

Re: "Intel Packet of Death" not Intel's problem

#27

Post by mrb »

fulafel wrote: post #9

Firmware images usually have checksums. Was this an Intel blob suffering from bitrot, or does Intel have some more or less error prone way to build your own FW images for NICs?

It is not the firmware that was corrupted. It is the EEPROM that was incorrectly programmed.

The EEPROM is typically 4kB on the 82754. When it is reprogrammed either by the end-user (eg. via ethtool(1)) or by the manufacturer, the programming procedure recomputes a checksum on the first 128 bytes IIRC (when reprogramming via ethtool, the kernel driver e1000e is responisble for automatically updating the checksum.

So all in all, no, the packet of death issue was not caused by bitrot.

Re: "Intel Packet of Death" not Intel's problem

#28

Post by jessaustin »

kkielhofner wrote: post #18
jessaustin wrote: post #14

Earlier quoted context omitted.

I am impressed by his original troubleshooting, but this followup seems impractical. Of his three suggestions, only the third (Intel providing improved board testing tools) even seems like it could possibly prevent this sort of problem. Asking for hardware-enforced "sane" behavior is like asking, "why doesn't my computer know I don't want my program to deadlock, segfault, or loop indefinitely?" That is, if the contro…

Kristian Kielhofner here - While I understand your analogy I don't think it's an accurate one. In fact, with the release of the successor to the 82574 Intel has already implemented some of the things I suggested: http://communities.intel.com/community/wired/blog/2012/10/18... Clearly they have learned from the various EEPROM issues on previous controllers (including the 82574) and implemented (among other things) EEP…

Let me preface this by saying that your epic trouble-shooting effort was really cool. That's what inspires me to pay so much attention to this.

If you'll excuse my ignorance, could you identify which points made on the linked page correspond to your suggestions? I can see how signature checking, if there is in fact such a mechanism on the controller, can help ensure that an EEPROM image is a member of a particular favored set of such images, but you'll admit that that's a less general approach than "in-hardware sane behavior". I don't know anything about µC design, but it would surprise me if the mistake here were as simple as setting a "die when you see this particular byte sequence" bit. It seems more likely that the behavior is an emergent property based on a combination of flags and coded behavior. I still don't think it would be possible for the controller to prevent that result in general. It is possible to test for bad behavior, as your customers proved. It's also possible for drivers to correctly handle the bad behavior of their hardware, and I'm sure appropriate patches are welcome.

Did your board vendor inform you of Intel's findings back in October? If so, could your original article have been a bit more explicit about the fact that Intel wasn't responsible for this? If not, are you looking for another board vendor?

Re: "Intel Packet of Death" not Intel's problem

#29

Post by specialist »

ersii wrote: post #5
brownbat wrote: post #3

Company from Taipei flashes some Intel equipment, then it appears to function correctly, but can be bricked remotely with a specially crafted incoming packet. Company has US branch that's a government contractor: http://government-contractor.bizdirlib.com/ceo/Synertron_Tec... Charming.

I think you're reading way more into this, than there is to it. Taiwan (Republic of China) is by the way, basically it's own country with it's own leadership and currency. I find it somewhat hard to put China (People's Republic of China) and Taiwan (Republic of China) together.

Trust, but verify. With everything that we know, it's silly to not be paranoid.

I got pretty excited about election integrity for a while.

The default position of the defenders of the status quo was "I can't believe you don't trust us. Prove there's something wrong. You 'experts' in computers, security, and elections are just a bunch of conspiracy freaks."

My default position is "show me". That skepticism merely makes me an informed consumer.

Re: "Intel Packet of Death" not Intel's problem

#30

Post by jessaustin »

viraptor wrote: post #16
jessaustin wrote: post #14

Earlier quoted context omitted.

I am impressed by his original troubleshooting, but this followup seems impractical. Of his three suggestions, only the third (Intel providing improved board testing tools) even seems like it could possibly prevent this sort of problem. Asking for hardware-enforced "sane" behavior is like asking, "why doesn't my computer know I don't want my program to deadlock, segfault, or loop indefinitely?" That is, if the contro…

> "why doesn't my computer know I don't want my program to deadlock, segfault, or loop indefinitely?" For a device which can be checked from outside (second subsystem for device self-monitoring), this is actually possible to implement and fairly common. Watchdogs are often implemented to restart automatically when the device is completely unresponsive.

I could imagine a board manufacturer doing something like this for an expensive, low-volume NIC. I still don't see how Intel could help.
Post reply on HN