Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Packets of Death

blog.krisk.org

Post reply on HN
Print view View original

61–70 of 119 posts

Re: Packets of Death

#61

Post by bigiain »

jevinskie wrote: post #47
EvanAnderson wrote: post #28

Earlier quoted context omitted.

Reading about the OP's issue got me to a doc from Intel ( http://www.intel.com/content/dam/doc/application-note/sideba... ) re: the "NC Sideband Interface", which sounds like the place where the bug that bit you "lives". Reading over that doc made me shudder a few times, thinking about the complexity and, thus, potential bugs that could be lurking there. I wonder if the OP's bug was related, too. Having the NIC inspe…

"IPMI operates independently of the OS and allows administrators to manage a system remotely even without an OS, system management software, and even if the monitored system is powered off (along as it is connected to a power source). IPMI can also function after an OS has started, offering enhanced features when used with system management software." Yikes! Sounds like system management mode in a BIOS!

"Yikes! Sounds like system management mode in a BIOS!"

Or, "Sounds like an advanced persistent hardware rootkit!"

Re: Packets of Death

#62

Post by ajross »

brazzy wrote: post #36
guylhem wrote: post #15

That is great HN content! Debugging deep down the rabbit hole, until you find a bug in the NIC EEPROM - and the disbelief many show when hearing a software message can bring down a NIC. I for one would enjoy reading more content like this on HN that what qualifies as best as a friday-night hack

> the disbelief many show when hearing a software message can bring down a NIC. Shouldn't be a surprise to anyone. Firmware is just software, and it necessarily deals with raw bytes. Not really surprising that it can contain bugs that are triggered by certain byte patterns.

Indeed. But unlike "just software", firmware is never, ever fixable in source form except deep within the bowels of hardware manufacturers. Even when found, firmware bugs are generally ignored. I wonder if anyone has checked: has the fixed image been released to linux-firmware, or the Windows driver yet? Will it ever?

Just browse through the ./drivers tree of the kernel source some time and look at all the "quirks" and "workarounds" there. Recognize that virtually all of those could have been fixed in firmware, but weren't because no one cared.

It's just deeply depressing. This was a virtuoso debugging performance, but it didn't have to be that way if hardware companies were sane. But they aren't, and we all pay for it.

(edit: I just checked for myself, the Linux e1000e driver apparently doesn't support runtime firmware update via the kernel API, so linux-firmware wouldn't be expected to have this. I don't know what the process is for an affected end user to get a copy. I suspect there is none.)

Re: Packets of Death

#63

Post by TapaJob »

Fantastic Article, Fantastic fine. Well done.

As a telecoms engineer predominantly selling Asterisk for the last 4 years and Asterisk experiance extending back to 2006 it's shocking to see this finally put right. For so many years, I have avoided the e1000 Intel controllers after a very public/embarassing situation when a conferencing server behaved in a wierd manner disrupting core services. Not having the expertise the author has, I narrowed it down to the Eth. Controller, Immediately replaced the server with IBM Hardware with Broadcom chipset and resumed our services in providing conferencing to some of the top FTSE100 companies.

Following this episode, I spend numerous days diagnosing the chipset with many conference calls with Digium engineers debugging the server remotely. In the end, no solution, recommendation to avoid the e1000 chipset and moved on.

Re: Packets of Death

#64

Post by astangl »

This seems much more serious than the much-ballyhooed Pentium FDIV bug. Hopefully Intel will be on the ball with notifying people and distributing the fix.

Re: Packets of Death

#65

Post by TapaJob »

TapaJob wrote: post #63

Fantastic Article, Fantastic fine. Well done. As a telecoms engineer predominantly selling Asterisk for the last 4 years and Asterisk experiance extending back to 2006 it's shocking to see this finally put right. For so many years, I have avoided the e1000 Intel controllers after a very public/embarassing situation when a conferencing server behaved in a wierd manner disrupting core services. Not having the expertise…

brings back memories....

http://lists.debian.org/debian-isp/2009/06/msg00018.html

Re: Packets of Death

#66

Post by samstave »

jacquesm wrote: post #60
samstave wrote: post #46

Earlier quoted context omitted.

That was a very responsible thing to do. :)

Professional paranoia. Stuff like this is really no fun at all. off-topic: A long time ago we found that a certain ping packet would be dropped with about 30%, which in turn triggered a monitoring system to register 'server down' when enough packets in a row were missed. This would happen about once every day or so, leading to an operator being paged (usually at 3am). Very annoying problem and incredibly hard to debu…

Heh, I have had experiences like that with T-connectors.

Heck, even just a few months ago I spent 2 hours troubleshooting a 10G fiber connection on all brand new gear before swapping out the brand-new cisco 10G SR SFP module which was DOA.

Thinking the same exact thing after swapping out everything else, including all patch cords on both ends "It can't possible be this SFP module"

Yup.

Re: Packets of Death

#68

Post by ChuckMcM »

Makes me wonder if this is related to in-band management? One of the interesting thing about working at NetApp, which had its own "OS" was that every driver was written by engineering. That allowed the full challenge of some of these devices to be experienced first hand.

One of the more painful summers resulted from a QLogic HBA which sometimes, for no apparent reason, injected a string of hex digits into the data it transmitted. There is a commemorative t-shirt of that bug with just the string of characters. It lead NetApp to putting in-block checksums into the file system so that corruption between the disk and memory, which was 'self inflicted' (and so passed various channel integrity checks) could be detected.

Here at Blekko we had a packet fragment that would simply vanish into the center switch. It would go in and never come out. We never got a satisfactory answer for that one. Keith, our chief architect, worked around it by randomizing the packet on a retransmit request.

The amount of code between your data and you that you can't control is, sadly, way larger than you probably would like.

Re: Packets of Death

#69

Post by rosser »

EvanAnderson wrote: post #49
jevinskie wrote: post #47

Earlier quoted context omitted.

"IPMI operates independently of the OS and allows administrators to manage a system remotely even without an OS, system management software, and even if the monitored system is powered off (along as it is connected to a power source). IPMI can also function after an OS has started, offering enhanced features when used with system management software." Yikes! Sounds like system management mode in a BIOS!

It's worse than that. It's not BIOS-- it's a freestanding computer. You'll enjoy this (or be horrified by it): http://fish2.com/ipmi/itrain.html

But it's a freestanding computer that means that I don't need to go to the data center at two in the morning to bring up a box that's kernel panicked.

Yeah, be careful with it. Firewall it silly. But recognize that it's a tool that can be very useful.

Re: Packets of Death

#70

Post by sideproject »

guylhem wrote: post #15

That is great HN content! Debugging deep down the rabbit hole, until you find a bug in the NIC EEPROM - and the disbelief many show when hearing a software message can bring down a NIC. I for one would enjoy reading more content like this on HN that what qualifies as best as a friday-night hack

That is some serious debugging, supported by even more serious persistence by the author. Total respect.
Post reply on HN