Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Packets of Death

blog.krisk.org

Post reply on HN
Print view View original

51–60 of 119 posts

Re: Packets of Death

#51

Post by belorn »

moconnor wrote: post #39
brazzy wrote: post #36

Earlier quoted context omitted.

> the disbelief many show when hearing a software message can bring down a NIC. Shouldn't be a surprise to anyone. Firmware is just software, and it necessarily deals with raw bytes. Not really surprising that it can contain bugs that are triggered by certain byte patterns.

In this light, the surprising thing is that we don't hear about firmware bugs more often. Have manufacturers have been doing a great job of keeping things simple and exhaustively testing, driven by the fear of the losses an incident and recall would cause?

Rare bugs on released hardware are almost by nature ignored in favor of new products. Its also why fuzzing is such as great method in finding security bugs.

Re: Packets of Death

#52

Post by homosaur »

jerdfelt wrote: post #31
meshko wrote: post #18

I have mixed feelings about the write up. I think it gets clear pretty early on that the issue is in the NIC hardware at which point it is time to stop wasting your time investigating problem you can't fix and start contacting the vendor.

In my experience dealing with a similar bug (see my other post in the thread), the vendors will immediately assume it's not their problem. They spent a long time "showing" us that a different version of the Linux kernel didn't exhibit the problem so it must be a Linux kernel bug. Turned out the different version just sent data differently so it didn't trigger the same bug with the same data. Other data would have tri…

THIS.

It's always someone else's testing procedures, someone's else's hardware... The thing is though, most of the time it is. Tech support at the lower levels especially are used to dealing with people who have bad configurations are are using the products incorrectly. The annoyance comes in when you as a customer narrow a problem down but can't get anyone on the phone who can help you at that level.

Re: Packets of Death

#53

Post by laughinghan »

wglb wrote: post #10

Very good detective work. However, a small suggestion, given: I’ve been working with networks for over 15 years and I’ve never seen anything like this. I doubt I’ll ever see anything like it again. This is a very excellent case for fuzz testing. My thinking is that you want to whip up your Ruby and your EventMachine and Redis going and run a constant fuzz with all sorts of packets in your pre-shipping lab. The idea i…

Fuzz testing would have been very unlikely to help since any byte value at that position besides ASCII 1, 2, or 3 "inoculated" the NIC from the bug. There is a very excellent case to be made for fuzz testing, but this isn't it.

Given that, for all we know the relevant parties did conduct extensive fuzz testing and your condescension is misplaced.

Re: Packets of Death

#54

Post by jacquesm »

engtech wrote: post #6

As someone who works with FPGAs/ASICs, this isn't that weird. Everything gets serialized/deserialized these days, so there's all kinds of boundary conditions where you can flip just the right bit and get the data to be deserialized the wrong way. What's more interesting is that it bypasses all of the checks to prevent this from happening. Here is the wiki page on the INVITE OF DEATH which sounds like the problem you…

It's the payload that triggers the bug, not a header!

Definitely not this bug, the one linked is not intel specific.

Re: Packets of Death

#55

Post by baq »

quentusrex wrote: post #29

It appears to work if you send the packet to the network broadcast address. Quick way to detect if any of the machines are vulnerable(they won't respond to the second ping).

you conveniently omitted the part in which you walk to the racks and reboot them all.

Re: Packets of Death

#56

Post by eps »

altcognito wrote: post #50
huhtenberg wrote: post #14

Earlier quoted context omitted.

That was an OS-level bug, it's far less exciting.

I'll agree it's more interesting in that the end-result was a box that required a hard boot, but still these two issues aren't that distantly related: it effected routers and many, many OS platforms, so it's not as if it was related to some implementation detail that MS left out of Windows. Correct me if I'm wrong (no, seriously) -- aren't both "packets of death" just poor handling of said malformed packets? Violatio…

You probably want to re-read the linked article a bit more closely.

Re: Packets of Death

#57

Post by bigiain »

huhtenberg wrote: post #13
engtech wrote: post #6

As someone who works with FPGAs/ASICs, this isn't that weird. Everything gets serialized/deserialized these days, so there's all kinds of boundary conditions where you can flip just the right bit and get the data to be deserialized the wrong way. What's more interesting is that it bypasses all of the checks to prevent this from happening. Here is the wiki page on the INVITE OF DEATH which sounds like the problem you…

> Everything gets serialized/deserialized these days, ... and get the data to be deserialized the wrong way. Can you elaborate? I recognize the words, but not the meaning.

Anybody else waiting for him to reply with something like:

"Oh yeah, I used to work at Intel - that nic's got a YAML parser in it"…

Re: Packets of Death

#58

Post by bigiain »

IheartApplesDix wrote: post #48
engtech wrote: post #6

As someone who works with FPGAs/ASICs, this isn't that weird. Everything gets serialized/deserialized these days, so there's all kinds of boundary conditions where you can flip just the right bit and get the data to be deserialized the wrong way. What's more interesting is that it bypasses all of the checks to prevent this from happening. Here is the wiki page on the INVITE OF DEATH which sounds like the problem you…

Well, it's very weird if you understand anything about Network protocols. It's a layer of complexity that shouldn't be being touched by your NIC, so there shouldn't be a bug there because there shouldn't be code there.

I wonder what _other_ data coming down the wire that nic is monitoring and executing code in response to?

Re: Packets of Death

#59

Post by DuskStar »

arnsholt wrote: post #38
Jabbles wrote: post #21

Earlier quoted context omitted.

Surely that's the manufacturer's job. Since it's caused by a specific byte at a specific place, surely you'd only need to fuzz an average of 256 packets (of the required length) to find it... which suggests it wasn't done at all... zero...

That's assuming you know the magical position. If you need to test all positions, it's 256 to the power of the number of bytes in the message.

But couldn't you test all the positions at once, by having the entire packet be random data? Unless the packet has to be a particular length for this to happen, which I didn't notice in the article.

Re: Packets of Death

#60

Post by jacquesm »

samstave wrote: post #46
jacquesm wrote: post #42

Earlier quoted context omitted.

Just to be on the safe side I just checked all my machines, the majority have broadcom cards, the three that have intel cards are all another type. I think I'll sleep soundly tonight but I felt compelled to check.

That was a very responsible thing to do. :)

Professional paranoia. Stuff like this is really no fun at all.

off-topic:

A long time ago we found that a certain ping packet would be dropped with about 30%, which in turn triggered a monitoring system to register 'server down' when enough packets in a row were missed.

This would happen about once every day or so, leading to an operator being paged (usually at 3am). Very annoying problem and incredibly hard to debug. We'd replaced just about every piece of hardware except for a stupid little T-connector. My buddy Jasper and me looked at it and we both more or less at the same time said 'it can't be'. We swapped out the T-connector, problems solved.

It took the better part of a day to nail that one, I still remember the hostname (chopper) of the SGI box that the thing was connected to (SGI Challenge, an Indy sold as a server with one of those silly thinnet adapters dangling off the back, even though it had a UTP connector too).

Some bugs... I can't say I'm mourning the demise of coaxial ethernet and the bus topology.

Post reply on HN