Hunting deep-seeded kernel bugs • Hacker News Board

Re: Hunting deep-seeded kernel bugs

#11

Post by pmjordan » Wed, Mar 28, 2012, 12:51 PM UTC

munin wrote: post #9

> MegaCli is still the ultimate cause strongly disagree! no kernel component should introduce memory corruption, and especially not in response to an ioctl from any user mode code. they say: > Specifically, various offsets provided from userspace are used by the driver without any checks. If these offsets are maliciously chosen, the driver can be induced to write to arbitrary kernel memory sure hope they patched that…

sure hope they patched that too!

The patch appears to discard the DMA flags coming from user space, and instead applies the ones that are correct given the kernel's knowledge of the buffer.

But yes, this is ultimately a kernel bug - specifically one of the driver's userspace API.

Re: Hunting deep-seeded kernel bugs

#12

Post by rwmj » Wed, Mar 28, 2012, 1:18 PM UTC

munin wrote: post #9

> MegaCli is still the ultimate cause strongly disagree! no kernel component should introduce memory corruption, and especially not in response to an ioctl from any user mode code. they say: > Specifically, various offsets provided from userspace are used by the driver without any checks. If these offsets are maliciously chosen, the driver can be induced to write to arbitrary kernel memory sure hope they patched that…

Bugs happen though.

And these sort are old. I wrote a super [root] exploit for OS-9 which worked by tricking the GETSTAT call [equivalent of ioctl] for some driver to write its result to a random, unchecked place in memory. That was in 1992 / OS-9 2.4.

Re: Hunting deep-seeded kernel bugs

#13

Post by davidwtbuxton » Wed, Mar 28, 2012, 1:28 PM UTC

topbanana wrote: post #7

rquantz wrote: post #6

Earlier quoted context omitted.

Ever since I learned the name for these kinds of mistakes, I can't help but call it out every time I see them: eggcorn! http://eggcorns.lascribe.net/english/46/seed/

This term was coined by Adam and Joe. You can listen to their their podcast here: http://www.bbc.co.uk/podcasts/series/adamandjoe (and I recommend that you do)

As great as Adam and Joe are, the term pre-dates their podcasts. I always thought it was coined on the language log, http://itre.cis.upenn.edu/~myl/languagelog/archives/000018.h...

Re: Hunting deep-seeded kernel bugs

#14

Post by jilebedev » Wed, Mar 28, 2012, 4:24 PM UTC

>MegaCli specifies the address in one format, but sets a flag indicating that it’s in another

Far as I can discern (I'm not a kernel hacker) that is the root cause. Can anyone with static code analysis experience chime in on whether it's possible to detect that kind of error by examining the source?

Re: Hunting deep-seeded kernel bugs

#15

Post by munin » Wed, Mar 28, 2012, 6:01 PM UTC

rwmj wrote: post #12

munin wrote: post #9

> MegaCli is still the ultimate cause strongly disagree! no kernel component should introduce memory corruption, and especially not in response to an ioctl from any user mode code. they say: > Specifically, various offsets provided from userspace are used by the driver without any checks. If these offsets are maliciously chosen, the driver can be induced to write to arbitrary kernel memory sure hope they patched that…

Bugs happen though. And these sort are old. I wrote a super [root] exploit for OS-9 which worked by tricking the GETSTAT call [equivalent of ioctl] for some driver to write its result to a random, unchecked place in memory. That was in 1992 / OS-9 2.4.

bugs DO happen.

the root cause is in the kernel component that deals with memory inappropriately though. user space should be free to blast whatever into kernel space via system calls and ioctl and rely on the operating system to not crash, and also for the kernel to continue enforcing the rules of the operating system.

this is like saying "the root cause is Dan, who typed in a `drop tables. our solution is to take away Dans keyboard"

Re: Hunting deep-seeded kernel bugs

#16

Post by kezza » Wed, Mar 28, 2012, 7:43 PM UTC

pmjordan wrote: post #4

jwildeboer wrote: post #3

They claim to use RHEL but instead of asking for support from Red Hat they decide to try themselves? Or am I missing something? Wouldn't such a patch be beneficial to other RHEL users too? (disclaimer: I work at Red Hat)

Well, they have pushed the fix upstream and it's been merged into mainline. I assume you guys have people monitoring the subsystems for backportable bugfixes? I have no doubt that you guys have great support, but since the MegaCLI utility that triggers the issue is a closed source third-party piece of software, I could imagine the process of getting Red Hat engineers to even reproduce the problem would probably take…

For what it is worth, It was reported to RedHat, but our experiences with them have proved to be fairly.. hmm.. lackluster.

We've talked about it before http://www.anchor.com.au/blog/2009/02/the-value-of-commercia...

Re: Hunting deep-seeded kernel bugs

#17

Post by spydum » Thu, Mar 29, 2012, 1:14 AM UTC

Enjoyed the read; hats off to you guys for digging in and finding/resolving and pushing the bugfix upstream.

I do have to wonder: did this bug all of a sudden spring up on you? Seems like the potential would have always been there on this platform. Could a change management process have lead you to find that "adding megaCLI monitoring to all KVM hosts" or "upgrading to megacli 8.01.06" was the cause of the problem? Just curious if you perceive a better CM or system inventory could have allowed you to deduce the problem area with a bit more efficiency?