Hunting deep-seeded kernel bugs
anchor.com.au
Hunting deep-seeded kernel bugs
1–10 of 17 posts
Re: Hunting deep-seeded kernel bugs
#2It's "deep-seated" (although I don't see what was wrong with just using the original article title).
Re: Hunting deep-seeded kernel bugs
#3They claim to use RHEL but instead of asking for support from Red Hat they decide to try themselves? Or am I missing something? Wouldn't such a patch be beneficial to other RHEL users too? (disclaimer: I work at Red Hat)
Re: Hunting deep-seeded kernel bugs
#4They claim to use RHEL but instead of asking for support from Red Hat they decide to try themselves? Or am I missing something? Wouldn't such a patch be beneficial to other RHEL users too? (disclaimer: I work at Red Hat)
Well, they have pushed the fix upstream and it's been merged into mainline. I assume you guys have people monitoring the subsystems for backportable bugfixes?
I have no doubt that you guys have great support, but since the MegaCLI utility that triggers the issue is a closed source third-party piece of software, I could imagine the process of getting Red Hat engineers to even reproduce the problem would probably take longer than just going in and fixing it themselves. I can imagine having hosts with dozens of VMs going down is a high-priority issue - OK, less so once they discovered they could work around it by disabling monitoring, and eventually by upgrading the monitoring utility, but at that point they were already close to a fix. (also: they didn't say they didn't report it to RH...)
On a more technical note, if the MegaRAID driver only does DMAs to/from 32-bit memory addresses, this could be an artificial performance limitation on systems with lots of RAM (like the 128GiB in the article) as it potentially means waiting for I wonder if a hardware IOMMU could have caught this bug.
Re: Hunting deep-seeded kernel bugs
#5They claim to use RHEL but instead of asking for support from Red Hat they decide to try themselves? Or am I missing something? Wouldn't such a patch be beneficial to other RHEL users too? (disclaimer: I work at Red Hat)
Your comment seems strangely defensive and dismissive. That they're an RHEL customer doesn't make them incapable of being proficient systems engineers, nor of participating in the wider eco-system themselves (and being capable of doing so doesn't devalue their RHEL investment either).
Re: Hunting deep-seeded kernel bugs
#6It's "deep-seated" (although I don't see what was wrong with just using the original article title).
Ever since I learned the name for these kinds of mistakes, I can't help but call it out every time I see them: eggcorn!
Re: Hunting deep-seeded kernel bugs
#7It's "deep-seated" (although I don't see what was wrong with just using the original article title).
Ever since I learned the name for these kinds of mistakes, I can't help but call it out every time I see them: eggcorn! http://eggcorns.lascribe.net/english/46/seed/
This term was coined by Adam and Joe. You can listen to their their podcast here: http://www.bbc.co.uk/podcasts/series/adamandjoe (and I recommend that you do)
Re: Hunting deep-seeded kernel bugs
#8They claim to use RHEL but instead of asking for support from Red Hat they decide to try themselves? Or am I missing something? Wouldn't such a patch be beneficial to other RHEL users too? (disclaimer: I work at Red Hat)
Well, they have pushed the fix upstream and it's been merged into mainline. I assume you guys have people monitoring the subsystems for backportable bugfixes? I have no doubt that you guys have great support, but since the MegaCLI utility that triggers the issue is a closed source third-party piece of software, I could imagine the process of getting Red Hat engineers to even reproduce the problem would probably take…
Lots of poor hardware has limitations like this. It does hurt performance.
Re: Hunting deep-seeded kernel bugs
#9> MegaCli is still the ultimate cause
strongly disagree! no kernel component should introduce memory corruption, and especially not in response to an ioctl from any user mode code. they say:
> Specifically, various offsets provided from userspace are used by the driver without any checks. If these offsets are maliciously chosen, the driver can be induced to write to arbitrary kernel memory
sure hope they patched that too!
Re: Hunting deep-seeded kernel bugs
#10Earlier quoted context omitted.
Well, they have pushed the fix upstream and it's been merged into mainline. I assume you guys have people monitoring the subsystems for backportable bugfixes? I have no doubt that you guys have great support, but since the MegaCLI utility that triggers the issue is a closed source third-party piece of software, I could imagine the process of getting Red Hat engineers to even reproduce the problem would probably take…
Lots of poor hardware has limitations like this. It does hurt performance.
I think it's a requirement of PCIe for devices to support DMA to memory with 64-bit physical addresses.[1] In any case, the hardware clearly supports it in the case - the bug was that a 32-bit I/O vector was being passed off to the hardware claiming to be a 64-bit one, and subsequently interpreted as such.
The situation was certainly bad with classic PCI - I have an nForce5-based motherboard whose PCI bus silently truncates 64-bit DMA buffer addresses to 32 bits. The onboard ethernet chip does in principle support 64-bit DMA, so I was getting all sorts of weird behaviour after upgrading my RAM from 2 to 6 GiB. The southbridge chip in question is now blacklisted for 64-bit DMA in the Linux kernel.
The 32-bit DMA cutoff isn't the only such constraint on buffer physical addresses of course; historically, ISA devices could only DMA to 20-bit addresses IIRC (or was it 24 bits?). I wrote an OS X driver for the virtio paravirtual ethernet adapter a while back[2] and its I/O buffers must be 4K page-aligned (low 12 bits 0), and its high bits are specified by a 32-bit number. This means you can only use buffers with physical addresses which fit in 44 bits. That's 16TiB, which are probably still a way off but could conceivably be reached not too far in the future. Current x86_64 CPUs certainly support 48 bit physical addresses in principle.
[1] That doesn't mean hardware bugs don't exist, but at least it's part of the basic spec now, as opposed to being tagged on later as for PCI.