Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

How is the Linux kernel tested?

embeddedbits.org

Post reply on HN
Print view View original

21–30 of 59 posts

Re: How is the Linux kernel tested?

#21

Post by varjag »

alkonaut wrote: post #8

Why is there such little emphasis on “traditional” testing, that is, regular unit tests? At least some portion of the code base is surely suitable for normal unit tests. For example data structures, scheduling algorithms, file systems, ...

Linux kernel project predates what you call traditional unit testing practices†. Its success in the first decade and a half coupled with pragmatics of hardware testing make the flow what it is now.

† Regression testing was certainly known then, but it was not a dogmatic movement yet.

Re: How is the Linux kernel tested?

#22

Post by thedance »

ndesaulniers wrote: post #19
alkonaut wrote: post #8

Why is there such little emphasis on “traditional” testing, that is, regular unit tests? At least some portion of the code base is surely suitable for normal unit tests. For example data structures, scheduling algorithms, file systems, ...

How do I test a driver for which I do not possess the hardware?

You write the software in such a way that instead of just reading and writing registers or memory you exercise some set of functions. In normal operation you pass the driver a set of real functions that read and write real registers. In testing you pass functions that do other things. This makes it quite easy to exercise the features of the hardware that are rarely seen in the wild. For example most IO adapters and NICs have some kind of signal that they are overheating. Most Linux drivers simply ignore or malfunction when these conditions are raised, because the author of the driver never got a chance to manually exercise that feature.

This is basic design for unit testing but it's impossible in Linux because Linux lacks a zero-cost abstraction that would let you mock out a device. C only has costly abstractions such as tables of function pointers.

Re: How is the Linux kernel tested?

#23

Post by vbezhenar »

ndesaulniers wrote: post #19
alkonaut wrote: post #8

Why is there such little emphasis on “traditional” testing, that is, regular unit tests? At least some portion of the code base is surely suitable for normal unit tests. For example data structures, scheduling algorithms, file systems, ...

How do I test a driver for which I do not possess the hardware?

Write hardware emulation for VM with any behaviour you want to test.

Re: How is the Linux kernel tested?

#24

Post by vbezhenar »

thedance wrote: post #22
ndesaulniers wrote: post #19

Earlier quoted context omitted.

How do I test a driver for which I do not possess the hardware?

You write the software in such a way that instead of just reading and writing registers or memory you exercise some set of functions. In normal operation you pass the driver a set of real functions that read and write real registers. In testing you pass functions that do other things. This makes it quite easy to exercise the features of the hardware that are rarely seen in the wild. For example most IO adapters and N…

You can compile object file in isolation and provide mocked implementations for all imported functions.

Re: How is the Linux kernel tested?

#25

Post by yjftsjthsd-h »

cat199 wrote: post #20
ndesaulniers wrote: post #18

We run our own ci for building Linux kernels with clang. (ClangBuiltLinux.github.io). We take Debian's nightly package of ToT, make a docker image with the minimum tools we need, and use buildroot ramdisks that have a custom init script that powers down the machine once it reaches init. Our CI fetches various kernel trees and branches, builds them, then boots them in Qemu. The machine has 2 minutes to power up and do…

so you have a very well tested boot sequence - is this not a slightly better kernel equivalent of 'it builds, ship it' ?

> a slightly better kernel equivalent of 'it builds, ship it' ?

Surely, "it runs, ship it"? That seems quite a bit better.

Re: How is the Linux kernel tested?

#27

Post by Bnshsysjab »

vbezhenar wrote: post #23
ndesaulniers wrote: post #19

Earlier quoted context omitted.

How do I test a driver for which I do not possess the hardware?

Write hardware emulation for VM with any behaviour you want to test.

Then I end up with a driver that conforms to emulated hardware and not it’s real counterpart

Re: How is the Linux kernel tested?

#28

Post by alkonaut »

varjag wrote: post #21
alkonaut wrote: post #8

Why is there such little emphasis on “traditional” testing, that is, regular unit tests? At least some portion of the code base is surely suitable for normal unit tests. For example data structures, scheduling algorithms, file systems, ...

Linux kernel project predates what you call traditional unit testing practices†. Its success in the first decade and a half coupled with pragmatics of hardware testing make the flow what it is now. † Regression testing was certainly known then, but it was not a dogmatic movement yet.

I understand there are hurdles due to legacy, language, low level etc. But if 1 or 2% of a huge code base is easily testable, shouldn’t it be? At least if/when regressions are found in functionality that can be easily testable (pure functions etc) it would seem prudent to add regression tests to prevent the thing from happening again. Even in a 30 year old C code base.

Re: How is the Linux kernel tested?

#29

Post by Twirrim »

ndesaulniers wrote: post #18

We run our own ci for building Linux kernels with clang. (ClangBuiltLinux.github.io). We take Debian's nightly package of ToT, make a docker image with the minimum tools we need, and use buildroot ramdisks that have a custom init script that powers down the machine once it reaches init. Our CI fetches various kernel trees and branches, builds them, then boots them in Qemu. The machine has 2 minutes to power up and do…

There was a bug in a released kernel I ended up having to hunt down back last summer. It required a very specific set of circumstances to actually trigger the bug, but luckily for us we found some obvious side effects from the bug such that it was possible to get the relevant information via a very simple statically compiled C program (there would be a certain value under /sys).

I used that C program as the initramfs for qemu, with it spinning up an instance based on the test kernel, and could grep for what I wanted in the output to get a relevant exit code.

Combine that git bisect's automatic bisecting process, and I was able to automatically git bisect through a few thousand commits in the kernel to find the cause. It only took it about 50 minutes to find it in the end, which was helped a bunch by ccache (and the sheer build parallelism granted by building on a 52 core system).

That was a fun, somewhat out of the blue, task.

The process I took was based on https://ldpreload.com/blog/git-bisect-run, with various adaptations to suit the needs and the relevant build environment.

Re: How is the Linux kernel tested?

#30

Post by eschaton »

EdSchouten wrote: post #9
alkonaut wrote: post #8

Why is there such little emphasis on “traditional” testing, that is, regular unit tests? At least some portion of the code base is surely suitable for normal unit tests. For example data structures, scheduling algorithms, file systems, ...

I think the main reason is that the Linux kernel (and similarly the *BSD kernels) are written in a programming language (C) that doesn't make it easy to do that. Code is often directly built on top of other kernel subsystems without any dependency injection whatsoever. This means that it's still possible to do unit testing of parts of the kernel, but it takes a crazy amount of effort, such as overriding symbols, over…

You don’t need a dependency injection framework to write unit tests, you just need cleanly separable units with well defined interfaces.
Post reply on HN