It’s a simple and practical solution. It irks me for some reason but I can’t think of something better.
Debug symbols, which can be resolved at load time. This would also make ebpf bytecode more or less kernel version independent.
Extending the Linux Kernel with Built-In Kernel Headers
31–40 of 62 posts
Re: Extending the Linux Kernel with Built-In Kernel Headers
#32It’s a simple and practical solution. It irks me for some reason but I can’t think of something better.
It's too bad the squashfs idea was dropped. Having to download and unpack the archive somewhere seems like an extra step that should be unnecessary vs just loading the module and having the tree appear in /sys for you to point your compiler directly at. Also a bummer that it looks like you have to be running the kernel to get access to this— perhaps the archive could be marked off in the binary somehow so that there'…
squashfs was dropped because Greg doesn't like squashfs, which I can understand somewhat (it doesn't have an active maintainer really, there's no userspace library, just binaries, etc.).
However, we're doing a project where I work that depends heavily on squashfs, so the fact that it was dropped from this even though it would have been nicer because people don't like it because it doesn't have a maintainer worries me. Hopefully someone picks it up :)
Re: Extending the Linux Kernel with Built-In Kernel Headers
#33Earlier quoted context omitted.
How so? Seems like an elegant enough solution to ensure you always have the right headers to build modules against the currently running kernel.
Isn't kernel header availability a solved problem on any linux system that isn't busy pretending not to be built on linux?
The nice thing about this is that you can keep kernel headers around in a much smaller format on-disk, while still making them available separately as a package if you want.
A few problems this solves:
1. Neither user nor tool knows specifically how to install the headers for specifically this version of the kernel.
2. Manually installing a header package on e.g. Ubuntu marks it as manually selected, meaning it doesn't get cleaned up with an 'autoremove'; selecting a generic kernel headers package (like linux-headers-4.14-generic or something) means that every time your system auto-updates the kernel package version (which happens by default on, for example, AWS IIRC, and happens a lot) you end up with yet another copy of the kernel headers, and then you blow your I/O budget uninstalling them.
3. It removes one extra step to running e.g. an eBPF program, or building an e.g. out-of-tree kernel module, so make scripts can now start to take advantage of that. For example:
# Get headers
if modprobe kernel_headers; then
else
fiRe: Extending the Linux Kernel with Built-In Kernel Headers
#34Earlier quoted context omitted.
java interfaces. I think the concept of coding to an interface (or an API specification) goes beyond what java did. In my professional life, I see people constantly downplaying and ignoring importance of a formal and stable API specification, just to suffer consequences later.
If you just give me a .jar do I get those interfaces in a consumable form? Said differently, can I download your .jar and write my own code to interface with it while on a desert island without any other resources?
Yes.
If you have a random JAR file, an IDE can introspect it to see what classes are in it and what methods are in those classes.
The only exception is if you run it through some kind of obfuscator first.
Re: Extending the Linux Kernel with Built-In Kernel Headers
#35Earlier quoted context omitted.
It's too bad the squashfs idea was dropped. Having to download and unpack the archive somewhere seems like an extra step that should be unnecessary vs just loading the module and having the tree appear in /sys for you to point your compiler directly at. Also a bummer that it looks like you have to be running the kernel to get access to this— perhaps the archive could be marked off in the binary somehow so that there'…
squashfs was dropped because Greg doesn't like squashfs, which I can understand somewhat (it doesn't have an active maintainer really, there's no userspace library, just binaries, etc.). However, we're doing a project where I work that depends heavily on squashfs, so the fact that it was dropped from this even though it would have been nicer because people don't like it because it doesn't have a maintainer worries me…
Oh, gosh. Since when it's without a maintainer? I would think it's quite an important piece of most embedded Linux projects. Those companies really should sponsor someone to take care of it.
My personal project also depends on squashfs. Are there any other options for read-only compressed rootfs?
Squashfs-tools are a bit rough around the edges that's for sure. I patched in ability to produce an image without root for my use, but I think it would be generally useful. Sort list file format is funky. It's: "filename_string priority_integer", so it will not work for files with spaces. Also new lines, but that's far less common. It was not yet a problem for me and I can always patch it more and one can say, that people having files with spaces deserve it, though it's entirely different issue.
Re: Extending the Linux Kernel with Built-In Kernel Headers
#36Earlier quoted context omitted.
squashfs was dropped because Greg doesn't like squashfs, which I can understand somewhat (it doesn't have an active maintainer really, there's no userspace library, just binaries, etc.). However, we're doing a project where I work that depends heavily on squashfs, so the fact that it was dropped from this even though it would have been nicer because people don't like it because it doesn't have a maintainer worries me…
Oh, gosh. Since when it's without a maintainer? I would think it's quite an important piece of most embedded Linux projects. Those companies really should sponsor someone to take care of it. My personal project also depends on squashfs. Are there any other options for read-only compressed rootfs? Squashfs-tools are a bit rough around the edges that's for sure. I patched in ability to produce an image without root for…
Looks like he might (?) be coming back: https://github.com/plougher/squashfs-tools/issues/54#issueco... although the release he's talking about didn't happen.
The last commit that wasn't an API refactoring in the kernel tree is a3f94cb99a854fa381fe7fadd97c4f61633717a5, which is from Aug 2, 2018.
Re: Extending the Linux Kernel with Built-In Kernel Headers
#37Earlier quoted context omitted.
squashfs was dropped because Greg doesn't like squashfs, which I can understand somewhat (it doesn't have an active maintainer really, there's no userspace library, just binaries, etc.). However, we're doing a project where I work that depends heavily on squashfs, so the fact that it was dropped from this even though it would have been nicer because people don't like it because it doesn't have a maintainer worries me…
Oh, gosh. Since when it's without a maintainer? I would think it's quite an important piece of most embedded Linux projects. Those companies really should sponsor someone to take care of it. My personal project also depends on squashfs. Are there any other options for read-only compressed rootfs? Squashfs-tools are a bit rough around the edges that's for sure. I patched in ability to produce an image without root for…
squashfs has horrible performance. All requests to the block layer are 512 Bytes. Other filesystems like ext4 make much bigger requests and perform much better in the end despite the compression of squashfs leading to lower overall data volume.
Disclaimer: Measured 2 years ago on ARM32, emmc, with a 4.1(?) kernel.
Re: Extending the Linux Kernel with Built-In Kernel Headers
#38Shipping the kernel headers is complicated, but somehow a whole C compiler is not? Or doesnt BPF compilation need a C compiler?
Re: Extending the Linux Kernel with Built-In Kernel Headers
#39Earlier quoted context omitted.
squashfs was dropped because Greg doesn't like squashfs, which I can understand somewhat (it doesn't have an active maintainer really, there's no userspace library, just binaries, etc.). However, we're doing a project where I work that depends heavily on squashfs, so the fact that it was dropped from this even though it would have been nicer because people don't like it because it doesn't have a maintainer worries me…
Oh, gosh. Since when it's without a maintainer? I would think it's quite an important piece of most embedded Linux projects. Those companies really should sponsor someone to take care of it. My personal project also depends on squashfs. Are there any other options for read-only compressed rootfs? Squashfs-tools are a bit rough around the edges that's for sure. I patched in ability to produce an image without root for…
Btrfs supports compression. A "seed" flag that makes it read-only. If you 'btrfs device add' a rw second device, it can be read-write with all writes directed to the second device, e.g. /dev/zram device for volatile overlay for e.g. a LiveOS boot, or you can add a blank partition and then remove the seed which causes the seed data to be replicated to the rw partition. Plus all metadata and data are checksummed.
zstd support since linux 4.14, and a mount time option for compression level since linux 5.1. So you could get very good compression ratios equivalent to squashfs, but squashfs will still come out slightly ahead because it also compresses its own metadata, where Btrfs doesn't.
My best guess is squashfs is a sufficiently successful project that it's allowed most distributions who depend on it (quite a few) to sit on its laurels.
Re: Extending the Linux Kernel with Built-In Kernel Headers
#40Earlier quoted context omitted.
Oh, gosh. Since when it's without a maintainer? I would think it's quite an important piece of most embedded Linux projects. Those companies really should sponsor someone to take care of it. My personal project also depends on squashfs. Are there any other options for read-only compressed rootfs? Squashfs-tools are a bit rough around the edges that's for sure. I patched in ability to produce an image without root for…
squashfs has horrible performance. All requests to the block layer are 512 Bytes. Other filesystems like ext4 make much bigger requests and perform much better in the end despite the compression of squashfs leading to lower overall data volume. Disclaimer: Measured 2 years ago on ARM32, emmc, with a 4.1(?) kernel.
A big part of the problem might be xz decompression, that's been my discovery anyway.