Too bad there's no mention of the new BTF file format, which is supposed to resolve this issue for the eBPF side:
Extending the Linux Kernel with Built-In Kernel Headers
51–60 of 62 posts
Re: Extending the Linux Kernel with Built-In Kernel Headers
#52What's wrong with shipping headers and using DKMS for building like in GNU/Linux?
Re: Extending the Linux Kernel with Built-In Kernel Headers
#53What's wrong with shipping headers and using DKMS for building like in GNU/Linux?
[deleted]
Re: Extending the Linux Kernel with Built-In Kernel Headers
#54Earlier quoted context omitted.
Doesn't necessarily work for the kernel, but GIR and typelibs provide machine-readable descriptions of C APIs: https://github.com/GNOME/gobject-introspection
From what I know about it, gobject-introspection has some nice properties, but one killer drawback: it's incompatible with cross compilation. This is apparently because it requires running binaries compiled for the target system as part of the build process. You actually can cross compile if you have an emulator for that system handy [1], but that's horrible. Apparently the reason it requires running the compiled bin…
Yeah, I'd love to have something like BTF or CTF used widely for machine-readable type information. (https://facebookmicrosites.github.io/bpf/blog/2018/11/14/btf... gives some further information there.)
The limitations regarding macros sound like the biggest issue to me (both code-like macros and just simple defined names for values via #define). I'd love to see solutions for that. What do you think that would look like?
Re: Extending the Linux Kernel with Built-In Kernel Headers
#55Earlier quoted context omitted.
From what I know about it, gobject-introspection has some nice properties, but one killer drawback: it's incompatible with cross compilation. This is apparently because it requires running binaries compiled for the target system as part of the build process. You actually can cross compile if you have an emulator for that system handy [1], but that's horrible. Apparently the reason it requires running the compiled bin…
Yeah, I'd love to have something like BTF or CTF used widely for machine-readable type information. ( https://facebookmicrosites.github.io/bpf/blog/2018/11/14/btf... gives some further information there.) The limitations regarding macros sound like the biggest issue to me (both code-like macros and just simple defined names for values via #define). I'd love to see solutions for that. What do you think that would look…
Interesting writeup! I didn't realize there was an active attempt to generate BTF for the kernel.
Regarding macros...
Well, to start with, there's the brute-force approach of simply embedding textual macro definitions. That might be good enough for most use cases in practice: as far as I know, most BPF hooks are written in either C or the C-like bpftrace language, so expanding macros as text would probably give a sensible result for the majority of macros that aren't particularly complex. And macro definitions are already included in the DWARF info, so the DWARF-to-BTF approach from your link could be easily extended to embed them.
But it would be nice to describe macros in a more structured format, which could allow use from non-C-like languages and would probably save on file size. Some prior art I'm familiar with is rust-bindgen, which generates Rust bindings for C headers using libclang, and supports translating C macros that expand to constants. Basically it checks each macro that's defined without arguments and uses libclang to try to evaluate it as a C constant expression; this will fail for macros that expand to things other than constant expressions, but it just ignores those. If evaluation succeeds, it translates the macro to a typed Rust constant declaration.
It might be possible to do something similar for BTF. As output format, either add a new 'constant integer' node, or translate such macros as if they were enum definitions. For Linux it would probably be best to avoid a dependency on libclang, but a custom parser might work, or maybe a hackier approach based on feeding things to the C compiler like:
enum { value_of_SOME_MACRO = ((((((((( SOME_MACRO ))))))))) };
Edit: Forgot to mention – functional macros would be nice too, but of course they're much harder to translate. And heck, what about inline functions? Convert them to BPF?
Re: Extending the Linux Kernel with Built-In Kernel Headers
#56Earlier quoted context omitted.
Yeah, I'd love to have something like BTF or CTF used widely for machine-readable type information. ( https://facebookmicrosites.github.io/bpf/blog/2018/11/14/btf... gives some further information there.) The limitations regarding macros sound like the biggest issue to me (both code-like macros and just simple defined names for values via #define). I'd love to see solutions for that. What do you think that would look…
Interesting writeup! I didn't realize there was an active attempt to generate BTF for the kernel. Regarding macros... Well, to start with, there's the brute-force approach of simply embedding textual macro definitions. That might be good enough for most use cases in practice: as far as I know, most BPF hooks are written in either C or the C-like bpftrace language, so expanding macros as text would probably give a sen…
> there's the brute-force approach of simply embedding textual macro definitions. That might be good enough for most use cases in practice
I very much want this for usage from Rust, so that doesn't suffice.
> It might be possible to do something similar for BTF. As output format, either add a new 'constant integer' node
That sounds promising to me, for the common case.
> Edit: Forgot to mention – functional macros would be nice too, but of course they're much harder to translate. And heck, what about inline functions? Convert them to BPF?
In an ideal world, 1) emit a symbol for them so they can be used from any language, albeit not "inline", and 2) compile them to bytecode that LTO can incorporate and optimize, for languages using the same linker.
Neither of those would work for macros designed for especially unusual usages that can't possibly work as functions. (The two most common cases I can think of: macros that accept names and use them as lvalues, and macros that emit partial syntax such as paired macros emitting unmatched braces.) But honestly, flagging those and handling all the common cases via BTF information would still be a huge improvement.
Perhaps we should continue this on an IRLO thread?
Re: Extending the Linux Kernel with Built-In Kernel Headers
#57Earlier quoted context omitted.
It's too bad the squashfs idea was dropped. Having to download and unpack the archive somewhere seems like an extra step that should be unnecessary vs just loading the module and having the tree appear in /sys for you to point your compiler directly at. Also a bummer that it looks like you have to be running the kernel to get access to this— perhaps the archive could be marked off in the binary somehow so that there'…
squashfs was dropped because Greg doesn't like squashfs, which I can understand somewhat (it doesn't have an active maintainer really, there's no userspace library, just binaries, etc.). However, we're doing a project where I work that depends heavily on squashfs, so the fact that it was dropped from this even though it would have been nicer because people don't like it because it doesn't have a maintainer worries me…
I'm the author and maintainer of Squashfs and I can assure you that Squashfs is not unmaintained. Over the last couple of years Squashfs has been stable, without requests for new features, and so work have been mostly only security improvements and bug fixes. But there is a big difference between that and claiming Squashfs is unmaintained.
In fact I don't know why you're claiming it is unmaintained. Got an axe to grind perhaps?
Re: Extending the Linux Kernel with Built-In Kernel Headers
#58Earlier quoted context omitted.
squashfs was dropped because Greg doesn't like squashfs, which I can understand somewhat (it doesn't have an active maintainer really, there's no userspace library, just binaries, etc.). However, we're doing a project where I work that depends heavily on squashfs, so the fact that it was dropped from this even though it would have been nicer because people don't like it because it doesn't have a maintainer worries me…
Oh, gosh. Since when it's without a maintainer? I would think it's quite an important piece of most embedded Linux projects. Those companies really should sponsor someone to take care of it. My personal project also depends on squashfs. Are there any other options for read-only compressed rootfs? Squashfs-tools are a bit rough around the edges that's for sure. I patched in ability to produce an image without root for…
It is not without a maintainer. But you make a good point that the companies that use it should sponsor it. I still have to maintain it for free in my spare time, and spare time gets less and less each year.
In fact it woudn't surprise me if the parent commenter is one of the people working for a multi-billion dollar company that got offended when I told them I wouldn't do some work for them, for free. Hence the suspicion there is an axe to grind.
Re: Extending the Linux Kernel with Built-In Kernel Headers
#59Earlier quoted context omitted.
Oh, gosh. Since when it's without a maintainer? I would think it's quite an important piece of most embedded Linux projects. Those companies really should sponsor someone to take care of it. My personal project also depends on squashfs. Are there any other options for read-only compressed rootfs? Squashfs-tools are a bit rough around the edges that's for sure. I patched in ability to produce an image without root for…
squashfs has horrible performance. All requests to the block layer are 512 Bytes. Other filesystems like ext4 make much bigger requests and perform much better in the end despite the compression of squashfs leading to lower overall data volume. Disclaimer: Measured 2 years ago on ARM32, emmc, with a 4.1(?) kernel.
Try using the SQUASHFS_4K_DEVBLK_SIZE config option next time
By default Squashfs sets the dev block size (sb_min_blocksize) to 1K or the smallest block size supported by the block device (if larger). This, because blocks are packed together and unaligned in Squashfs, should reduce latency.
This, however, gives poor performance on MTD NAND devices where the optimal I/O size is 4K (even though the devices can support smaller block sizes).
Using a 4K device block size may also improve overall I/O performance for some file access patterns (e.g. sequential accesses of files in filesystem order) on all media.
Setting this option will force Squashfs to use a 4K device block size by default.
If unsure, say N.
Re: Extending the Linux Kernel with Built-In Kernel Headers
#60Earlier quoted context omitted.
squashfs was dropped because Greg doesn't like squashfs, which I can understand somewhat (it doesn't have an active maintainer really, there's no userspace library, just binaries, etc.). However, we're doing a project where I work that depends heavily on squashfs, so the fact that it was dropped from this even though it would have been nicer because people don't like it because it doesn't have a maintainer worries me…
I'm the author and maintainer of Squashfs and I can assure you that Squashfs is not unmaintained. Over the last couple of years Squashfs has been stable, without requests for new features, and so work have been mostly only security improvements and bug fixes. But there is a big difference between that and claiming Squashfs is unmaintained. In fact I don't know why you're claiming it is unmaintained. Got an axe to gri…
I'm not claiming it's unmaintained. I'm saying that the maintainer is not active. You yourself mention elsewhere in this thread that you have little free time to maintain squashfs, so that seems fair.
No axe to grind; it's a cool project and that's why we decided to use it. Thanks for your work on it, and we'll send patches if/when we have them :)