If you do this with a dynamically loaded library that you write to during execution, can C become a dynamic language?
You've invented JIT.
Embedding Binary Objects in C
121–130 of 141 posts
Re: Embedding Binary Objects in C
#122Earlier quoted context omitted.
This is true, to my knowledge, I would NOT relie on the order to be always consistent. But I think you can enforce the order in the linker file, i.e. load the arrays one after the other, and then the order should keep constant. But you still have several places that you need to keep synchronized and, really, most people don't look in the linker file at all.
You can sort the array during program initialization, if you care about the order.
It's probably already in link order. If you have to know the names of the symbols in code to be able to sort them, it sort of defeats the purpose of the scheme in the first place.
Re: Embedding Binary Objects in C
#123If we're in the realm of "non-standard linker tricks"... Compilers will concatenate sections of the same name. You can use this trick to produce a concatenation of arrays across several files: $ cat t1.c __attribute__((section("some_array"))) int a[] = {1, 2, 3}; $ cat t2.c __attribute__((section("some_array"))) int b[] = {4, 5, 6}; $ cat t.c #include extern const int __start_some_array; extern const int __stop_some_…
What a weird coincidence seeing a comment about mergeable sections today...
A patch I helped review that's related to this in LLVM just landed today: https://reviews.llvm.org/D72194. (LLVM previously had support, but would perform bad merges in specific and rare edge cases; that patch fixes those cases).
Also, note that you need a custom linker script to define those symbols (__start_some_array, __stop_some_array). The Linux kernel does this, as noted in: https://nickdesaulniers.github.io/blog/2020/04/06/off-by-two...
Re: Embedding Binary Objects in C
#124Unfortunately, this technique is a bit problematic with modern C compilers. Because the `start` and `end` symbols are unrelated objects, as far as the compiler is concerned, the subtraction `&end - &start` to get the length of the data invokes undefined behavior. Just for that reason, I feel the include file with a hex dump is the better method.
That's what [u]intptr_t is for.
Re: Embedding Binary Objects in C
#125Earlier quoted context omitted.
I don't see how const would save memory, unless you have a separate ROM region that keeps that out of the main memory.
VMM. const memory can just be evicted and be swapped back in with zero cost. rw memory must be backed somehow, which us much more expensive. I submitted a patch
Not true unless you actively modify it (or the page it was in). "RW" mapped memory that is not dirty is also discarded on memory pressure.
Re: Embedding Binary Objects in C
#126Earlier quoted context omitted.
Yes, so long as you're willing to take a build-dep on vim. An od+awk+sed+sh combo will get you there from a POSIX base—and you have to add some wrapper text around the output of xxd anyway…
xxd exists in standalone form. Compile it. https://github.com/ConorOG/xxd
I mean, so long as you're willing to take a build-dep on that, in that case.
Re: Embedding Binary Objects in C
#127If we're in the realm of "non-standard linker tricks"... Compilers will concatenate sections of the same name. You can use this trick to produce a concatenation of arrays across several files: $ cat t1.c __attribute__((section("some_array"))) int a[] = {1, 2, 3}; $ cat t2.c __attribute__((section("some_array"))) int b[] = {4, 5, 6}; $ cat t.c #include extern const int __start_some_array; extern const int __stop_some_…
What a weird coincidence seeing a comment about mergeable sections today... A patch I helped review that's related to this in LLVM just landed today: https://reviews.llvm.org/D72194 . (LLVM previously had support, but would perform bad merges in specific and rare edge cases; that patch fixes those cases). Also, note that you need a custom linker script to define those symbols (__start_some_array, __stop_some_array).…
> A patch I helped review that's related to this in LLVM just landed today
Cool. :)
> Also, note that you need a custom linker script to define those symbols
The code I posted above compiles and runs without any custom linker script.
Re: Embedding Binary Objects in C
#128Earlier quoted context omitted.
What a weird coincidence seeing a comment about mergeable sections today... A patch I helped review that's related to this in LLVM just landed today: https://reviews.llvm.org/D72194 . (LLVM previously had support, but would perform bad merges in specific and rare edge cases; that patch fixes those cases). Also, note that you need a custom linker script to define those symbols (__start_some_array, __stop_some_array).…
> A patch I helped review that's related to this in LLVM just landed today Cool. :) > Also, note that you need a custom linker script to define those symbols The code I posted above compiles and runs without any custom linker script.
> The code I posted above compiles and runs without any custom linker script.
What?! Ok, I've seen the Linux kernel's custom linker script define symbols for custom ELF section boundaries, but TIL that you can get these automagically. I'm curious to see if LLD implements this logic, too.
Re: Embedding Binary Objects in C
#129My favorite is simply using a tool to create a C file with your binary data: static uint8_t mydata[] = {0xDE, 0xAD, 0xBE, 0xEF, ... }; The advantage is that it works anywhere, with any compiler. The disadvantage is that it can increase compile time. I would limit each .c file to 10MB; it seems there's a quadratic increase in build time with file size, at least with gcc. Also, instead of "0x%02x" I use decimal notatio…
I also did quite a bit of benchmarking, and never saw any quadratic behaviour [0]. However, there is a floor for RSS and and elapsed time, but you would need to test assets less than ~1MB to see it. For assets over a limited range, the graph might look quadratic, but if you cover a large enough range you will see there are two regimes of behaviour.
Re: Embedding Binary Objects in C
#130Earlier quoted context omitted.
C has a proposal for #embed. http://www.open-std.org/jtc1/sc22/wg14/www/docs/n2499.pdf
In the section on search paths… It follows the same implementation experience guidelines as #include by leaving the search paths implementation defined, with the understand that implementations are not monsters and will generally provide… Really? We are going to "understand that implementations are not monsters" after what they've done with "undefined"? I think maybe these standards should be written from the perspec…
May I ask what's wrong with using xxd? I don't have an opinion about C, but C++ is already pretty complicated, and adding features to the language when there are already well-known solutions doesn't seem wise.
(For the record, I'm aware that there are size limitation when using xxd, but there are also other solutions).