Live data from Hacker News

Embedding Binary Objects in C

flak.tedunangst.com

71–80 of 141 posts

Re: Embedding Binary Objects in C

#71
post #69

If you do this with a dynamically loaded library that you write to during execution, can C become a dynamic language?

Any turing language can be as dynamic as you want if you're motivated enough. After all, many interpreted languages are implemented in C...

Many dynamic recompiler/JIT implementations effectively do something like what you describe: they map some executable portion of memory and output native code that is then executed.

Re: Embedding Binary Objects in C

#72
One of the Capture the Flag challenges at CCC last year was to figure out a way to leak the contents of a file on a remote compiler server. The server would accept some C code and just give you a boolean true/false value whether the code compiled or not---without ever executing it.

Others came up with a solution abusing the C preprocessor by defining macros that would make the known structure of the file valid C and therefore they could just #include it. But my solution works with arbitrary files, without knowing the structure beforehand.

As others have pointed out here, you can use inline assembly and the .incbin directive to include a file. But how could that influence whether the compilation succeeds or fails? I figured out how to guess a byte of the file and create metadata sections only accepted by the linker if the guess was correct.

https://ryan.govost.es/2019/12/18/compilerbot.html

Re: Embedding Binary Objects in C

#73

If we're in the realm of "non-standard linker tricks"... Compilers will concatenate sections of the same name. You can use this trick to produce a concatenation of arrays across several files: $ cat t1.c __attribute__((section("some_array"))) int a[] = {1, 2, 3}; $ cat t2.c __attribute__((section("some_array"))) int b[] = {4, 5, 6}; $ cat t.c #include extern const int __start_some_array; extern const int __stop_some_…

Can you give an example of a use case where this is needed?

Re: Embedding Binary Objects in C

#74
post #73

If we're in the realm of "non-standard linker tricks"... Compilers will concatenate sections of the same name. You can use this trick to produce a concatenation of arrays across several files: $ cat t1.c __attribute__((section("some_array"))) int a[] = {1, 2, 3}; $ cat t2.c __attribute__((section("some_array"))) int b[] = {4, 5, 6}; $ cat t.c #include extern const int __start_some_array; extern const int __stop_some_…

Can you give an example of a use case where this is needed?

One place it's used is to avoid central "registration" for multiple "things". For example, consider a program with many fairly separate "subcommands". Using this mechanism allows adding a new subcommand simply by linking in a new source file (without changing a shared source file to list the new command).

Re: Embedding Binary Objects in C

#76
post #5

There's also gnu's objcopy. This post covers a lot of different approaches, and has lots of tips: https://www.devever.net/~hl/incbin

I got about 90% of the way to having a working ruby single-file executable builder which used objcopy to embed a sqlite database of the source files into an MRI build. Then YARV happened and the ruby build chain changed just enough that I needed to throw it out and start again.

Every now and again I ponder having another go with mruby...

Re: Embedding Binary Objects in C

#77

I needed this last week! Building an embedded firmware image for a dashboard display, with lots of PNG files for icons etc. 61 of them. The original developer wrote a tool to expand the PNGs to BMPs (arrays of 32-bit pixel values) and generate a C array definition as a text file. Which is lots bigger than the original PNG (13K => 100K sort of thing). Then included that C source in the build. Used up 700K of my firmwa…

Might be worth recompressing the PNGs too if you haven't already. In particular, using 8-bit PNG can size a lot on filesize if you don't need loads of colours.

Re: Embedding Binary Objects in C

#79
post #59

Totally not relevant for this post, but this is how we do this in Nim: const a = readFile("mydata") The `const` makes the expression evaluate at compile time, conveniently slurping the file into `a`, where it will be available at run time.

That is pretty cool, but also not the behaviour I'd expect coming from other languages where this would be "get the contents of this file from the current directory of my runtime environment, and assign them to immutable variable `a`."

Understandable; there's `let` for run time and `const` for compile time evaluation.

Re: Embedding Binary Objects in C

#80
post #55

Unfortunately, this technique is a bit problematic with modern C compilers. Because the `start` and `end` symbols are unrelated objects, as far as the compiler is concerned, the subtraction `&end - &start` to get the length of the data invokes undefined behavior. Just for that reason, I feel the include file with a hex dump is the better method.

Can you elaborate on this? Does "unrelated" have any special meaning in this context? Why is this operation undefined?

In C, accessing data outside the bounds of an object is undefined (like, you can’t legally “go past the end” of an array and end up in some other object). The “start” and “end” pointers, as far as the compiler is concerned, are totally different objects, so it may optimize a loop from one pointer to the other out since it’s impossible to increment an address so it’ll go from pounding at one thing to another.
Post reply on HN