Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Embedding Binary Objects in C

flak.tedunangst.com

Post reply on HN
Print view View original

101–110 of 141 posts

Re: Embedding Binary Objects in C

#101

Post by stefan_ »

haneefmubarak wrote: post #36
haberman wrote: post #13

If we're in the realm of "non-standard linker tricks"... Compilers will concatenate sections of the same name. You can use this trick to produce a concatenation of arrays across several files: $ cat t1.c __attribute__((section("some_array"))) int a[] = {1, 2, 3}; $ cat t2.c __attribute__((section("some_array"))) int b[] = {4, 5, 6}; $ cat t.c #include extern const int __start_some_array; extern const int __stop_some_…

How exactly is the ordering of items across the section decided? Is it relatively random or is it based on the order that the object files are passed to the linker?

Assume it's random. See also the static initialization order ‘fiasco’: https://isocpp.org/wiki/faq/ctors#static-init-order

Re: Embedding Binary Objects in C

#102

Post by rkeene2 »

regularfry wrote: post #76
tyingq wrote: post #5

There's also gnu's objcopy. This post covers a lot of different approaches, and has lots of tips: https://www.devever.net/~hl/incbin

I got about 90% of the way to having a working ruby single-file executable builder which used objcopy to embed a sqlite database of the source files into an MRI build. Then YARV happened and the ruby build chain changed just enough that I needed to throw it out and start again. Every now and again I ponder having another go with mruby...

FWIW, this is roughly how Tclkits work in the Tcl world. Although by default they use a Metakit database instead of SQLite.

Currently they append the database to the end of the executable, which has some problems and I'm working to make including it in the image more standardized as part of XVFS [0].

[0] https://chiselapp.com/user/rkeene/repository/xvfs/

Re: Embedding Binary Objects in C

#103

Post by hikarudo »

KMnO4 wrote: post #54
hikarudo wrote: post #14

My favorite is simply using a tool to create a C file with your binary data: static uint8_t mydata[] = {0xDE, 0xAD, 0xBE, 0xEF, ... }; The advantage is that it works anywhere, with any compiler. The disadvantage is that it can increase compile time. I would limit each .c file to 10MB; it seems there's a quadratic increase in build time with file size, at least with gcc. Also, instead of "0x%02x" I use decimal notatio…

I've never noticed the "quadratic" build time increase you mention, so I did a test[0]. Files in size from 1mb to 50mb, 3 trials each. These are the results, and they look absolutely linear[1]. [0]: https://pastebin.com/Z9329xkc [1]: https://i.imgur.com/I3XSBDg.png

I stand corrected! I shouldn't have guessed "quadratic" without measuring.

Large files will make gcc use more memory, and if you're building on a low-RAW machine, the OS will start to swap. That's probably the effect I saw. Nothing to do with "quadratic" compilation time.

Re: Embedding Binary Objects in C

#104

Post by jhallenworld »

qppo wrote: post #73
haberman wrote: post #13

If we're in the realm of "non-standard linker tricks"... Compilers will concatenate sections of the same name. You can use this trick to produce a concatenation of arrays across several files: $ cat t1.c __attribute__((section("some_array"))) int a[] = {1, 2, 3}; $ cat t2.c __attribute__((section("some_array"))) int b[] = {4, 5, 6}; $ cat t.c #include extern const int __start_some_array; extern const int __stop_some_…

Can you give an example of a use case where this is needed?

I like to use this technique for CLI command and help tables. It eliminates having a single giant table in a file, instead you have a C macro which makes the system aware of your command right next to your command definition.

   int my_command(..) { ... }
   COMMAND(my_command, "command_name", "example command")
   HELP("use --foo argument for this...")
   HELP("use --bar argument for that...")
Also used this all the time back when I programmed assembly language.

It's definitely a problem that this is not standardized. For example, TI has their own C compiler for their ARM CPUs, not everybody uses gcc..

Re: Embedding Binary Objects in C

#105

Post by JoeAltmaier »

RealityVoid wrote: post #68
JoeAltmaier wrote: post #37

Earlier quoted context omitted.

Nope. Dumb Visual Studio doesn't support embedded binary in cross-compile. That's the tool the client wanted, so I'm stuck with it. Funny, they (VisualGDB) even have a tutorial on doing it. But the option 'Embedded Resource' doesn't exist in my (modern) Visual Studio. Ha! And the feedback I gave VisualGDB about their outdated tutorial, they responded "Just pay HERE and we'll be glad to solve your problem!" They can g…

Could you not try using the objcopy method mentioned here outside Visual Studio (or somewhere before the build process) and then just using the .o at link time, just referencing it from the outside at linkage? I expect at least a way to: - hook custom tool calls - link external objects into the project Also, hello fellow automotive embedded SW eng! I know your suffering! :))

VisualGDB uses a tool-created makefile for cross-compiling and building. Not a lot of room for me to innovate. I suppose the .o trick could work. But it still requires careful hand-scripted builds. I already have that - I run a script to read a binary file and emit .c declaration, which I pipe to a C file and include in the project.

I suppose it lets me drop support of the script, which is something.

Re: Embedding Binary Objects in C

#106

Post by JoeAltmaier »

anthk wrote: post #97
JoeAltmaier wrote: post #16

I needed this last week! Building an embedded firmware image for a dashboard display, with lots of PNG files for icons etc. 61 of them. The original developer wrote a tool to expand the PNGs to BMPs (arrays of 32-bit pixel values) and generate a C array definition as a text file. Which is lots bigger than the original PNG (13K => 100K sort of thing). Then included that C source in the build. Used up 700K of my firmwa…

imagemagick and netpbm can do that. Gimp too.

I assume so. But how many tools do I want to include in my client's build dependencies?

Re: Embedding Binary Objects in C

#107

Post by jws »

beefhash wrote: post #33
dirtydroog wrote: post #21

C++ has a proposal for std::embed http://www.open-std.org/jtc1/sc22/wg21/docs/papers/2018/p104...

C has a proposal for #embed. http://www.open-std.org/jtc1/sc22/wg14/www/docs/n2499.pdf

In the section on search paths…

It follows the same implementation experience guidelines as #include by leaving the search paths implementation defined, with the understand that implementations are not monsters and will generally provide…

Really? We are going to "understand that implementations are not monsters" after what they've done with "undefined"? I think maybe these standards should be written from the perspective that implementations are sociopathic demons summoned against their will.

(But I really want this feature. I regularly use xxd and some Makefile rules to embed assets in my executables. For instance, in a web service I might have all my default configuration and templates in the executable with command line options to send them to standard output and override them from external files. Then on the chance someone needs to make a change they can just make their own file and use it.)

Re: Embedding Binary Objects in C

#109

Post by balnaphone »

samrat wrote: post #99

I am in India and I cannot access the site. The DNS resolves to `23.227.131.12` but the server doesn't seem to respond to pings either. I can access the site through a VPN. Anyone else facing the same issue? Anyone know what's up with that?

Same, from Canada. My guess is that it will come back up later.

Re: Embedding Binary Objects in C

#110

Post by 08-15 »

pdw wrote: post #55

Unfortunately, this technique is a bit problematic with modern C compilers. Because the `start` and `end` symbols are unrelated objects, as far as the compiler is concerned, the subtraction `&end - &start` to get the length of the data invokes undefined behavior. Just for that reason, I feel the include file with a hex dump is the better method.

> Because the `start` and `end` symbols are unrelated objects

That's irrelevant. What matters is whether they point to (or just past the end of) the same object, which they do.

Post reply on HN