Live data from Hacker News

Embedding Binary Objects in C

flak.tedunangst.com

81–90 of 141 posts

Re: Embedding Binary Objects in C

#81

I needed this last week! Building an embedded firmware image for a dashboard display, with lots of PNG files for icons etc. 61 of them. The original developer wrote a tool to expand the PNGs to BMPs (arrays of 32-bit pixel values) and generate a C array definition as a text file. Which is lots bigger than the original PNG (13K => 100K sort of thing). Then included that C source in the build. Used up 700K of my firmwa…

Afaik `xxd -i` is much simplier. No need to write another tool.

Re: Embedding Binary Objects in C

#82
post #24

My favorite is simply using a tool to create a C file with your binary data: static uint8_t mydata[] = {0xDE, 0xAD, 0xBE, 0xEF, ... }; The advantage is that it works anywhere, with any compiler. The disadvantage is that it can increase compile time. I would limit each .c file to 10MB; it seems there's a quadratic increase in build time with file size, at least with gcc. Also, instead of "0x%02x" I use decimal notatio…

For those wondering, the easiest way to do this is with: xxd -i filename.bin

Yes, so long as you're willing to take a build-dep on vim. An od+awk+sed+sh combo will get you there from a POSIX base—and you have to add some wrapper text around the output of xxd anyway…

Re: Embedding Binary Objects in C

#83

I needed this last week! Building an embedded firmware image for a dashboard display, with lots of PNG files for icons etc. 61 of them. The original developer wrote a tool to expand the PNGs to BMPs (arrays of 32-bit pixel values) and generate a C array definition as a text file. Which is lots bigger than the original PNG (13K => 100K sort of thing). Then included that C source in the build. Used up 700K of my firmwa…

There are even smaller png libs than libpng, try a stripped down stb_image for example. Wouldn't use that for user-supplied images, but since you control them all it should be fine.

I do this with embedded graphics in boards for a lot of reasons. Works great - and STB is pretty easy to modify if you need a funny one (eg : monochrome pbm for small displays). Animated gif, png, bunch of others all work pretty smoothly. Just include the bits you need so it can be even smaller.

Re: Embedding Binary Objects in C

#84
post #24

My favorite is simply using a tool to create a C file with your binary data: static uint8_t mydata[] = {0xDE, 0xAD, 0xBE, 0xEF, ... }; The advantage is that it works anywhere, with any compiler. The disadvantage is that it can increase compile time. I would limit each .c file to 10MB; it seems there's a quadratic increase in build time with file size, at least with gcc. Also, instead of "0x%02x" I use decimal notatio…

For those wondering, the easiest way to do this is with: xxd -i filename.bin

Too bad that xxd doesn't include the const before the array decl. With const it saves a lot of memory.

Re: Embedding Binary Objects in C

#87

One of the Capture the Flag challenges at CCC last year was to figure out a way to leak the contents of a file on a remote compiler server. The server would accept some C code and just give you a boolean true/false value whether the code compiled or not---without ever executing it. Others came up with a solution abusing the C preprocessor by defining macros that would make the known structure of the file valid C and…

We didn't actually use .incbin on that challenge, interestingly; however, we did use it (along with some nested static constructor function trickery) for Online Calc from TokyoWesterns CTF. For that we had some straightforward tricks to get around the forbidden characters, then we abused the flag format to get the #include to work. After that we could leak the flag byte-by-byte using Linux's BUILD_BUG_ON_ZERO, which is basically an upgraded static_assert. I think this is the code we ran:

  """
  _Pragma("clang diagnostic push")
  _Pragma("clang diagnostic ignored \\"-Wtrigraphs\\"")
  ??= define STRINGIZE(...) ??=__VA_ARGS__
  ??= define EXPAND_AND_STRINGIZE(...) STRINGIZE(__VA_ARGS__)
  ??= define hxp EXPAND_AND_STRINGIZE(
  ??= define BUILD_BUG_ON_ZERO(e) (sizeof(struct ))
  ??= define BUILD_BUG_ON(condition) ((void)BUILD_BUG_ON_ZERO(condition))
  const char flag =
  ??= include "flag"
  )[{}];
  BUILD_BUG_ON(flag == {});
  _Pragma("clang diagnostic pop")
  """
(The {}, of course, being Python format parameters from our script.)

Re: Embedding Binary Objects in C

#88

Earlier quoted context omitted.

Can you elaborate on this? Does "unrelated" have any special meaning in this context? Why is this operation undefined?

In C, accessing data outside the bounds of an object is undefined (like, you can’t legally “go past the end” of an array and end up in some other object). The “start” and “end” pointers, as far as the compiler is concerned, are totally different objects, so it may optimize a loop from one pointer to the other out since it’s impossible to increment an address so it’ll go from pounding at one thing to another.

So is the idea that just because the two objects happen to appear sequentially in memory under a certain implementation, the compiler (or linker in this case?) has no obligation to ensure that assumption holds?

Re: Embedding Binary Objects in C

#89
post #73

If we're in the realm of "non-standard linker tricks"... Compilers will concatenate sections of the same name. You can use this trick to produce a concatenation of arrays across several files: $ cat t1.c __attribute__((section("some_array"))) int a[] = {1, 2, 3}; $ cat t2.c __attribute__((section("some_array"))) int b[] = {4, 5, 6}; $ cat t.c #include extern const int __start_some_array; extern const int __stop_some_…

Can you give an example of a use case where this is needed?

This is used for constructor & destructor functions in C code. The function pointers get put in an array in a certain section which is collated then the startup and exit code calls them. Ordering is not guaranteed.

Re: Embedding Binary Objects in C

#90

According to https://news.ycombinator.com/item?id=22865842 #embed proposal is in the review process http://www.open-std.org/jtc1/sc22/wg14/www/docs/n2499.pdf . This similar (same?) proposal is also presented and in the review at WG21 to be included in C++ standard. The author puts a lot of work towards making this proposal a reality (as far as I as a casual twitter/slack observer can see) and I'm looking forward to i…

Unfortunately, the proposal has been stalled by the C++ committee and the author is uninterested in continuing it. See their post here https://thephd.github.io/full-circle-embed
Post reply on HN