Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Embedding Binary Objects in C

flak.tedunangst.com

Post reply on HN
Print view View original

111–120 of 141 posts

Re: Embedding Binary Objects in C

#111

Post by 08-15 »

saagarjha wrote: post #80
ghostpepper wrote: post #60

Earlier quoted context omitted.

Can you elaborate on this? Does "unrelated" have any special meaning in this context? Why is this operation undefined?

In C, accessing data outside the bounds of an object is undefined (like, you can’t legally “go past the end” of an array and end up in some other object). The “start” and “end” pointers, as far as the compiler is concerned, are totally different objects, so it may optimize a loop from one pointer to the other out since it’s impossible to increment an address so it’ll go from pounding at one thing to another.

Where is data outside any object accessed? Note that forming a pointer just past the end of an object is well-defined; only dereferencing it isn't.

Re: Embedding Binary Objects in C

#112

Post by claytongulick »

gmueckl wrote: post #51
kazinator wrote: post #35

I wouldn't do it this way, within the toolchain. Better to just append the data to the finished executable, with a header that contains an identifying marker. The program can scan its own image and look for that marker to get at the data. That will work on any OS with any executable and linker format and can be done post-production (users or downstream distributors can receive the binary and add customized data to it…

You should check out how self-extracting archives work. All of those that I am aware of have a first part that is a regular executable and the archive to ve extracted is appended to that. File formats like zip have an advantage here as they keep their index at the end of the file after the compressed data. So there is no need to scan for the start of the archive.

IIRC this is how nwjs works too.

Re: Embedding Binary Objects in C

#114

Post by jagrsw »

saagarjha wrote: post #3

Or, as any CTFer worth their salt will tell you, __asm__(“.incbin file”);

A real world example of use: https://github.com/google/honggfuzz/blob/075756bea8d1f08eb19...

I think it's a bit better than

  ld -b binary ...
cause it doesn't depend on the static linker generating symbols, which feels a bit like a non-standard/non-portable feature. But, who knows, maybe the vast majority of ld variants actually implement it.

Re: Embedding Binary Objects in C

#115

Post by iainmerrick »

nicoburns wrote: post #77
JoeAltmaier wrote: post #16

I needed this last week! Building an embedded firmware image for a dashboard display, with lots of PNG files for icons etc. 61 of them. The original developer wrote a tool to expand the PNGs to BMPs (arrays of 32-bit pixel values) and generate a C array definition as a text file. Which is lots bigger than the original PNG (13K => 100K sort of thing). Then included that C source in the build. Used up 700K of my firmwa…

Might be worth recompressing the PNGs too if you haven't already. In particular, using 8-bit PNG can size a lot on filesize if you don't need loads of colours.

Yes! And having tried a few different tools for this, I recommend pngcrush (easy to use, excellent results).

Edit to add: I settled on pngcrush 3 or 4 years ago; if there's something newer and better, I'm interested in hearing about it!

Re: Embedding Binary Objects in C

#116

Post by MaxBarraclough »

felixguendling wrote: post #108

The linked page is currently not available. Archive link: https://web.archive.org/web/20200416183057/https://flak.tedu... There are also ways to do this on Windows and Mac OS X. This library provides CMake support to do this: https://github.com/motis-project/res

Do you know how it compares with CMakeRC? See https://news.ycombinator.com/item?id=22888879

Re: Embedding Binary Objects in C

#117

Post by berti »

codys wrote: post #74
qppo wrote: post #73

Earlier quoted context omitted.

Can you give an example of a use case where this is needed?

One place it's used is to avoid central "registration" for multiple "things". For example, consider a program with many fairly separate "subcommands". Using this mechanism allows adding a new subcommand simply by linking in a new source file (without changing a shared source file to list the new command).

I use this idea in embedded work, where the code is split into "modules". The same firmware source is shared amongst various differing pieces of hardware, each of which utilise a different subset of modules.

Re: Embedding Binary Objects in C

#118

Post by rurban »

saagarjha wrote: post #94
rurban wrote: post #84

Earlier quoted context omitted.

Too bad that xxd doesn't include the const before the array decl. With const it saves a lot of memory.

I don't see how const would save memory, unless you have a separate ROM region that keeps that out of the main memory.

VMM. const memory can just be evicted and be swapped back in with zero cost. rw memory must be backed somehow, which us much more expensive.

I submitted a patch

Re: Embedding Binary Objects in C

#119

Post by loeg »

haberman wrote: post #13

If we're in the realm of "non-standard linker tricks"... Compilers will concatenate sections of the same name. You can use this trick to produce a concatenation of arrays across several files: $ cat t1.c __attribute__((section("some_array"))) int a[] = {1, 2, 3}; $ cat t2.c __attribute__((section("some_array"))) int b[] = {4, 5, 6}; $ cat t.c #include extern const int __start_some_array; extern const int __stop_some_…

The BSDs define some macros in sys/linker_set.h to make this slightly easier. The mechanism is used somewhat widely in both kernel and userspace in FreeBSD.

Re: Embedding Binary Objects in C

#120

Post by loeg »

RealityVoid wrote: post #66
monocasa wrote: post #44

Earlier quoted context omitted.

It's unspecified, and I've seen it change over time as the linker changes it's data structures.

This is true, to my knowledge, I would NOT relie on the order to be always consistent. But I think you can enforce the order in the linker file, i.e. load the arrays one after the other, and then the order should keep constant. But you still have several places that you need to keep synchronized and, really, most people don't look in the linker file at all.

You can sort the array during program initialization, if you care about the order.
Post reply on HN