Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Show HN: Hexi – Modern header-only network binary serialisation for C++

github.com

Post reply on HN
Print view View original

1–10 of 49 posts

Show HN: Hexi – Modern header-only network binary serialisation for C++

#1

Post by Chaosvex »

Over the last few years, I've needed an easy way to quickly serialise and deserialise various network protocols safely and efficiently. Most of the libraries that existed at the time were either quite heavy, had less than stellar performance, or were an abstraction level above what I was looking for.

I decided to put together my own class to do the job, starting with an easy, low-overhead way to move bytes in and out of arbitrary buffers. Along the way, it picked up useful bits and pieces, such as buffer structures and allocators that made the byte shuffling faster, often being able to do it with zero allocations and zero copies. Safety features came along to make sure that malicious packet data or mistakes in the code wouldn't result in segfaults or vulnerabilities.

It's become useful enough to me that I've packaged it up in its own standalone library on the chance that it might be useful to others. It has zero dependencies other than the standard library and has been designed for quick integration into any project within minutes, or seconds with a copy paste of the amalgamated header. It can be used in production code but it's also ideal for for those that want to quickly hack away at binary data with minimal fuss.

Show HN: Hexi – Modern header-only network binary serialisation for C++
github.com

Re: Show HN: Hexi – Modern header-only network binary serialisation for C++

#3

Post by Chaosvex »

gregschlom wrote: post #2

Semi off-topic, but I just love the header image and the advice frog in the readme. Makes reading the documentation more fun and enjoyable.

Thanks, that was the hope! :)

I would have liked a different froggy reaction for each section but the project budget was zero. :^)

Re: Show HN: Hexi – Modern header-only network binary serialisation for C++

#4

Post by nightowl_games »

Wow that api looks fantastic! Bravo!

I'd like to read an even more thorough overview of how it works and all the gotchas before I'd consider using this 'in production' but the API looks very easy to use and very elegant.

EDIT: just hit the section on portability, seems like you would always have to use that API, yeah? I feel like when you are writing network code you simply have to make it portable from the get-go. I guess I'm always thinking about having it run on client machines.

Re: Show HN: Hexi – Modern header-only network binary serialisation for C++

#6

Post by Chaosvex »

nightowl_games wrote: post #4

Wow that api looks fantastic! Bravo! I'd like to read an even more thorough overview of how it works and all the gotchas before I'd consider using this 'in production' but the API looks very easy to use and very elegant. EDIT: just hit the section on portability, seems like you would always have to use that API, yeah? I feel like when you are writing network code you simply have to make it portable from the get-go. I…

Thanks. The documentation could definitely be fleshed out with some more examples.

You'd likely want to always use that API (or layer something on top of it) unless you're in control of both ends and know they were built with the same toolchain & settings. One area where I've skipped over it is by writing a basic code gen tool (albeit unfinished as most personal projects) that generates the serialisation functions at compile-time from a very basic DSL that describes the network structures (of a game protocol I don't control). If it detects that the current toolchain is going to generate a binary-compatible struct layout and there aren't any variable length fields in there (no strings, basically), it'll generate a memcpy (via using get/put on the stream) rather than per-field (de)serialisation. If it can guarantee alignment of the buffer, which is a tougher requirement to meet, it'll give you a view directly into the network buffer so you effectively have zero-overhead deserialisation. Very much a work in progress but there's scope for making things quite efficient with just a few basic building blocks.

Re: Show HN: Hexi – Modern header-only network binary serialisation for C++

#9

Post by Chaosvex »

codedokode wrote: post #8

It doesn't look like zero-copy though in this example: UserPacket packet; stream >> packet; That is at least one copy.

Correct but it doesn't claim to be zero-copy overall. Apologies if it was misleading. In the README, zero-copy is mentioned in context of using view() and span() to obtain views into the buffer, which does allow for it. Very much a "there be dragons but if you're sure..." feature but it's there as an option.

I've also built some tooling on top that makes use of those functions to do zero-copy deserialisation where viable, so it is possible in the right scenarios with a bit of work but it definitely isn't going to always fit.

Re: Show HN: Hexi – Modern header-only network binary serialisation for C++

#10

Post by Jyaif »

Your lib requires manually creating both a serializing and deserializing function. If the functions are out of sync, bad things happen.

Consider copying Cereal, which solves this problem by requiring you to create a single templated function ( https://uscilab.github.io/cereal/ )

Post reply on HN