Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Stop Memsetting Structures

anmolsarma.in

Post reply on HN
Print view View original

131–140 of 277 posts

Re: Stop Memsetting Structures

#131

Post by shaklee3 »

repsilat wrote: post #90
kllrnohj wrote: post #81

Earlier quoted context omitted.

> Do you write a separate serialization function for every single struct and sub-struct you send over a network? And then continue maintaining every single one to keep it consistent whenever a field is added/removed/modified in any struct? Yes? Of course you do that? Or pick one of a hundred libraries that do that for you - protobuf, cap'n proto, flatbuffers, etc...

That's the protobuf approach, but things like Capnp are different, right? Isn't its wire format its in-memory format? I thought it was closer to the "straight memcpy" approach. Protobuf serialization functions can turn into a lot of generated code too (read: big binaries) and stomp all over your icache, because they're different for each type. More "data-driven" approaches using type-generic functions with some runti…

There are dozens, if not hundreds of serialization libraries. None of them are going to compare to something like sending a raw struct on the wire, if you need absolute performance. Probably the biggest advantage of things like grpc, flatbuffers, capnproto, etc, are they provide serialization libraries for many different languages. Sending a raw struct will be up to the implementer to get the other side correct.

And by sticking with the serialization libraries with higher performance, like flatbuffers and capnproto, you also lose a little bit of future flexibility, since its typically harder to change the format of these structures compared to higher level libraries.

Re: Stop Memsetting Structures

#132

Post by shaklee3 »

v_lisivka wrote: post #95
Someone wrote: post #12

Earlier quoted context omitted.

Sending structures that include padding or that you don’t know the exact layout of over the network is another thing you shouldn’t do, period. Code doing that may break with a compiler change or a compilation flag change on one end of the connection, even if you can guarantee both ends use the same CPU.

RDMA (Infiniband) does DMA to/from network. Why we should not use RDMA?

RDMA, both roce and infiniband have some significant limitations, specifically at higher rates. But they're the best we have right now for direct dma.

Re: Stop Memsetting Structures

#133

Post by Gibbon1 »

jasonhansel wrote: post #9
_bxg1 wrote: post #5

Earlier quoted context omitted.

Same. But also, I feel like most of The Force's efforts are directed at C++. I could be wrong.

C++ programmers need to be converted. C programmers just need sympathy.

Most C programmers I know are generally happy.

C++ programmers seem 'stressed'

Re: Stop Memsetting Structures

#134

Post by jdblair »

anyfoo wrote: post #2

Does that also zero the padding that may exist between elements of the structure (or at its end)? Because if not, not memset'ing will open you up to all kinds of info leaks, especially if you plan to send that struct over the network (or to another process, or from kernel to userspace...). EDIT: Actually I have to apologize, because I’m not certain what the C standard says about whether the padding bytes can change w…

But if you're slamming in-memory structs over the wire, you're using packed structs. You explicitly don't want padding when you use structs that way.

Whether you should do this in 2019 is another question, but sometimes it's still the easiest way to implement a binary protocol.

Re: Stop Memsetting Structures

#135

Post by nineteen999 »

jasonhansel wrote: post #127
nineteen999 wrote: post #119

Earlier quoted context omitted.

We don't really need sympathy, we get paid a lot and can afford to drown our sorrows.

At least until the next zero-day caused by a buffer overrun. (Kidding, kidding.)

I know you're kidding, but C programmers also get paid to provide security fixes. There's plenty of C code out there that doesn't interface with untrusted data/input, or does it in a safe way. So I'm not really in fear of losing my job over a simple buffer overflow.

Re: Stop Memsetting Structures

#136

Post by cperciva »

tedunangst wrote: post #112
cperciva wrote: post #7

Another reason to not memset structures: In the code struct foo { void * bar; } baz; ... memset(&baz, 0, sizeof(struct foo)); assert(baz.bar == NULL); it's possible for the assertion to fail, since NULL it not guaranteed to be represented in memory by a zero bytes.

Are you accepting bug reports for such issues in tarsnap? :)

Yes. If you look at the commit history you'll find that we've fixed a few of them already.

Re: Stop Memsetting Structures

#137

Post by taneq »

noobermin wrote: post #39
mehrdadn wrote: post #21

Earlier quoted context omitted.

> Sending structures that include padding or that you don’t know the exact layout of over the network is another thing you shouldn’t do, period. What if you do know the exact layout and it happens to have a gap between fields?

Tbh I'm not a network person, but my naive brain says instead of casting to char* and writing to the buffer, couldn't you just write out individual fields to a buffer so you control every byte? That seems smarter and safer.

That's what I do these days for simple stuff, or for stuff which needs to talk to nonstandard hardware (eg. talking to a PLC, where I need to know the exact bit layout of the packet in order to unpack it manually on the other side).

If you're doing a large project on supported platforms it's probably worth using a pre-existing serialisation library like protobuffers or something.

Re: Stop Memsetting Structures

#138

Post by cperciva »

hedora wrote: post #126
cperciva wrote: post #28

Earlier quoted context omitted.

> you'd be hard pressed to find a system in 2019 where NULL != (void *)0 You'd be hard pressed to find a system at any time where NULL is not equal to a compile-time constant zero. ;-)

> The Prime 50 series used segment 07777, offset 0 for the null pointer, BAM!!! (All I had to do was click on the link in the comment you replied to, but the pedantry in this subthread is glorious, and I had to contribute — downvoters, this is your chance to school me on HN etiquette!)

You're missing the point. "(void *)0" is guaranteed to be a NULL pointer, even when said pointer is not represented in memory by zeroes.

Re: Stop Memsetting Structures

#139

Post by Gibbon1 »

teej wrote: post #79
nocman wrote: post #44

Earlier quoted context omitted.

No, you are not the only one. The author of that code is trying to make it explicit -- "yes, I want to allow reuse of local addressses". Some will prefer this, some won't. I personally like the use of the variable, though I think it's not worth getting all bent out of shape about it one way or another. But I guess we all have our pet peeves, and I know I'm guilty of getting all bent out of shape about things that ult…

Code is read many times more than it is written. If someone scrutinizes code for clear intent, I don’t think that should be considered a “pet peeve” or some quirk to be explained away.

I feel that code that needs to be read a lot is badly named, commented or dodgy.

Re: Stop Memsetting Structures

#140

Post by Gibbon1 »

deathanatos wrote: post #56
userbinator wrote: post #34

Because it’s 2019! It really irritates me when people give arguments like this. I seriously don't care what year it is, I care that the code I write works. A lot of projects still use C89 because they want the widest portability. C99 isn't very new, but especially in the embedded and other niche industries, C89 (often with some extensions) is all you get. That said, "= { 0 };" is almost always usable instead of a mem…

> but especially in the embedded and other niche industries, C89 (often with some extensions) is all you get. Embedded and niche industries are forcing their programmers to work with 30 year old tooling? (Instead of 20 year tooling?) I think at some point, writing yourself a transpiler from C99 to C89 becomes profitable… (The JS community did it for a lot less than that w/ babel…)

I remember reading a doc's for a library targeting ARM Cortex processors that crowed about it being C89 compatible.

I pretty much spit on my keyboard.

I also knew the code base was going to be kinda trashy and it was.

Post reply on HN