Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

The Byte Order Fiasco

justine.lol

Post reply on HN
Print view View original

231–240 of 378 posts

Re: The Byte Order Fiasco

#231

Post by lanstin »

vlmutolo wrote: post #131

> If you program in C long enough, stuff like this becomes second nature, and it starts to almost feel inappropriate to even have macros like the above, since it might be more appropriately inlined into the specific code. Since there have simply been too many APIs introduced over the years for solving this problem. To name a few for 32-bit byte swapping alone: bswap_32, htobe32, htole32, be32toh, le32toh, ntohl, and…

The point is that keeping the distinction clear in your head between numeric semantics and sequence of octets semantics makes the problem universally tractible. You have a data structure where with a numeric value. Here you have a sequence of octets described by some protocol formalism, BNF in the old days. The mapping from one to the other occurs in the math between octets and numeric values and the various network protocols for representing numbers. There are many more choices than just big endian or little endian. Could be ASN infinite precision ints. Could be 32 bit IEEE floats or 64 bit IEEE floats. The distinction is universal between language semantics and external representations.

This is why people that memcpy structs right into the buf get such derision, even if it’s faster and written for a mono-Implementation of a language semantics. It is sloppy thought made manifest.

Re: The Byte Order Fiasco

#232

Post by [deleted] »

st_goliath wrote: post #67

FWIW there is a on various BSDs that contains "beXXtoh", "leXXtoh", "htobeXX", "htoleXX" where XX is a number of bits (16, 32, 64). That header is also available on Linux, but glibc (and compatible libraries) named it instead. See: man 3 endian ( https://linux.die.net/man/3/endian ) Of course it gets a bit hairier if the code is also supposed to run on other systems. MacOS has OSSwapHostToLittleIntXX, OSSwapLittleToH…

[deleted]

Re: The Byte Order Fiasco

#233

Post by jart »

lanstin wrote: post #216
jart wrote: post #144

Earlier quoted context omitted.

Even the individual unicode codepoints themselves are variable width if we consider that things like cjk and emoji take up >1 monospace cells.

Every time I see one of these threads, my gratitude to only do backend grows. Human behavior is too complex, let the webdevs handle UI, and human languages are too complex, not sure what speciality handles that. Give me out of order packets and parsing code that skips a character if the packet length lines up just so any day. I am thankful that almost all the Unicode text I see is rendered properly now, farewell the…

I think we really have the iPhone jailbreakers to thank for that. U.S. developers were allergic almost offended by anything that wasn't ASCII and then someone released an app that unlocked the emoji icons that Apple had originally intended only for Japan. Emoji is defined in the astral planes so almost nothing at the time was capable of understanding them, yet were so irresistible that developers worldwide who would otherwise have done nothing to address their cultural biases immediately fixed everything overnight to have them. So thanks to cartoons, we now have a more inclusive world.

Re: The Byte Order Fiasco

#234

Post by nine_k »

captainmuon wrote: post #41

It is a ridiculous feature of modern C that you have to write the super verbose "mask and shift" code, which then gets compiled to a simple `mov` and maybe a `bswap`. Wheras, the direct equivalent in C, an assignment with a (type changing) cast, is illegal. There is a huge mismatch between the assumptions of the C spec and actual machine code. One of the few reasons I ever even reached to C is the ability to slurp in…

I suspect you might like C--.

https://en.m.wikipedia.org/wiki/C--

Re: The Byte Order Fiasco

#235

Post by jeffreygoesto »

professoretc wrote: post #191
jeffreygoesto wrote: post #189

Earlier quoted context omitted.

Wouldn't that cast be UB because it is type punning?

char* is a allowed to alias to other pointer types.

Hm. Afsik, you are always allowed to convert _to_ a char, but _from_ is not ok in general. See i.e. [0]

[0] https://gist.github.com/shafik/848ae25ee209f698763cffee272a5...

Re: The Byte Order Fiasco

#236

Post by raphlinus »

jstimpfle wrote: post #213
raphlinus wrote: post #194

Earlier quoted context omitted.

I agree with the bulk of this post. Re the anecdata at the end. Have you ever run your code through the sanitizers? I have. CVE-2016-2414 is one of my battle scars, and I consider myself a pretty good programmer who is aware of security implications.

Very little, quite frankly. I've used valgrind in the past, and found very few problems. I just ran -fsanitize=undefined for the first time on one of my current projects, which is an embedded network service of 8KLOC, and with a quick test covering probably 50% of the codepaths by doing network requests, no UB was detected (I made sure the sanitizer works in my build by introducing a (1 Admittedly I'm not the type of…

This one wasn't specifically "betrayal by compiler," but it was a confusion between signed and unsigned quantities for a size field, which is very similar to the UB exhibited in OP.

Also, the fact that you can't see the problem is actually evidence of how insidious these problems are :)

The rules for this are arcane, and, while the solution suggested in OP is correct, it skates close to the edge, in that there are many similar idioms that are not ok. In particular, (p[1] << 8) & 0xff00, which is code I've written, is potentially UB (hence "mask, and then shift" as a mantra). I'd be surprised if anyone other than jart or someone who's been part of the C or C++ standards process can say why.

Re: The Byte Order Fiasco

#237

Post by 8jy89hui »

amelius wrote: post #20
amelius wrote: post #3

Byte order is one of the great unnecessary historical fuck ups in computing. A similar one is that signedness of char is machine dependent. It's typically signed on Intel and unsigned on ARM. Sigh!

By the way, mathematicians also have their fuck ups: https://tauday.com/tau-manifesto

For anyone curious or who is still attached to pi, here is a response to the tau manifesto:

https://blog.wolfram.com/2015/06/28/2-pi-or-not-2-pi/

Re: The Byte Order Fiasco

#238

Post by pm215 »

captainmuon wrote: post #41

It is a ridiculous feature of modern C that you have to write the super verbose "mask and shift" code, which then gets compiled to a simple `mov` and maybe a `bswap`. Wheras, the direct equivalent in C, an assignment with a (type changing) cast, is illegal. There is a huge mismatch between the assumptions of the C spec and actual machine code. One of the few reasons I ever even reached to C is the ability to slurp in…

So in your 'machine model is the physical machine' flavour, should "I cast an unaligned pointer to a byte array to int32_t and deref" on SPARC (a) do a bunch of byte-load-and-shift-and-OR or (b) emit a simple word load which segfaults? If the former, it's not what the physical machine does, and if the latter, then you still need to write the code as "some portable other thing". Which is to say that the spec's UB here is in service of "allow the compiler to just emit a word load when you write *(int32_t)p".

What I think the language is missing is a way to clearly write "this might be unaligned and/or wrong endianness, handle that". (Sometimes compilers provide intrinsics for this sort of gap, as they do with popcount and count-leading-zeroes; sometimes they recognize common open-coded idioms. But proper standardised support would be nicer.)

Re: The Byte Order Fiasco

#239

Post by jart »

patrakov wrote: post #228
tails4e wrote: post #6

Ubsan should default on. If people don't like it, then they should be made turn it off with a switch, so at least it's more likely to be run than not run. Could save a huge amount of time debugging when compilers or architecture changes. Without it, I'd say many a programmer would be caught by these subtleties in the standard. Coming from a HW background (Verilog) I'd more naturally default to masking and shifting wh…

There was a blog post and a FOSDEM presentation by (misguided) Gentoo developers a few years ago, and it was retracted, because sanitizers add their own exploitable vulnerabilities due to the way they work. https://blog.hboeck.de/archives/879-Safer-use-of-C-code-runn... https://www.openwall.com/lists/oss-security/2016/02/17/9

Sanitizers have the ability to bring Rust-like safety assurances to all the C/C++ code that exists. The fact that existing ASAN runtimes weren't designed for setuid binaries shouldn't dissuade us from pursuing those benefits. We just need a production-worthy runtime that does less things. For example, here's the ASAN runtime that's used for the redbean web server: https://github.com/jart/cosmopolitan/blob/master/libc/intrin...

Re: The Byte Order Fiasco

#240

Post by user-the-name »

klodolph wrote: post #171
captainmuon wrote: post #41

It is a ridiculous feature of modern C that you have to write the super verbose "mask and shift" code, which then gets compiled to a simple `mov` and maybe a `bswap`. Wheras, the direct equivalent in C, an assignment with a (type changing) cast, is illegal. There is a huge mismatch between the assumptions of the C spec and actual machine code. One of the few reasons I ever even reached to C is the ability to slurp in…

You don’t have to mask and shift. You can memcpy and then byte swap in a function. It will get inlined as mov/bswap. Practically speaking, common compilers have intrinsics for bswap. The memcpy function can be thought of as an intrinsic for unaligned load/store.

When do you byte swap?
Post reply on HN