Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

The Byte Order Fiasco

justine.lol

Post reply on HN
Print view View original

281–290 of 378 posts

Re: The Byte Order Fiasco

#281

Post by remexre »

MaxBarraclough wrote: post #278
pornel wrote: post #260

Earlier quoted context omitted.

Of course nobody wants C to backstab them with UB, but at the same time programmers want compilers to generate optimal code. That's the market pressure that forces optimizers to be so aggressive. If you can accept less optimized code, why aren't you using tcc? The idea of C that "just" does a straightforward machine translation breaks down almost immediately. For example, you'd want `int` to just overflow instead of…

> nobody wants C to backstab them with UB, but at the same time programmers want compilers to generate optimal code The value of compiler optimization isn't the same thing as the value of having extensive undefined behaviour in a programming language. Rust and Ada perform about the same as C, but lack C's many footguns. > indexing `arr[i]` can't use 64-bit memory addressing modes What do you mean here?

Typically, the assembly instruction that would do the read in arr[i] can do something like:

    x = *(y + z);
where y and z are both 64-bit integers. If I had

    int arr[1000];
    initialize(&arr);
    int i = read_int();
    int x = arr[i];
    print(x);
then to get x I'd need to do something like,

    tmp = i * 4;
    tmp1 = (uint64_t)tmp;
    x = *(arr + tmp1);
Which, since i is signed, can't just be a cheap shift, and then needs to be upcasted to a uint64_t (which is cheap, at least).

Re: The Byte Order Fiasco

#282

Post by einpoklum »

This is valid code in C++20:

    if constexpr (std::endian::native == std::endian::big) {
        std::cout
Doesn't solve everything, but it's saner even if what you're writing is C-style low-level code.

Re: The Byte Order Fiasco

#283

Post by jcelerier »

jart wrote: post #218
pjmlp wrote: post #133

Earlier quoted context omitted.

Really? Apparently the first year students at my university didn't had any issue going from Standard Pascal to C++, in the mid-90's. Proper C++ was taught using our string, vector and collection classes, given that we were still a couple of years away from ISO C++ being fully defined. C style programming with low level tricks were only introduced later as advanced topics. Apparently thousands of students managed to g…

Well there's a reason universities switched to Java when teaching algorithms and containers after the 90's. C++ is a weaker abstraction that encourages the kind of curiosity that's going to cause a student's brain to melt the moment they try to figure out how things work and encounter the sorts of demons the coursework hasn't prepared them to face. If I was going to teach it, I'd start with octal machine codes and wo…

> Well there's a reason universities switched to Java when teaching algorithms and containers after the 90's

Where ? I learned algorithms in C and C++ (and also a bit in Caml and LISP) and I was in university 2011-2014

Re: The Byte Order Fiasco

#284

Post by thechao »

froh wrote: post #252
infradig wrote: post #249

Earlier quoted context omitted.

There are equivalent functions in C too. The point of the article is about not using them. So how would you implement the above functions in Rust would be more pertinent.

isnt the point to be careful when implementing them? so the compiler detects the intention to byteswap? when we ported little endian x86 Linux to the big endian mainframe we sprinkled hton/ntoh all over the place, happily so. they are the way to go and they should be implemented properly, not be replaced by a homegrown version. all that said, I'm surprised 64bit htonll and ntohll are not standard yet. anybody knows w…

Blech. I learned to program (around ‘99) by implementing the crusty old FCS1.0 format, which allows for aggressively weird wire formats. Our machine was a PDP-11/72 with its head sawzalled off and custom wire wrap boards dropped in. The “native” format (coming from analog) was 2143 order as a 36b packet. The bits were [8,0:7] (using verilog notation). However, sprinkled randomly in the binary header were chunks of 7- and 8- bit ANSI (packed) and some mutant knockoff 6-bit EBCDIC.

The original listing was written by “Jennifer — please call me if you have troubles”, an undergraduate from MIT. It was hand-assembled machine code, in a neat hand in a big blue binder. That code ran non-stop except for a few hurricanes from 1988 until 2008; bug-free as far as I could tell. Jennifer last-name-unknown, you were my idol & my demon!

I swore off programming for nearly a year after that.

Re: The Byte Order Fiasco

#285

Post by saagarjha »

jeffreygoesto wrote: post #189
nly wrote: post #23

This is why, in 2021, the mantra that C is a good language for these low level byte twiddling tasks needs to die. Dealing with alignment and endianness properly requires a language that allows you to build abstractions. The following is perfectly well defined in C++, despite looking like almost the same as the original unsafe C: #include #include using namespace boost::endian; unsigned char b[5] = {0x80,0x01,0x02,0x0…

Wouldn't that cast be UB because it is type punning?

No, because no punning exists here. The code is C++, so this calls a conversion function that likely does the bit manipulation internally in a legal way.

Re: The Byte Order Fiasco

#286

Post by remexre »

userbinator wrote: post #261
jart wrote: post #243

Earlier quoted context omitted.

I agree but language of the standard very unambiguously lets them do it. Quoth X3.159-1988 * Undefined behavior --- behavior, upon use of a nonportable or erroneous program construct, of erroneous data, or of indeterminately-valued objects, for which the Standard imposes no requirements. Permissible undefined behavior ranges from ignoring the situation completely with unpredictable results, to behaving during transla…

So let's fix our code. No; I say we force the compiler writers to fix their idiotic assumptions instead of bending over backwards to please what's essentially a tiny minority. There's a lot more programmers who are not compiler writers. The standard is really a minimum bar to meet, and what's not defined by it is left to the discretion of the implementers, who should be doing their best to follow the "spirit of C", w…

For essentially every form of UB that compilers actually take advantage of, there's a real program optimization benefit. Are there any particular UB cases where you think the benefit isn't worth it, or it should be implementation-specific behavior instead of undefined behavior?

Re: The Byte Order Fiasco

#287

Post by jart »

froh wrote: post #252
infradig wrote: post #249

Earlier quoted context omitted.

There are equivalent functions in C too. The point of the article is about not using them. So how would you implement the above functions in Rust would be more pertinent.

isnt the point to be careful when implementing them? so the compiler detects the intention to byteswap? when we ported little endian x86 Linux to the big endian mainframe we sprinkled hton/ntoh all over the place, happily so. they are the way to go and they should be implemented properly, not be replaced by a homegrown version. all that said, I'm surprised 64bit htonll and ntohll are not standard yet. anybody knows w…

Functions like ntohl and htonl are the biggest blemish in the design of the Berkeley Sockets API because it's defined to read memory off the wire without deserializing it. Those functions shouldn't have been invented for the reasons described in the linked blog posts. The C standard isn't going to evolve to include functions that only exist to accommodate code that misunderstands the standard.

Re: The Byte Order Fiasco

#288

Post by Animats »

Aeolun wrote: post #277
musicale wrote: post #276

Earlier quoted context omitted.

> game on a mainframe Maybe your program isn't a game. Maybe you have to deal a server that uses Power, or an embedded system that uses PowerPC (or ARM or MIPS in big-endian mode). Maybe you're running on an older architecture (SPARC, PowerPC, 68K.) Maybe you have to deal with a pre-defined data format (e.g. TCP/IP packet headers) that uses big-endian byte ordering for some of its components.

That’s theoretically possible. But I’d be very interested in why. Especially if you are doing anything involving networking.

Because it makes it clear what's going on. Most of those functions just generate a move, but it's the correct move.

I had to read through excessively-clever C++ code that did the same thing to figure out what conversions were happening, then re-express it in Rust. I'm re-implementing a legacy mess that people are afraid to work on. As it happens, in this message system, some items, mainly packet sequence numbers, are big-endian, because they were following what IP and UDP do, and everything else is little endian.

I know how to do this with shifts and masks, and I've done things like that when programming in assembly. That was a long time ago. There's been progress in how to write programs.

Re: The Byte Order Fiasco

#289

Post by Negitivefrags »

Animats wrote: post #288
Aeolun wrote: post #277

Earlier quoted context omitted.

That’s theoretically possible. But I’d be very interested in why. Especially if you are doing anything involving networking.

Because it makes it clear what's going on. Most of those functions just generate a move, but it's the correct move. I had to read through excessively-clever C++ code that did the same thing to figure out what conversions were happening, then re-express it in Rust. I'm re-implementing a legacy mess that people are afraid to work on. As it happens, in this message system, some items, mainly packet sequence numbers, are…

Obviously if you are interacting with an old protocol that uses network byte order for some things, you will need to use these functions.

But what's the argument for using them in new game code?

The code will not be running anywhere that has big-endianness. No current platform a game could run on uses it, and I can't imagine a scenario where a new platform would come into existance and use it either.

If you insist on using network-byte order anyway, then you have to do an extra bswap op for each bit of data you send. Sure, the cost of that is super minor and probably not worth worrying about.

But the bigger cost is that you can't just send whole structures at a time. You have to individually serialise each thing. Now you have to have a whole serialisation concept. You have to have some way of enumerating all the fields. You have to walk all the structures. What a pain.

If you want to send a thing over the network, just send it.

Re: The Byte Order Fiasco

#290

Post by jstimpfle »

sdenton4 wrote: post #226
jstimpfle wrote: post #175

Earlier quoted context omitted.

I find you missed the point of the post and the issues described in it. In my estimation, libraries like boost are way too big and way too clever and they create more problems than they solve. Also, they don't make me happy. You're overfocusing on a "problem" that is almost completely irrelevant for most of programming. Big endian is rare to be found (almost no hardware to be found, but some file formats and networki…

Every day I spend futzing around with endianness is a day I'm not solving 'real' problems. These things are a distraction and a complete waste of developer time: It should be solved 'once' and only worried about by people specifically looking to improve on the existing solution. If it can't be handled by a library call, there's something really broken in the language. (imo, both c and cpp are mainly advocated by peop…

But that's the point: No one spends a day futzing around with endianness, and there are in fact functions for swapping endianness. You can just call them, no need to hide the swap in a pointer cast expression to a type that has the dereferencing operator overloaded.
Post reply on HN