Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

The Byte Order Fiasco

justine.lol

Post reply on HN
Print view View original

301–310 of 378 posts

Re: The Byte Order Fiasco

#301

Post by bitwize »

dragonwriter wrote: post #270
bitwize wrote: post #269

Earlier quoted context omitted.

Hint: The reason why it's called "endianness" comes from the novel Gulliver's Travels , in which the neighboring nations of Lilliput and Blefuscu went to bitter, bloody war over which end to break your eggs from: the big end or the little end. The warring factions were also known as Big-Endians and Little-Endians, and each thought themselves superior to the dirty heathens on the other side. If one side were objective…

> if there were an inherent advantage to breaking your egg from one side or the other, would there be a war at all? Fascism vs. not-fascism, Stalinist Communism vs. Western Capitalism, Islamism vs. liberal democracy... I’m not sure “the existence of war around a divide in ideas proves that neither sides ideas are correct” is a particularly comfortable maxim to consider the ramifications of.

Two similar societies warring over a trivial idea probably means neither is right. Swift's Big Endians and Little Endians are a satire of the Catholic-Anglican schism in England.

Re: The Byte Order Fiasco

#302

Post by klodolph »

user-the-name wrote: post #240
klodolph wrote: post #171

Earlier quoted context omitted.

You don’t have to mask and shift. You can memcpy and then byte swap in a function. It will get inlined as mov/bswap. Practically speaking, common compilers have intrinsics for bswap. The memcpy function can be thought of as an intrinsic for unaligned load/store.

When do you byte swap?

24 hours a day, man. I'm always byte swapping.

(I'm not sure how to answer the question... what do you mean, "when?")

Re: The Byte Order Fiasco

#303

Post by klodolph »

BeeOnRope wrote: post #244
klodolph wrote: post #171

Earlier quoted context omitted.

You don’t have to mask and shift. You can memcpy and then byte swap in a function. It will get inlined as mov/bswap. Practically speaking, common compilers have intrinsics for bswap. The memcpy function can be thought of as an intrinsic for unaligned load/store.

How do you detect if a byte swap is needed? I.e. wether the (fixed) wire endianness matches the current platform endianness?

Using the preprocessor, something like this:

    uint32_t swap32(uint32_t x) { ... }

    #if __BYTE_ORDER__ == __ORDER_LITTLE_ENDIAN__
    uint32_t swap32be(uint32_t x) { return swap32(x); }
    #elif __BYTE_ORDER__ == __ORDER_BIG_ENDIAN__
    uint32_t swap32be(uint32_t x) { return x; }
    #else
    #error "Unknown endian"
    #endif
You can make the preprocessor condition broader if you care about more compilers and more platforms. Yes, I'm making assumptions about which platforms you want to target... which is fine. No, I don't care about your PDP-11, nor about dynamically changing your endian at runtime. Nearly any problem in C can be made arbitrarily difficult if you care about sufficiently bizarre platforms, or ask that people write code that is correct on any theoretical conforming C implementation. So we pick some platforms to support.

The above code is fairly simple. You can separate the part where you care about unaligned memory access and the part where you care about endian.

Some irrelevant details left out above.

Re: The Byte Order Fiasco

#304

Post by bombcar »

mrlonglong wrote: post #165
bombcar wrote: post #96

Earlier quoted context omitted.

https://stackoverflow.com/questions/5185551/why-is-x86-littl... It simplifies certain instructions internally. Practically everything is little endian because x86 won. > And if you think about a serial machine, you have to process all the addresses and data one-bit at a time, and the rational way to do that is: low-bit to high-bit because that’s the way that carry would propagate. So it means that [in] the jump instr…

And does middle endian even exist?

I suspect it does somewhere as a system that had words as the main addressable memory but also allowed byte addressing could have little endian double words but big endian ordering inside the words (as bytes).

Re: The Byte Order Fiasco

#305

Post by bombcar »

CydeWeys wrote: post #158
kangalioo wrote: post #140

Earlier quoted context omitted.

I feel like big endian is more _intuitive_ because that's what our number notation has evolved to be. But more _natural_ is little endian because, well, it's just more straightforward to have the digits' magnitude be in ascending order (2^0, 2^1, 2^2, 2^3...) instead of putting it in reverse. Plus you encounter less roadblocks in practice with little endian (e.g. address changes with casts) which is often a sign of g…

I'm curious how you're defining "natural", and if you think ISO-8601 is the reverse of "natural" too. All human number systems I've ever seen write numbers out as big Endian (yes, even Roman numerals), so I'm really struggling to see how that wouldn't be considered natural.

Counting out change is little endian - usually you start cents then dollars.

I wonder if we went big endian “by mistake” with Arabic numerals given that Arabic is written right to left.

Some ancient texts have “four and twenty” which is little endian.

We also add commas to large numbers to help with a human processing problem - you have to get to the end of the number to know what the first digit represents and then count backwards (groups of three help).

Re: The Byte Order Fiasco

#306

Post by chowells »

remexre wrote: post #286
userbinator wrote: post #261

Earlier quoted context omitted.

So let's fix our code. No; I say we force the compiler writers to fix their idiotic assumptions instead of bending over backwards to please what's essentially a tiny minority. There's a lot more programmers who are not compiler writers. The standard is really a minimum bar to meet, and what's not defined by it is left to the discretion of the implementers, who should be doing their best to follow the "spirit of C", w…

For essentially every form of UB that compilers actually take advantage of, there's a real program optimization benefit. Are there any particular UB cases where you think the benefit isn't worth it, or it should be implementation-specific behavior instead of undefined behavior?

Most performance wins from UB come from removing code that someone wrote intentionally. If that code wasn't meant to be run, it shouldn't be written. If it was written, it should be run.

Now obviously there are lots of counter-examples for that. You can probably list ten in a minute. But it should be the guiding philosophy of compiler optimizations. If the programmer wrote some code, it shouldn't just be removed. If the program would be faster without that code, the programmer should be the one responsible for deciding whether the code gets removed or not.

Re: The Byte Order Fiasco

#307

Post by raphlinus »

jstimpfle wrote: post #292
raphlinus wrote: post #236

Earlier quoted context omitted.

This one wasn't specifically "betrayal by compiler," but it was a confusion between signed and unsigned quantities for a size field, which is very similar to the UB exhibited in OP. Also, the fact that you can't see the problem is actually evidence of how insidious these problems are :) The rules for this are arcane, and, while the solution suggested in OP is correct, it skates close to the edge, in that there are ma…

> the fact that you can't see the problem is actually evidence of how insidious these problems are I've looked for a while now, but still can't see it, would you be willing to share? > (p[1] With p[1] being uint8_t? Because then I cannot imagine why, and also fail to see a reason to apply the 0xff00 mask here. If this is for int8_t instead, the problem you are alluding to is sign extension? If p[1] gets promoted to a…

Yes, I was assuming it was char *, as in the OP, which can be signed. And any left shift of a negative quantity is UB in C (I'm not sure if this is fixed in recent C++), it doesn't have to be what's commonly thought of as overflow.

Re: The Byte Order Fiasco

#308

Post by pjmlp »

saagarjha wrote: post #280
pjmlp wrote: post #136

Earlier quoted context omitted.

Typical C culture, you would also expect that by now something like SDS would be part of the standard as well. https://github.com/antirez/sds

Adding API that introduces an entirely new string model that is incompatible with the rest of the standard library seems like a nonstarter.

Yeah, because that is the only way that they could ever do it, assuming WG14 would ever care about a safer C.

Re: The Byte Order Fiasco

#309

Post by [deleted] »

tsimionescu wrote: post #267
cbmuser wrote: post #75

Earlier quoted context omitted.

> One of the few reasons I ever even reached to C is the ability to slurp in data and reinterpret it as a struct, or the ability to reason in which registers things will show up and mix in some `asm` with my C. Which results in undefined behavior according to the C ISO standard. Quote: “2 All declarations that refer to the same object or function shall have compatible type; otherwise, the behavior is undefined.” From…

It should be perfectly fine to do this: union reinterpret { char raw[100]; struct myStruct interpreted; } example; read(fd, &example.raw) struct myStruct dest = interpreted; This is standard-compliant C code, and it is a common way of reading IP addresses from packets, for example.

[deleted]

Re: The Byte Order Fiasco

#310

Post by joshuamorton »

userbinator wrote: post #300
wruza wrote: post #296

Earlier quoted context omitted.

force the compiler writers to fix their idiotic assumptions instead of bending over backwards to please what's essentially a tiny minority As far as I understand it, they do neither. Transforming an AST to any level of target code is not done by handcrafted recipes, but instead is feeded into efficient abstract solvers which have these assumptions as an operational detail. E.g.: p = &x; if (p != &x) foo(); // optimiz…

In your example, why should it optimise out the second case? Maybe foo() changed p so it's no longer null. Compiler writers do not format disks just to punish your UBs. IMHO if the compiler exploiting UB is leading to counterintuitive behaviour that's making it harder to use the language, the compiler is the one that needs fixing, regardless of whether the standard allows it. "But we wrote the compiler so it can't be…

You would need to pass *p or declare it as volatile I assume, otherwise by what means would foo change p?
Post reply on HN