That huge macro appears to be wrong, as there are little endian PowerPC systems, where __ppc__ and _powerpc__ macros are also defined, making the outcome of the detection invalid.
The Byte Order Fiasco
311–320 of 378 posts
Re: The Byte Order Fiasco
#312Rust gets this right. These primitives are available for all the numeric types. u32::from_le_byte(bytes) // u32 from 4 bytes, little endian u32::from_be_byte(bytes) // u32 from 4 bytes, big endian u32::to_le_bytes(num) // u32 to 4 bytes, little endian u32::to_be_bytes(num) // u32 to 4 bytes, big endian This was very useful to me recently as I had to write the marshaling and un-marshaling for a game networking format…
It's about accessing memory plus an extra conversion step vs. accessing memory in the right order in one step. As an extra, platform-dependent implementations of the accessors could be done, like using the LWL+LWR instruction pair on MIPS.
For reference, check how Linux does it. https://elixir.bootlin.com/linux/latest/source/include/linux...
Re: The Byte Order Fiasco
#313Earlier quoted context omitted.
> if there were an inherent advantage to breaking your egg from one side or the other, would there be a war at all? Fascism vs. not-fascism, Stalinist Communism vs. Western Capitalism, Islamism vs. liberal democracy... I’m not sure “the existence of war around a divide in ideas proves that neither sides ideas are correct” is a particularly comfortable maxim to consider the ramifications of.
Two similar societies warring over a trivial idea probably means neither is right. Swift's Big Endians and Little Endians are a satire of the Catholic-Anglican schism in England.
> Two similar societies warring over a trivial idea probably means neither is right.
Well, sure, that it’s a trivial idea pretty much inherently means either that neither is right or (and this is very much not an exclusive or) being right doesn’t matter.
The problem with real cases is that people inside the conflict don’t believe the idea is trivial (conversely, to people outside rhe conflict—or caught in the middle—even the conflicts we think of as about foundational ideas seem like trivial or irrelevant differences.)
Re: The Byte Order Fiasco
#314Earlier quoted context omitted.
In your example, why should it optimise out the second case? Maybe foo() changed p so it's no longer null. Compiler writers do not format disks just to punish your UBs. IMHO if the compiler exploiting UB is leading to counterintuitive behaviour that's making it harder to use the language, the compiler is the one that needs fixing, regardless of whether the standard allows it. "But we wrote the compiler so it can't be…
You would need to pass *p or declare it as volatile I assume, otherwise by what means would foo change p?
The address of p could have been taken somewhere earlier and stored in a global that foo accesses, or a similar path to that; and of course, p could itself be a global. Indeed, if the purpose of foo is to make p non-null and point to valid memory, then by optimising away that code you have broken a valid program.
If the compiler doesn't know if foo may modify p, then it can't remove the call. Even if it can prove that foo does not modify p, it still can't remove the call: foo may still have some other side-effects that matter (like not returning --- either longjmp()'ing elsewhere or perhaps printing an error message about p being null and exiting?), so it won't even get to the null dereference.
As a programmer, if I write code like that, I either intend for foo to be doing something to p to make it non-null, or if it doesn't for whatever reason, then it will actually dereference the null and whatever happens when that's attempted on the particular platform, happens. One of the fundamental principles of C is "trust the programmer". In other words, by trying to be "helpful" and second-guessing the intent of the code while making assumptions about UB, the compiler has completely broken the expectations of the programmer. This is why assumptions based on UB are stupid.
The standard allows this, but the whole intent of UB is not so compiler-writers can play language-lawyer and abuse programmers; things it leaves undefined are usually because existing and possible future implementations vary so widely that they didn't even try to consider or enumerate the possibilities (unlike with "implementation-defined").
Re: The Byte Order Fiasco
#315Earlier quoted context omitted.
We really should have moved to 32 bit bytes when moving to 64 bit words. Would have simplified Unicode considerably.
Not really. Unicode is a variable width abstract encoding; a single character can be made up of multiple code points. For Unicode, 32-bit bytes would be an incredibly wasteful in memory encoding.
> Unicode is a variable width abstract encoding;
To be a bit more explicit: Unicode is a character encoding, to 20-and-a-half-bit 'bytes', that is variable-width in those 'bytes', even before considering how the 'bytes' are encoded to actual bytes. Eg "ψ̊" (greek small psi with ring above) is U+3C8 U+30A (two 'bytes').
Re: The Byte Order Fiasco
#316Rust gets this right. These primitives are available for all the numeric types. u32::from_le_byte(bytes) // u32 from 4 bytes, little endian u32::from_be_byte(bytes) // u32 from 4 bytes, big endian u32::to_le_bytes(num) // u32 to 4 bytes, little endian u32::to_be_bytes(num) // u32 to 4 bytes, big endian This was very useful to me recently as I had to write the marshaling and un-marshaling for a game networking format…
Unless you are planning on running your game on a mainframe, just don’t bother with endianness for the networking. Big endian is dead for game developers. Copy entire arrays of structs onto the wire without fear! (Just #pragma pack them first)
I disagree... but what I would do is use little endian, so the legacy machines need to byte swap.
Re: The Byte Order Fiasco
#317Earlier quoted context omitted.
What happens is that all machines that matter are little endian but network works always in Big Endian.
Isn’t big endian a bit more natural considered on a bit level? The bits start from highest to lowest on a serial connection.
> The bits start from highest to lowest on a serial connection.
This is only true on a big endian serial connection (that is, one that, tautologically, sends the most-significant bit first). Offhand, I think most serial protocols are big endian, but by that logic, most CPUs are little endian, so that doesn't really help.
The thing that's actually useful about big endian is not that it's natural (as kangalioo points out, that's little endian) or that it's how humans write numbers (by that logic crap like BCD or decimal floats is a good idea), but that big endian preserves lexicographic order of fixed-width integers.
Re: The Byte Order Fiasco
#318Earlier quoted context omitted.
You would need to pass *p or declare it as volatile I assume, otherwise by what means would foo change p?
The address of p could have been taken somewhere earlier and stored in a global that foo accesses, or a similar path to that; and of course, p could itself be a global. Indeed, if the purpose of foo is to make p non-null and point to valid memory, then by optimising away that code you have broken a valid program. If the compiler doesn't know if foo may modify p, then it can't remove the call. Even if it can prove tha…
But in fact compilers do regularly prove such things as, "this function call did not touch that local variable". Escape analysis is a term related to this.
I'm more of two minds about that other step, where the compiler goes like, "here in the printf call the p will be dereferenced, so it surely is non-null, so we silently optimize that other thing out where we consider the possibility of it being null".
Also @joshuamorton, couldn't the compiler at least print a warning that it removed code based on an assumption that was inferred by the compiler? I really don't know a lot about those abstract logic solver approaches, but it feels like it should be easy to do.
Re: The Byte Order Fiasco
#319Earlier quoted context omitted.
There was a blog post and a FOSDEM presentation by (misguided) Gentoo developers a few years ago, and it was retracted, because sanitizers add their own exploitable vulnerabilities due to the way they work. https://blog.hboeck.de/archives/879-Safer-use-of-C-code-runn... https://www.openwall.com/lists/oss-security/2016/02/17/9
Sorry for my ignorance, but surely some UB being used for optimization by the compiler is compile time only. This is the part that should default on. Runtime detection is a different thing entirely, but compile time is a no brainer.
UBSAN detects undefined behavior at run-time. Compile-time detection of undefined behavior is present in the form of compiler warnings, but catches far from all cases of undefined behavior. The compiler does not actively exploit undefined behavior in the sense that it does not contain code like this:
if (undefined_behavior) break_program()
bool function(int x) {
return x + 1 > x;
}
The compiler could tell you for every instance of signed integer arithmetic that it is making assumptions about your program, and that the signed integer arithmetic could potentially overflow, but that doesn't seem particularly helpful.
Re: The Byte Order Fiasco
#320Earlier quoted context omitted.
Well, obviously it would have delayed the transition. However you can only go so far with 4Go-limited memory. And do you have examples of still widely used 8-bit sized data formats ?
I assume you wrote this comment in UTF-8 over HTTP (ASCII-based) and TLS (lots of uint8 fields).
To clarify : I'm more concerned about "final" data formats, less about "transport" ones, which need much longer legacy support.