Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

The Byte Order Fiasco

justine.lol

Post reply on HN
Print view View original

291–300 of 378 posts

Re: The Byte Order Fiasco

#291

Post by jsrcout »

Of course, the canonical work on this subject is Danny Cohen's On Holy Wars And A Plea For Peace [0]. It's an informative and highly readable article. My favorite quote, from the conclusion, is:

    The  "Be reasonable, do it my way" approach does not work.  Neither does the Esperanto approach of "let's all switch to yet a new language".
His bottom line conclusion being

    It is more important to  agree  upon an order than which order is agreed upon.
[0] https://www.rfc-editor.org/ien/ien137.txt

Re: The Byte Order Fiasco

#292

Post by jstimpfle »

raphlinus wrote: post #236
jstimpfle wrote: post #213

Earlier quoted context omitted.

Very little, quite frankly. I've used valgrind in the past, and found very few problems. I just ran -fsanitize=undefined for the first time on one of my current projects, which is an embedded network service of 8KLOC, and with a quick test covering probably 50% of the codepaths by doing network requests, no UB was detected (I made sure the sanitizer works in my build by introducing a (1 Admittedly I'm not the type of…

This one wasn't specifically "betrayal by compiler," but it was a confusion between signed and unsigned quantities for a size field, which is very similar to the UB exhibited in OP. Also, the fact that you can't see the problem is actually evidence of how insidious these problems are :) The rules for this are arcane, and, while the solution suggested in OP is correct, it skates close to the edge, in that there are ma…

> the fact that you can't see the problem is actually evidence of how insidious these problems are

I've looked for a while now, but still can't see it, would you be willing to share?

> (p[1] With p[1] being uint8_t? Because then I cannot imagine why, and also fail to see a reason to apply the 0xff00 mask here.

If this is for int8_t instead, the problem you are alluding to is sign extension? If p[1] gets promoted to an int in the negative range, (then its representation has the high order bit set), and shifting that to the left is UB.

Re: The Byte Order Fiasco

#293

Post by lilyball »

infradig wrote: post #249
Animats wrote: post #207

Rust gets this right. These primitives are available for all the numeric types. u32::from_le_byte(bytes) // u32 from 4 bytes, little endian u32::from_be_byte(bytes) // u32 from 4 bytes, big endian u32::to_le_bytes(num) // u32 to 4 bytes, little endian u32::to_be_bytes(num) // u32 to 4 bytes, big endian This was very useful to me recently as I had to write the marshaling and un-marshaling for a game networking format…

There are equivalent functions in C too. The point of the article is about not using them. So how would you implement the above functions in Rust would be more pertinent.

Given that Rust isn’t C, the answer is Rust has a compiler intrinsic for bswap and it calls that as appropriate. LLVM will then turn that into the correct instruction(s) for the target platform.

Re: The Byte Order Fiasco

#294

Post by [deleted] »

lilyball wrote: post #293
infradig wrote: post #249

Earlier quoted context omitted.

There are equivalent functions in C too. The point of the article is about not using them. So how would you implement the above functions in Rust would be more pertinent.

Given that Rust isn’t C, the answer is Rust has a compiler intrinsic for bswap and it calls that as appropriate. LLVM will then turn that into the correct instruction(s) for the target platform.

[deleted]

Re: The Byte Order Fiasco

#295

Post by jart »

userbinator wrote: post #261
jart wrote: post #243

Earlier quoted context omitted.

I agree but language of the standard very unambiguously lets them do it. Quoth X3.159-1988 * Undefined behavior --- behavior, upon use of a nonportable or erroneous program construct, of erroneous data, or of indeterminately-valued objects, for which the Standard imposes no requirements. Permissible undefined behavior ranges from ignoring the situation completely with unpredictable results, to behaving during transla…

So let's fix our code. No; I say we force the compiler writers to fix their idiotic assumptions instead of bending over backwards to please what's essentially a tiny minority. There's a lot more programmers who are not compiler writers. The standard is really a minimum bar to meet, and what's not defined by it is left to the discretion of the implementers, who should be doing their best to follow the "spirit of C", w…

Get the teamsters and workers world party to occupy clang. You should fork C to restore the spirit of C and call it Spiritual C since we need a new successor to Holy C.

Re: The Byte Order Fiasco

#296

Post by wruza »

userbinator wrote: post #261
jart wrote: post #243

Earlier quoted context omitted.

I agree but language of the standard very unambiguously lets them do it. Quoth X3.159-1988 * Undefined behavior --- behavior, upon use of a nonportable or erroneous program construct, of erroneous data, or of indeterminately-valued objects, for which the Standard imposes no requirements. Permissible undefined behavior ranges from ignoring the situation completely with unpredictable results, to behaving during transla…

So let's fix our code. No; I say we force the compiler writers to fix their idiotic assumptions instead of bending over backwards to please what's essentially a tiny minority. There's a lot more programmers who are not compiler writers. The standard is really a minimum bar to meet, and what's not defined by it is left to the discretion of the implementers, who should be doing their best to follow the "spirit of C", w…

force the compiler writers to fix their idiotic assumptions instead of bending over backwards to please what's essentially a tiny minority

As far as I understand it, they do neither. Transforming an AST to any level of target code is not done by handcrafted recipes, but instead is feeded into efficient abstract solvers which have these assumptions as an operational detail. E.g.:

  p = &x;
  if (p != &x) foo(); // optimized out
is not much different from

  if (p == NULL) foo(); // optimized out
  printf("%c", *p);
No assumption here is idiotic, cause no single human was involved, it’s just a class of constraints, which alone to separate properly you’ll have to scratch your head extensively (imagine telling a logic system that p is both 0 and not-0 when 0-test is “explicit” and asking it to normally operate). Compiler writers do not format disks just to punish your UBs. Of course you can write a boring compiler that emits opcodes at face expr value, without most UBs being a problem. Plenty of these, why not just take one?

Re: The Byte Order Fiasco

#297

Post by jart »

pornel wrote: post #265
jart wrote: post #239

Earlier quoted context omitted.

Sanitizers have the ability to bring Rust-like safety assurances to all the C/C++ code that exists. The fact that existing ASAN runtimes weren't designed for setuid binaries shouldn't dissuade us from pursuing those benefits. We just need a production-worthy runtime that does less things. For example, here's the ASAN runtime that's used for the redbean web server: https://github.com/jart/cosmopolitan/blob/master/libc…

Run-time detection and heuristics on a language that is hard to analyze (e.g. due to weak aliasing, useless const, ad-hoc ownership and thread-safety rules) aren't in the same ballpark as compile-time safety guaranteed by construction, and an entire modern ecosystem centered around safety. Rust can use LLVM sanitizers in addition to its own checks, so that's not even a trade-off.

Oh I believe you but as you point out we need ASAN to make Rust codebases safer too. One of the things that's helped Rust be successful is that we're able to quickly write bindings for legacy C/C++/FORTRAN code using the unsafe keyword. The last Rust codebase I worked on had about 70k unsafe lines. One day Rust will be complete and we will rewrite all the legacy code but until then we depend on the low level C tooling to provide assurances like byte-granular invalid address access trapping.

Re: The Byte Order Fiasco

#298

Post by userbinator »

remexre wrote: post #286
userbinator wrote: post #261

Earlier quoted context omitted.

So let's fix our code. No; I say we force the compiler writers to fix their idiotic assumptions instead of bending over backwards to please what's essentially a tiny minority. There's a lot more programmers who are not compiler writers. The standard is really a minimum bar to meet, and what's not defined by it is left to the discretion of the implementers, who should be doing their best to follow the "spirit of C", w…

For essentially every form of UB that compilers actually take advantage of, there's a real program optimization benefit. Are there any particular UB cases where you think the benefit isn't worth it, or it should be implementation-specific behavior instead of undefined behavior?

MSVC and ICC have traditionally been far less keen on exploiting UB, yet are extremely competitive on performance (ICC in particular). That alone is enough evidence to convince me that UB is not the performance-panacea that the gcc/clang crowd think it is, and from my experience with writing Asm, good instruction selection and scheduling is far more important than trying to pull tricks with UB.

Re: The Byte Order Fiasco

#299

Post by xeeeeeeeeeeenu »

secondcoming wrote: post #69

Isn't the 'modern' solution to memcpy into a temp and swap the bytes in that? C++ has added/will add std::launder and std::bless to deal with this issue

>C++ has added/will add std::launder and std::bless to deal with this issue

You're thinking of std::bit_cast. std::launder solves a different, much more obscure problem: https://miyuki.github.io/2016/10/21/std-launder.html

Re: The Byte Order Fiasco

#300

Post by userbinator »

wruza wrote: post #296
userbinator wrote: post #261

Earlier quoted context omitted.

So let's fix our code. No; I say we force the compiler writers to fix their idiotic assumptions instead of bending over backwards to please what's essentially a tiny minority. There's a lot more programmers who are not compiler writers. The standard is really a minimum bar to meet, and what's not defined by it is left to the discretion of the implementers, who should be doing their best to follow the "spirit of C", w…

force the compiler writers to fix their idiotic assumptions instead of bending over backwards to please what's essentially a tiny minority As far as I understand it, they do neither. Transforming an AST to any level of target code is not done by handcrafted recipes, but instead is feeded into efficient abstract solvers which have these assumptions as an operational detail. E.g.: p = &x; if (p != &x) foo(); // optimiz…

In your example, why should it optimise out the second case? Maybe foo() changed p so it's no longer null.

Compiler writers do not format disks just to punish your UBs.

IMHO if the compiler exploiting UB is leading to counterintuitive behaviour that's making it harder to use the language, the compiler is the one that needs fixing, regardless of whether the standard allows it. "But we wrote the compiler so it can't be fixed" just feels like a "but the AI did it, not me" excuse.

Post reply on HN