Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

The Byte Order Fiasco

justine.lol

Post reply on HN
Print view View original

151–160 of 378 posts

Re: The Byte Order Fiasco

#151

Post by phkahler »

kingsuper20 wrote: post #149

I've never been very satisfied with these approaches for C where you hope the compiler does the right thing. It makes sense to provide some C implementation for portability's sake but any sizeable reordering cries out for a handtuned, processor specific, approach (and the non-sizeable probably doesn't require high speed). I would expect any SIMD instruction set to include a shuffle.

It can also be a good idea to swap recursively. First swap the upper and lower half, then swap the upper and lower quarters (bytes for a 32bit) which can be done with only 2 masks. Then if its 64bit value swap alternate bytes, again with only 2 masks. This can be extended all the way to full bit reverse in 3 more lines each with 2 masks and shifts.

Re: The Byte Order Fiasco

#152

Post by amluto »

Why mask and then shift instead of casting to the correct type and then shifting, like this:

    (uint32_t)x[0]
Of course, this requires that x[0] be unsigned.

Re: The Byte Order Fiasco

#153

Post by marcosdumay »

enqk wrote: post #110
mrlonglong wrote: post #34

In an ideal world which endian format would one go for?

https://fgiesen.wordpress.com/2014/10/25/little-endian-vs-bi...

Why would one choose the memory representation of the number based on the advantages of the internal ALU wiring?

Of all those reasons, the only one I can make sense of is the "I can’t transparently widen fields after the fact!", and that one is way too niche to explain anything.

Re: The Byte Order Fiasco

#154

Post by kortex »

BlueTemplar wrote: post #90
ChrisSD wrote: post #48

Earlier quoted context omitted.

Not really. Unicode is a variable width abstract encoding; a single character can be made up of multiple code points. For Unicode, 32-bit bytes would be an incredibly wasteful in memory encoding.

One byte = one "character" makes for much easier programming. Text generally uses a small fraction of memory and storage these days.

> One byte = one "character" makes for much easier programming.

Only if you are naively operating in the Anglosphere / world where the most complex thing you have to handle is larger character sets. In reality, there's ligatures, diacritics, combining characters, RTL, nbsp, locales, and emoji (with skin tones!). Not to mention legacy encoding.

And no, it does not use a "small fraction of memory and storage" in a huge range of applications, to the point where some regions have transcoding proxies still.

Re: The Byte Order Fiasco

#155

Post by toast0 »

jart wrote: post #115
BlueTemplar wrote: post #36

Earlier quoted context omitted.

We really should have moved to 32 bit bytes when moving to 64 bit words. Would have simplified Unicode considerably.

Use Erlang. It has 32-bit char.

Not really. Strings are a list of integers [1], integers are signed and fill a system word, but there's also 4 bits of type information. So you can have a 28-bit signed integer char on a 32-bit system or a signed 60-bit integer.

However, since Unicode is limited to 21-bits by utf-16 encoding, a unicode code point will fit in a small integer.

[1] unless you use binaries, which is often a better choice.

Re: The Byte Order Fiasco

#156

Post by nly »

raphlinus wrote: post #146
nly wrote: post #120

Earlier quoted context omitted.

Who decides what methods to add to the bytes type/abstraction? If I have a 3 byte big endian integer can I access it easily in rust without resorting to shifts? In C++ I could probably create a fairly convincing big_uint24_t type and use it in a packed struct and there would be no inconsistencies with how it's used with respect to the more common varieties

In Rust, [u8; N] and &[u8] are both primitive types, and not abstractions. It's possible to create an abstraction around either (the former even more so now with const generics), but that's not necessary. It's also possible to use "extension traits" to add methods, even to existing and built-in types[1]. I'm not sure about a 3 byte big endian integer. I mean, that's going to compile down to some combination of shifti…

I agree then that in Rust you could make something consistent.

I think there's no need for explicit shifts. You need to memcpy anyway to deal with alignment issues, so you may as well just copy in to the last 3 bytes of a zero-initialized, big endian, 32bit uint.

https://gcc.godbolt.org/z/jEnsW8WfE

Re: The Byte Order Fiasco

#157

Post by Brian_K_White »

nly wrote: post #55
vladharbuz wrote: post #44

Earlier quoted context omitted.

Correct me if I'm wrong, but your example is just using a library to do the same task, rather than illustrating any difference between C and C++. If you want to pull boost in to do this, that's great, but that hardly seems like a fair comparison to the OP, since instead of implementing code to solve this problem yourself you're just importing someone else's code.

No, the fact that this can be done in a library and looks like a native language feature demonstrates the power of C++ as a language. This example is demonstrating: - First class treatment of user (or library) defined types - Operator overloading - The fact that it produces fast machine code. Try changing big_uint32_t to regular uint32_t to see how this changes. When you use the later ubsan will introduce a trap for…

It demonstrates that c++ is even less safe.

Re: The Byte Order Fiasco

#158

Post by CydeWeys »

kangalioo wrote: post #140
ben509 wrote: post #124

Earlier quoted context omitted.

Big-endian is natural when you're comparing numbers, which is probably why people represent numbers in a big-endian fashion. Little-endian is natural with casts because the address doesn't change, and it's the order in which addition takes place.

I feel like big endian is more _intuitive_ because that's what our number notation has evolved to be. But more _natural_ is little endian because, well, it's just more straightforward to have the digits' magnitude be in ascending order (2^0, 2^1, 2^2, 2^3...) instead of putting it in reverse. Plus you encounter less roadblocks in practice with little endian (e.g. address changes with casts) which is often a sign of g…

I'm curious how you're defining "natural", and if you think ISO-8601 is the reverse of "natural" too.

All human number systems I've ever seen write numbers out as big Endian (yes, even Roman numerals), so I'm really struggling to see how that wouldn't be considered natural.

Re: The Byte Order Fiasco

#159

Post by ta_ca »

ta_ca wrote: post #24
amelius wrote: post #3

Byte order is one of the great unnecessary historical fuck ups in computing. A similar one is that signedness of char is machine dependent. It's typically signed on Intel and unsigned on ARM. Sigh!

the greatest of all is lisp not being the most mainstream language, and we can only blame the lisp companies for this fiasco. in an ideal world we all would be using a lisp with parametric polymorphism. from highest level abstractions to machine level, all in one language.

i hope these downvotes are due to my failure at english or the comment being off-topic (or both). if not, can i just replace lisp with rust and be friends again?

Re: The Byte Order Fiasco

#160

Post by microtherion »

jart wrote: post #46
nly wrote: post #23

This is why, in 2021, the mantra that C is a good language for these low level byte twiddling tasks needs to die. Dealing with alignment and endianness properly requires a language that allows you to build abstractions. The following is perfectly well defined in C++, despite looking like almost the same as the original unsafe C: #include #include using namespace boost::endian; unsigned char b[5] = {0x80,0x01,0x02,0x0…

C is perfect for these problems. I like teaching the endian serialization problem because it broaches so many of the topics that are key to understanding C/C++ in general. Even if we choose to spend the majority of our time plumbing together functions written by better men, it's nice to understand how the language is defined so we could write those functions, even if we don't need to.

Yes, there is some value in using C for teaching these concepts. But the problem I see is that, once taught, many people will then continue to use C and their hand written byte swapping functions, instead of moving on to languages with better abstraction facilities and/or availing themselves of the (as you point out) many available library implementations of this functionality.
Post reply on HN