Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

The Byte Order Fiasco

justine.lol

Post reply on HN
Print view View original

331–340 of 378 posts

Re: The Byte Order Fiasco

#331

Post by spacechild1 »

BlueTemplar wrote: post #95
wongarsu wrote: post #56

Earlier quoted context omitted.

People were holding off on transitioning because pointers use twice as much space in x64. If bytes had quadrupled in space with x64 we would still be using 32 bit software everywhere

Well, obviously it would have delayed the transition. However you can only go so far with 4Go-limited memory. And do you have examples of still widely used 8-bit sized data formats ?

MIDI (8 bit), 16 bit PCM, 24 PCM and basically any compressed data format (which is always byte based, because the idea is to save memory). You obviously don't care about memory, but many people do!

Re: The Byte Order Fiasco

#332

Post by spacechild1 »

BlueTemplar wrote: post #324
spacechild1 wrote: post #119

Earlier quoted context omitted.

You know, bytes are not only about text, they are also used to represent binary data... Not to mention that bytes have nothing to do with unicode. Unicode codepoints can be encoded in many different ways: UTF8, UTF16, UTF32, etc.

https://news.ycombinator.com/item?id=27086928 These various ways to encode Unicode have quite a lot to do with bytes being 8-bit sized !

Generally, I think you are conflating/confusing the concept of “byte“ (= smallest unit of memory) with the concept of “character“ resp. “code unit“ (= smallest unit of text encoding). The size of the former depends on the CPU architecture and on modern systems it's always 8 bits. The size of the latter depends on the specific text encoding.

Re: The Byte Order Fiasco

#333

Post by spacechild1 »

ChrisSD wrote: post #48
BlueTemplar wrote: post #36

Earlier quoted context omitted.

We really should have moved to 32 bit bytes when moving to 64 bit words. Would have simplified Unicode considerably.

Not really. Unicode is a variable width abstract encoding; a single character can be made up of multiple code points. For Unicode, 32-bit bytes would be an incredibly wasteful in memory encoding.

Unicode is not a text encoding. UTF8, UTF16, UTF32, etc. are text encodings.

> a single character can be made up of multiple code points.

It's really the other way round...

Re: The Byte Order Fiasco

#334

Post by matheusmoreira »

pajko wrote: post #312
Animats wrote: post #207

Rust gets this right. These primitives are available for all the numeric types. u32::from_le_byte(bytes) // u32 from 4 bytes, little endian u32::from_be_byte(bytes) // u32 from 4 bytes, big endian u32::to_le_bytes(num) // u32 to 4 bytes, little endian u32::to_be_bytes(num) // u32 to 4 bytes, big endian This was very useful to me recently as I had to write the marshaling and un-marshaling for a game networking format…

It's about accessing memory plus an extra conversion step vs. accessing memory in the right order in one step. As an extra, platform-dependent implementations of the accessors could be done, like using the LWL+LWR instruction pair on MIPS. For reference, check how Linux does it. https://elixir.bootlin.com/linux/latest/source/include/linux...

Doesn't this run into the same issue described by the author?

> Because unsigned char in C expressions gets type promoted to the signed type int.

> So if we say 0x80Does the u8 type protect against this?

Re: The Byte Order Fiasco

#335

Post by gnufx »

londons_explore wrote: post #15

Remember how we used to have machines with a 7 bit byte? And everything was written to handle either 6, 7, or 8 bit bytes? And now we've settled on all machines being 8 bit bytes, and programmers no longer have to worry about such details? Is it time to do the same for big endian machines? Is it time to accept that all machines that matter are little endian, and the extra effort keeping everything portable to big end…

Machines that matter to who? Maintainers of packages for OSes that support s390x (RHEL/Fedora, SUSE, Debian), Arduino AVR users, AIX users, embedded systems people with, say, Coldfire or some DSP? I don't know to what extent portable code is relevant to embedded systems, but I guess they care about cryptography libraries, for instance. (I know it's bi- in principle, but SPARC is still even in the Top500.)

Re: The Byte Order Fiasco

#336

Post by gnufx »

CountHackulus wrote: post #80
bregma wrote: post #7

Earlier quoted context omitted.

And which is the correct byte ordering, pray tell?

Middle-endian is the only correct answer. It's a tradeoff between both little-endian and big-endian. The PDP-11 got it right.

I don't suppose it's being modified, but I wonder how much -11 code is still running, even on real hardware.

Re: The Byte Order Fiasco

#337

Post by spacechild1 »

spacechild1 wrote: post #333
ChrisSD wrote: post #48

Earlier quoted context omitted.

Not really. Unicode is a variable width abstract encoding; a single character can be made up of multiple code points. For Unicode, 32-bit bytes would be an incredibly wasteful in memory encoding.

Unicode is not a text encoding. UTF8, UTF16, UTF32, etc. are text encodings. > a single character can be made up of multiple code points. It's really the other way round...

Or maybe you meant to say: A single abstract character (or code point) can be made up of multiple code units.

Unfortunately, the term “character“ alone is ambiguous because depending on the context it can refer to either code points or code units.

Re: The Byte Order Fiasco

#338

Post by dooglius »

wruza wrote: post #296
userbinator wrote: post #261

Earlier quoted context omitted.

So let's fix our code. No; I say we force the compiler writers to fix their idiotic assumptions instead of bending over backwards to please what's essentially a tiny minority. There's a lot more programmers who are not compiler writers. The standard is really a minimum bar to meet, and what's not defined by it is left to the discretion of the implementers, who should be doing their best to follow the "spirit of C", w…

force the compiler writers to fix their idiotic assumptions instead of bending over backwards to please what's essentially a tiny minority As far as I understand it, they do neither. Transforming an AST to any level of target code is not done by handcrafted recipes, but instead is feeded into efficient abstract solvers which have these assumptions as an operational detail. E.g.: p = &x; if (p != &x) foo(); // optimiz…

> No assumption here is idiotic

Yes, the assumption that p is non-null is idiotic. Also, the implicit assumption that foo will always return.

> no single human was involved

Humans implemented the compilers that use the spec adversarially and humans lobby the standards committee to not fix the bugs

> Of course you can write a boring compiler that emits opcodes at face expr value, without most UBs being a problem. Plenty of these, why not just take one

The majority of optimizations are harmless and useful, only a handful are idiotic and harmful. I want a compiler that has the good optimizations and not the bad ones.

Re: The Byte Order Fiasco

#339

Post by ChrisSD »

spacechild1 wrote: post #337
spacechild1 wrote: post #333

Earlier quoted context omitted.

Unicode is not a text encoding. UTF8, UTF16, UTF32, etc. are text encodings. > a single character can be made up of multiple code points. It's really the other way round...

Or maybe you meant to say: A single abstract character (or code point) can be made up of multiple code units . Unfortunately, the term “character“ alone is ambiguous because depending on the context it can refer to either code points or code units.

To be technical, by "character" I mean "user-perceived character" or (in Unicode speak) "extended grapheme cluster". This is the thing a user will think of as one character when looking at it on their screen.

A code point is the atomic unit of the abstract Unicode encoding. By "abstract" I mean it is not an actual text encoding you can write to a file.

A code unit is the atomic unit of an actual text encoding, such as UTF-8, UTF-16LE or UTF-32LE (and their BE equivalents).

---

So to put it together a "user-perceived character" is made up of one or more "code points". When implemented in an application, each "code point" is encoded using one or more "code units".

Re: The Byte Order Fiasco

#340

Post by wruza »

jstimpfle wrote: post #318
userbinator wrote: post #314

Earlier quoted context omitted.

The address of p could have been taken somewhere earlier and stored in a global that foo accesses, or a similar path to that; and of course, p could itself be a global. Indeed, if the purpose of foo is to make p non-null and point to valid memory, then by optimising away that code you have broken a valid program. If the compiler doesn't know if foo may modify p, then it can't remove the call. Even if it can prove tha…

But in fact compilers do regularly prove such things as, "this function call did not touch that local variable". Escape analysis is a term related to this. I'm more of two minds about that other step, where the compiler goes like, "here in the printf call the p will be dereferenced, so it surely is non-null, so we silently optimize that other thing out where we consider the possibility of it being null". Also @joshua…

warning that it removed code based on an assumption that was inferred by the compiler

That would dump a ton of warnings from various macro/meta routines, which real-world C is usually peppered with. Not that it’s particularly hard to do (at the very least compilers know which lines are missing from debug info alone).

Post reply on HN