Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

The Byte Order Fiasco

justine.lol

Post reply on HN
Print view View original

211–220 of 378 posts

Re: The Byte Order Fiasco

#211

Post by lanstin »

nly wrote: post #114
foldr wrote: post #101

Earlier quoted context omitted.

What are the advantages of this over a simple function with the following signature? uint32_t read_big_uint32(char *bytes); Having a big_uint32_t type seems wrong to me conceptually. You should either deal with sequences of bytes with a defined endianness or with native 32-bit integers of indeterminate endianness (assuming that your code is intended to be endian neutral). Having some kind of halfway house just confus…

The library provides those functions too, but I don't see how having an arithmetic type with well defined size, endiannness and alignment is a bad thing. If you're defining a struct to mirror a data structure from a device, protocol or file format then the language / type system should let you define the properties of the fields, not necessarily force you to introduce a parsing/decoding stage which could be more easi…

It is no longer arithmetic if there is an endianness. Some things are numbers and some things are sequences of bytes. Arithmetic only works on the former.

Re: The Byte Order Fiasco

#212

Post by [deleted] »

jart wrote: post #202
jstimpfle wrote: post #175

Earlier quoted context omitted.

I find you missed the point of the post and the issues described in it. In my estimation, libraries like boost are way too big and way too clever and they create more problems than they solve. Also, they don't make me happy. You're overfocusing on a "problem" that is almost completely irrelevant for most of programming. Big endian is rare to be found (almost no hardware to be found, but some file formats and networki…

Do you use UBSAN and ASAN? When you write unit tests do you feed numbers like 0x80000000 into your algorithm? When you allocate test memory have you considered doing it with mmap(4096) and putting the data at the end of the map? (Or better yet, double it and use mprotect). Those are some good examples of torture tests if you're in the mood to feel haunted.

[deleted]

Re: The Byte Order Fiasco

#213

Post by jstimpfle »

raphlinus wrote: post #194
jstimpfle wrote: post #175

Earlier quoted context omitted.

I find you missed the point of the post and the issues described in it. In my estimation, libraries like boost are way too big and way too clever and they create more problems than they solve. Also, they don't make me happy. You're overfocusing on a "problem" that is almost completely irrelevant for most of programming. Big endian is rare to be found (almost no hardware to be found, but some file formats and networki…

I agree with the bulk of this post. Re the anecdata at the end. Have you ever run your code through the sanitizers? I have. CVE-2016-2414 is one of my battle scars, and I consider myself a pretty good programmer who is aware of security implications.

Very little, quite frankly. I've used valgrind in the past, and found very few problems. I just ran -fsanitize=undefined for the first time on one of my current projects, which is an embedded network service of 8KLOC, and with a quick test covering probably 50% of the codepaths by doing network requests, no UB was detected (I made sure the sanitizer works in my build by introducing a (1Admittedly I'm not the type of person who spends his time fuzzing his own projects, so my statement was just to say that the kind of bugs that I hit by just testing my software casually are almost all of the very trivial kind - I've never experienced the feeling that the compiler "betrayed" me and introduced an obscure bug for something that looks like correct code.

I can't immediately see the problem in your CVE here [0], was that some kind of betrayal by compiler situation? Seems like strange things could happen if (end - start) underflows.

[0] https://android.googlesource.com/platform/frameworks/minikin...

Re: The Byte Order Fiasco

#214

Post by lanstin »

simias wrote: post #66
captainmuon wrote: post #41

It is a ridiculous feature of modern C that you have to write the super verbose "mask and shift" code, which then gets compiled to a simple `mov` and maybe a `bswap`. Wheras, the direct equivalent in C, an assignment with a (type changing) cast, is illegal. There is a huge mismatch between the assumptions of the C spec and actual machine code. One of the few reasons I ever even reached to C is the ability to slurp in…

> Wheras, the direct equivalent in C, an assignment with a (type changing) cast, is illegal. I don't understand what you mean by that. The direct equivalent of what? Endianess is not part of the type system in C so I'm not sure I follow. > I think there should really be a dialect of C(++) where the machine model is exactly the physical machine. Linus agrees with you here, and I disagree with both of you. Some UBs cou…

I am sympathetic, but portability was more important in the past and gets less important each year. I used to write code strictly keeping the difference between numeric types and sequences of bytes in mind, hoping to one day run on an Alpha or a Tandem or something, but it has been a long time since I have written code that runs on non-(Intel AMD or le ARM)

Re: The Byte Order Fiasco

#215

Post by dahart »

gpanders wrote: post #81
BenoitEssiambre wrote: post #53

Earlier quoted context omitted.

Also, this might be irrelevant at the cpu level, but within a byte, bits are usually displayed most significant bit first, so with little endian you end up with bit order: 7 6 5 4 3 2 1 0 15 14 13 12 11 10 9 8 instead of 15 to 0 This is because little endian is not how humans write numbers. For consistency with little endianness we would have to switch to writing "one hundred and twenty three" as 321

It might not be how humans write numbers but it is consistent with how we think about numbers in a base system. 123 = 3x10^0 + 2x10^1 + 1x10^2 So if you were to go and label each digit in 123 with the power of 10 it represents, you end up with little endian ordering (eg the 3 has index 0 and the 1 has index 2). This is why little endian has always made more sense to me, personally.

I always think about values in big endian, largest digit first. Scientific notation, for example, since often we only care about the first few digits.

I sometimes think about arithmetic in little endian, since addition always starts with the least significant digit, due to the right-to-left dependency of carrying.

Except lately I’ve been doing large additions big-endian style left-to-right, allowing intermediate “digits” with a value greater than 9, and doing the carry pass separately after the digit addition pass. It feels easier to me to think about addition this way, even though it’s a less efficient notation.

Long division and modulus are also big-endian operations. My favorite CS trick was learning how you can compute any arbitrarily sized number mod 7 in your head as fast as people are reading the digits of the number, from left to right. If you did it little-endian you’d have to remember the entire number, but in big endian you can forget each digit as soon as you use it.

Re: The Byte Order Fiasco

#216

Post by lanstin »

jart wrote: post #144
cygx wrote: post #107

Earlier quoted context omitted.

Not all user-perceived characters can be represented as a single Unicode codepoint. Hence, Unicode text encodings (almost[1]) always have to be treated as variable length, even UTF-32. [1] at runtime, you could dynamically assign 'virtual' codepoints to grapheme clusters and get a fixed-length encoding for strings that way

Even the individual unicode codepoints themselves are variable width if we consider that things like cjk and emoji take up >1 monospace cells.

Every time I see one of these threads, my gratitude to only do backend grows. Human behavior is too complex, let the webdevs handle UI, and human languages are too complex, not sure what speciality handles that. Give me out of order packets and parsing code that skips a character if the packet length lines up just so any day.

I am thankful that almost all the Unicode text I see is rendered properly now, farewell the little boxes. Good job lots of people.

Re: The Byte Order Fiasco

#217

Post by BenjiWiebe »

pjmlp wrote: post #133
jart wrote: post #94

Earlier quoted context omitted.

One does not simply introduce C++. It's the most insanely hardcore language there is. I wouldn't have stood any chance understanding it had it not been for my gentle introduction with C for several years.

Really? Apparently the first year students at my university didn't had any issue going from Standard Pascal to C++, in the mid-90's. Proper C++ was taught using our string, vector and collection classes, given that we were still a couple of years away from ISO C++ being fully defined. C style programming with low level tricks were only introduced later as advanced topics. Apparently thousands of students managed to g…

C++ in the mid 90s was a lot simpler than C++ now.

Re: The Byte Order Fiasco

#218

Post by jart »

pjmlp wrote: post #133
jart wrote: post #94

Earlier quoted context omitted.

One does not simply introduce C++. It's the most insanely hardcore language there is. I wouldn't have stood any chance understanding it had it not been for my gentle introduction with C for several years.

Really? Apparently the first year students at my university didn't had any issue going from Standard Pascal to C++, in the mid-90's. Proper C++ was taught using our string, vector and collection classes, given that we were still a couple of years away from ISO C++ being fully defined. C style programming with low level tricks were only introduced later as advanced topics. Apparently thousands of students managed to g…

Well there's a reason universities switched to Java when teaching algorithms and containers after the 90's. C++ is a weaker abstraction that encourages the kind of curiosity that's going to cause a student's brain to melt the moment they try to figure out how things work and encounter the sorts of demons the coursework hasn't prepared them to face. If I was going to teach it, I'd start with octal machine codes and work my way up. https://justine.lol/blinkenlights/realmode.html Sort of like if I were to teach TypeScript then I'd start with JavaScript. My approach to native development probably has more in common with web development than it does with modern c++ practices to be honest, and that's something I talk about in one of my famous hacks: https://github.com/jart/cosmopolitan/blob/4577f7fe11e5d8ef0a...

Re: The Byte Order Fiasco

#219

Post by lanstin »

gpanders wrote: post #81
BenoitEssiambre wrote: post #53

Earlier quoted context omitted.

Also, this might be irrelevant at the cpu level, but within a byte, bits are usually displayed most significant bit first, so with little endian you end up with bit order: 7 6 5 4 3 2 1 0 15 14 13 12 11 10 9 8 instead of 15 to 0 This is because little endian is not how humans write numbers. For consistency with little endianness we would have to switch to writing "one hundred and twenty three" as 321

It might not be how humans write numbers but it is consistent with how we think about numbers in a base system. 123 = 3x10^0 + 2x10^1 + 1x10^2 So if you were to go and label each digit in 123 with the power of 10 it represents, you end up with little endian ordering (eg the 3 has index 0 and the 1 has index 2). This is why little endian has always made more sense to me, personally.

Next you are going to want little endian polynomials, and that is just too far. Also, the advantage of big endian is it naturally extends to decimals/negative exponents where the later on things are less important. X squared plus x plus three minus one over x plus one over x squared etc.

Loss of big endian chips saddens me like the loss of underscores in var names in Go Lang. The homogeneity is worth something, thanks intel and camelCase, but the old order that passes away and is no more had the beauty of a new world.

Re: The Byte Order Fiasco

#220

Post by lanstin »

occamrazor wrote: post #198
BenoitEssiambre wrote: post #53

Earlier quoted context omitted.

Also, this might be irrelevant at the cpu level, but within a byte, bits are usually displayed most significant bit first, so with little endian you end up with bit order: 7 6 5 4 3 2 1 0 15 14 13 12 11 10 9 8 instead of 15 to 0 This is because little endian is not how humans write numbers. For consistency with little endianness we would have to switch to writing "one hundred and twenty three" as 321

In German _ein hundert drei und zwanzig_, literally _one hundred three and twenty_. The hardest part is are telephone numbers, that are usually given in blocks of two digits.

Well that would be hard for me to learn. I always find the small numbers between like 10 and 100 or 1000 the hardest for me to remember in languages I am trying to learn a bit of.
Post reply on HN