Live data from Hacker News

Zig's new bitCast semantics and LLVM back end improvements

ziglang.org

101–110 of 159 posts

Re: Zig's new bitCast semantics and LLVM back end improvements

#101
post #81

Earlier quoted context omitted.

Does that mean there are no file formats thatbuse big endian? And network byte order isn't a thing?

> there are no file formats thatbuse big endian if someone chooses to do that they own the problems. > network byte order isn't a thing if the network serializes/deserializes for you (kernel primitives) then you don't care what it does. if it doesn't and for some reason you choose to use big endian, again, you own the problem.

Network byte order has nothing to do with the kernel and you have to care about it

It’s a standard because neither side of the connection knows the endianness of the other side so there must be a standard. That standard is big endian regardless of your architecture or kernel or anything else

So any serialization intended go over the network should be big endian

Re: Zig's new bitCast semantics and LLVM back end improvements

#102
post #81

Earlier quoted context omitted.

> there are no file formats thatbuse big endian if someone chooses to do that they own the problems. > network byte order isn't a thing if the network serializes/deserializes for you (kernel primitives) then you don't care what it does. if it doesn't and for some reason you choose to use big endian, again, you own the problem.

Network byte order has nothing to do with the kernel and you have to care about it It’s a standard because neither side of the connection knows the endianness of the other side so there must be a standard. That standard is big endian regardless of your architecture or kernel or anything else So any serialization intended go over the network should be big endian

right, so a zig app will just do little endian. in the very unlikely event you have it running on a big endian machine you have to do extra work.

Re: Zig's new bitCast semantics and LLVM back end improvements

#103
post #100
post #81

Earlier quoted context omitted.

> there are no file formats thatbuse big endian if someone chooses to do that they own the problems. > network byte order isn't a thing if the network serializes/deserializes for you (kernel primitives) then you don't care what it does. if it doesn't and for some reason you choose to use big endian, again, you own the problem.

If someone chooses to load a TIFF or a PSD or an AIFF or…

then you either use an existing C library (the most likely approach) or if you are determined to re-implement it you have to be careful parsing their bytes.

Re: Zig's new bitCast semantics and LLVM back end improvements

#104
post #35

Earlier quoted context omitted.

Andrew doesn't strike me as someone who does any marketing at all. He just wants to make the language he wants to use, and does it well. Sometimes its just right time, right place. But also, Zig has received attention via projects like Ghostty, TigerBeetle, and Bun (prior to rewrite of course)

They have definitely done a lot of marketing through social media and forums like HN. There have been large numbers of posts here by Zig's developers for years, and a few releases of LLVM even mentioned Zig prominently in their release notes.

Maybe people just like the language

Re: Zig's new bitCast semantics and LLVM back end improvements

#105
FTA: “Under the new semantics, because we only care about logical bit representation (which is endian-agnostic), the operation behaves identically on every target: the first array element becomes the 8 least significant bits”

I wouldn’t call that endian-agnostic. It’s explicitly picking little-endian.

It also makes things look weird for beginners. I know how it works, but in the

  test "bitcast [2]u3 to @Vector(3, u2)"
example, turning two 3-bit values [abc def] into three 2-bit values [bc fa de] is way less intuitive than turning it into [ab cd ef].

Re: Zig's new bitCast semantics and LLVM back end improvements

#106

Earlier quoted context omitted.

Does that mean there are no file formats thatbuse big endian? And network byte order isn't a thing?

Generally those edge cases are always the same endianness. You don't need big and little endianness versions of the structures. What's important is that everyone agrees on the same thing.

There are some cursed data formats where something is little endian in some places, big endian in other places

Generally speaking though the types you handle in business logic (what your application actually do) shouldn't have any endianness

Re: Zig's new bitCast semantics and LLVM back end improvements

#107
post #96

Earlier quoted context omitted.

> The 24 Bits (3 Bytes) [3]u8 to u24 example is exactly related to utf-8 that covers all the languages but excludes the emojis. I'm not familiar with Zig, so maybe it's doing something weird here, but that doesn't really make sense with Unicode in general. First, the largest Unicode codepoint that will ever be allocated is U+10FFFF [0], which is less than 2^21, so all Unicode characters will fit in a 24-bit integer.…

Note the utf-8[0] in my response, the answers are on the pages you linked, but not in the sections you linked, utf-8 encodes code points in one to four bytes , it is byte oriented vs utf-16 etc. In zig u8 is a byte, and is also (by convention) a char, although there isn't an explicit char type in zig. Technically there are chars in languages that need all 4 bytes in utf-8, but almost all of them are historical or emo…

> utf-8 encodes code points in one to four bytes, it is byte oriented vs utf-16 etc. In zig u8 is a byte, and is also (by convention) a char, although there isn't an explicit char type in zig. […]

> 24bits (3 bytes) in utf-8 gets you Chinese, Japanese, Korean. 16 bits (2 bytes) gets you Latin letters with diacritics, Greek, and Arabic scripts. With 8 bits (1 byte) getting you Standard ASCII etc...

Ah ok, so if I understand you correctly, you're taking a variable-length encoding (UTF-8), and limiting and/or padding it to 3 octets (24 bits)? In that case, what you said in your original post makes sense, but I'm not really sure why you'd ever want to encode something this way: you have to deal with the complexities of a variable-length encoding to parse each u24, you have the poor space usage of a fixed-length encoding, and you're using 24 bits to encode only 0xFFFF characters (even though you can fit all of Unicode in only 21 bits).

> Technically there are chars in languages that need all 4 bytes in utf-8, but almost all of them are historical or emoji's in utf-8.

Yes, the majority of the characters in the non-BMP planes are for archaic languages, but that's not really the right way to look at it, since most languages only need lots more [1] [2] [3] [4] [5] [6].

Now, it's fine to not support these characters, but the argument in that case should be that you've decided that the characters aren't important enough to outweigh the technical challenges, not that nobody needs the characters.

> 24bits (3 bytes) in utf-8 gets you Chinese, Japanese, Korean.

It gets you a subset of CJK that's probably sufficient for many purposes, but there are nearly 75k CJK characters outside of the BMP.

> There is a point you could make that it may have been better to use utf-16 etc... and that we should have dropped ascii/latin-1 support, but once again go up to the 'Basic Multilingual Plane' in your [3] and notice that is covered by 24bits (3 bytes) in utf-8 encoding.

If you are willing and able to use a 24-bit encoding, then I'd argue that you should just use UCS-3/UTF-24, since those allow you to encode every Unicode character. The only downside is that these encodings aren't formally-defined so other programs won't understand them, but if that's an issue you can use UCS-4/UTF-32.

[0]: https://news.ycombinator.com/item?id=48682043

[1]: https://en.wikipedia.org/wiki/Unified_Canadian_Aboriginal_Sy...

[2]: https://en.wikipedia.org/wiki/Chakma_(Unicode_block)

[3]: https://en.wikipedia.org/wiki/Mro_(Unicode_block)

[4]: https://en.wikipedia.org/wiki/Kirat_Rai_(Unicode_block)

[5]: https://en.wikipedia.org/wiki/Nag_Mundari_(Unicode_block)

[6]: https://en.wikipedia.org/wiki/Ethiopic_Extended-B

Re: Zig's new bitCast semantics and LLVM back end improvements

#108
post #67

This change + the existing packed struct logic will be great for working with bit packed binary headers w/o having to manually twiddle so much about the bit handling along the way.

It's so interesting to read comments like this and contrast them with the "don't read the code" type of vibes out right now. It feels like half of the developer world is optimizing low-level struct packing and the other half is YOLO'ing 300 KLOC Electron apps. Very confusing.

I think it makes sense, if one sees that LLMs exposed various pre-existing splits in the developer world.

Those who viewed code as a means to build something else, are happy to switch to LLMs if they can build that something faster/cheaper.

Whereas, those who liked coding for its own sake, don't want to use LLMs, and fear for their jobs and their happiness.

Unfortunately for the latter group, we're moving to a world where most development is done by LLMs, and only cutting-edge or hobbyist work is done manually. E.g., Japanese artisanal wood-working and joinery is beautiful and elegant... but modern carpentry doesn't build that way.

Re: Zig's new bitCast semantics and LLVM back end improvements

#109
post #97
post #85

Earlier quoted context omitted.

The ease of dealing with arbitrary bit-width integers and packed structs is actually one of the 'killer features' for me in zig. Zig natively supports arbitrary bit-width integers, the ABI is defined and you could simply think it as a slice of the next larger backing integer. The[3]u8 to u24 bitCast will simply be backed by a 32bit int, using the same ABI. As you have u1 - u65535, sometimes it can be multiple words.…

> ... utf-8 that covers all the languages but excludes the emojis ... Ah, but the U+0000 to U+FFFF plane does not cover all the languages. You might think that only historical and archaic languages are found in Unicode's astral planes (e.g., U+20000 to U+2A6DF is used for historical Chinese characters no longer used today), but in fact there are modern languages found in the U+10000 plane. You might not care about Os…

> You don't lose half a million people from an ethnic group in just ten years without some kind of war or genocide.

Nothing happened to the people, they are growing year on year. But languages can die very easily if governments don't put efforts on teaching it to children. That is exactly what happened to the Ho language. There is no advantage on learning these small regional languages so children put their effort on more popular languages like Hindi, Odia and English.

Here is a good article on this topic:

https://www.vogue.in/content/when-languages-in-india-disappe...

Re: Zig's new bitCast semantics and LLVM back end improvements

#110
post #98
post #96

Earlier quoted context omitted.

Note the utf-8[0] in my response, the answers are on the pages you linked, but not in the sections you linked, utf-8 encodes code points in one to four bytes , it is byte oriented vs utf-16 etc. In zig u8 is a byte, and is also (by convention) a char, although there isn't an explicit char type in zig. Technically there are chars in languages that need all 4 bytes in utf-8, but almost all of them are historical or emo…

> ... but almost all of them are historical or emoji's in utf-8. I just posted a comment, five minutes after you wrote that, which I won't repeat here since it was quite long. But one of the languages whose alphabet is found in the higher multilingual plane is Fulani, spoken natively by 37 million people (plus another two and a half million who have learned it as a second language). While it can be written in other a…

To be clear, I was talking about a use case, not all use cases.

There are very real times where you have to support all 4 bytes, there are others where other drivers require you to restrict the domain of discorse.

It doesn't change the value/cost of bit casting in a language with arbitrary bit width languages, especially when combined with the fact that int overflows are detectable illegal behaviour and you have saturating and wrapping operators.

This is in addition to the ease of using packed structs I mentioned above.

A list of some advantages:

* Zig's arbitrary-sized integers have a fully defined ABI for padding

* Allows for strict domain modeling using them as platform independent refinement types

* Precise memory packing, allowing more utilization of register space etc...

* OOB compile time checks

* Bit masking optimization, where sequential changes to packed values are often merged into a small number of and/or masks

To move to a more information theory example:

DNA nucleotides (A, C, G, T) represents quaternary state pairs.

If you wanted to store an array of 1,000 DNA nucleotides, each symbol is one of 4 bases, requiring exactly 2 bits of information. The Shannon Information would be: 1000 * 2bits = 2000 bits.

With uint8_t this would take 8k bits, vs 2k bits of u2. That is 300% more for uint8_t.

It is still horses for courses, but as an example consider 12-bit sensor reading in a standard u16, the data type allows invalid states. To ensure safety, requires manual defensive logic throughout your program in the traditional C/Rust/...

That traditional model in zig:

     fn processSensor(value: u16) !void {
         if (value > 4095) return error.InvalidSensorData; // Extra logic branch
         // ... logic ...
And the lower overall Kolmogorov complexity (cherry picked) form:

     fn processSensor(value: u12) void {
         // Zero validation boilerplate code required here

C23 does have _BitInt types for structs which can help if bit packing is your primary need, IMHO it doesn't offer the same advantages.

As an example, and I may be wrong, but I think you cant easily perform checked arithmetic or use standard overflow operations on individual C bit-fields without copying them out into standard standard types (like int), modifying them, masking them, and copying them back.

With Zig the invariant is maintained implicitly at the type layer, removing runtime validation branches, error paths, and testing code

Does it solve all problems, no. Is @bitCast, a zero runtime overhead, compile-time checked bit reinterpretation and [3]u8 \to u24 useless and silly, no.

Post reply on HN