It’s not wrong that "\u{1F926}\u{1F3FC}\u200D\u2642\uFE0F".length == 7 (2019)

Post by koakuma-chan » Fri, Aug 22, 2025, 6:26 PM UTC

simonask wrote: post #97

spyrja wrote: post #56

Earlier quoted context omitted.

Just as an example of what I am talking about, this is my current UTF-8 parser which I have been using for a few years now. bool utf_append_plaintext(utf* result, const char* text) { #define msk(byte, mask, value) ((byte & mask) == value) #define cnt(byte) msk(byte, 0xc0, 0x80) #define shf(byte, mask, amount) ((byte & mask) = 3) && (nxt[0] == 0xef) && (nxt[1] == 0xbb) && (nxt[2] == 0xbf)) nxt += 3; while (nxt Not exa…

That's a reasonable implementation in my opinion. It's not that complicated. You're also apparently insisting on three-letter variable names, and are using a very primitive language to boot, so I don't think you're setting yourself up for "maintainability" here. Here's the implementation in the Rust standard library: https://doc.rust-lang.org/stable/src/core/str/validations.rs... It even includes an optimized fast pa…

Post by ramses0 » Fri, Aug 22, 2025, 6:31 PM UTC

arcticbull wrote: post #35

DavidPiper wrote: post #11

I think that string length is one of those things that people (including me) don't realise they never actually want. In a production system, I have never actually wanted string length. I have wanted: - Number of bytes this will be stored as in the DB - Number of monospaced font character blocks this string will take up on the screen - Number of bytes that are actually being stored in memory "String length" is just a…

Taking this one step further -- there's no such thing as the context-free length of a string. Strings should be thought of more like opaque blobs, and you should derive their length exclusively in the context in which you intend to use it. It's an API anti-pattern to have a context-free length property associated with a string because it implies something about the receiver that just isn't true for all relevant usage…

Post by acuozzo » Fri, Aug 22, 2025, 6:33 PM UTC

mid-kid wrote: post #104

osener wrote: post #88

Python does an exceptionally bad job. After dragging the community through a 15-year transition to Python 3 in order to "fix" Unicode, we ended up with support that's worse than in languages that simply treat strings as raw bytes. Some other fun examples: https://gist.github.com/ozanmakes/0624e805a13d2cebedfc81ea84...

Yeah I have no idea what is wrong with that. Python simply operates on arrays of codepoints, which are a stable representation that can be converted to a bunch of encodings including "proper" utf-8, as long as all codepoints are representable in that encoding. This also allows you to work with strings that contain arbitrary data falling outside of the unicode spectrum.

Post by zahlman » Fri, Aug 22, 2025, 6:37 PM UTC

xigoi wrote: post #32

eru wrote: post #25

Earlier quoted context omitted.

Python 3 deals with this reasonable sensibly, too, I think. They use UTF-8 by default, but allow you to specify other encodings.

I prefer languages where strings are simply sequences of bytes and you get to decide how to interpret them.

Post by rileymat2 » Fri, Aug 22, 2025, 6:41 PM UTC

bigstrat2003 wrote: post #152

baq wrote: post #17

Earlier quoted context omitted.

ASCII is very convenient when it fits in the solution space (it’d better be, it was designed for a reason), but in the global international connected computing world it doesn’t fit at all. The problem is all the tutorials, especially low level ones, assume ASCII so 1) you can print something to the console and 2) to avoid mentioning that strings are hard so folks don’t get discouraged. Notably Rust did the correct th…

> in the global international connected computing world it doesn’t fit at all. Most people aren't living in that world. If you're working at Amazon or some business that needs to interact with many countries around the globe, sure, you have to worry about text encoding quite a bit. But the majority of software is being written for a much narrower audience, probably for one single language in one single country. There…