Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Unicode 18.0.0 Beta

unicode.org

Post reply on HN
Print view View original

51–60 of 68 posts

Re: Unicode 18.0.0 Beta

#51

Post by numpad0 »

lifthrasiir wrote: post #41
account42 wrote: post #38

Earlier quoted context omitted.

You mean theoretically. Effectively, nothing is fixed yet.

IVD works, theoretically and practically (recent versions of OpenType have an explicit support for them). It's not their fault that Japanese vendors have been not very quick to adopt them.

Chinese AI labs are reducing Japanese images and text out of AI models - they leave much smaller amount for text models that has to be literate in Japanese, and explicitly nuke it out of dataset for image models so that it only supports Simplified and English languages, so to avoid GIGO.

I mean, making or help making sovereign AI models is nowhere near responsibilities of Unicode, but Han Unification and sort of a default-enforced IVD support is literally adding small but non-zero amount of fuel to cultural division and xenophobia perpetuate in East Asia. I doubt blaming users would work here.

Re: Unicode 18.0.0 Beta

#52

Post by xigoi »

wongarsu wrote: post #10
poulpy123 wrote: post #5

looking at the changes it makes me wonder: - is there an usable font the cover all unicode ? - if not is there really a point to include everything possible in unicode ? - how many space is remaining for new alphabet and smileys ? - how do they handle changes in scripts, for example if new proto-cuneiform or seal script symbols are discovered ?

> if not is there really a point to include everything possible in unicode ? Needing to load three fonts to show a single document that mixes vastly different character sets is still infinitely better than not being able to have those different characters in the same .txt or .md file at all > how many space is remaining for new alphabet and smileys ? Unicode can encode about 1100k code points, and about 800k of those…

Also, the 1.1M limit is because of UTF-16. If UTF-16 was deprecated in favor of UTF-8, the limit could be much higher.

Re: Unicode 18.0.0 Beta

#53

Post by lifthrasiir »

numpad0 wrote: post #51
lifthrasiir wrote: post #41

Earlier quoted context omitted.

IVD works, theoretically and practically (recent versions of OpenType have an explicit support for them). It's not their fault that Japanese vendors have been not very quick to adopt them.

Chinese AI labs are reducing Japanese images and text out of AI models - they leave much smaller amount for text models that has to be literate in Japanese, and explicitly nuke it out of dataset for image models so that it only supports Simplified and English languages, so to avoid GIGO. I mean, making or help making sovereign AI models is nowhere near responsibilities of Unicode, but Han Unification and sort of a de…

While I agree that Han Unification is not optimal (and fixing them is a welcoming development), it is already too late to reverse it. Even counter-proposals like TRON didn't work at all so far. IVD is the best compromise we can have in this situation.

> cultural division and xenophobia perpetuate in East Asia

By the way, I recently have seen multiple claims from Japanese Twitter users that Korea would have been better keeping Chinese characters (Hanja) in use. If this is a cultural division and xenophobia we are talking about, I will gladly take it---why on earth do they have any saying in Korea's choice of scripts? The "sinosphere" is an illusion, the fact that CJKV countries have or had shared the same set of characters is just a fun fact and not a cultural mandate or anything else like that.

Re: Unicode 18.0.0 Beta

#54

Post by numpad0 »

lifthrasiir wrote: post #53
numpad0 wrote: post #51

Earlier quoted context omitted.

Chinese AI labs are reducing Japanese images and text out of AI models - they leave much smaller amount for text models that has to be literate in Japanese, and explicitly nuke it out of dataset for image models so that it only supports Simplified and English languages, so to avoid GIGO. I mean, making or help making sovereign AI models is nowhere near responsibilities of Unicode, but Han Unification and sort of a de…

While I agree that Han Unification is not optimal (and fixing them is a welcoming development), it is already too late to reverse it. Even counter-proposals like TRON didn't work at all so far. IVD is the best compromise we can have in this situation. > cultural division and xenophobia perpetuate in East Asia By the way, I recently have seen multiple claims from Japanese Twitter users that Korea would have been bette…

> IVD is the best compromise we can have in this situation.

Maybe, but no one is running an ivdfy-filter through every single Japanese documents and the issue keeps going. Maybe one way to make it happen is to make the Simplified forms singularly canonical to the CJK Unified Ideographs so to classify everything in that form as Chinese, and define Japanese script as being always flagged with IVDs, though I don't know what the storage and processing implication of that might be. But my point is that maintaining the position that users can optionally choose to not display text in a wrong language and Unification issues are merely user errors don't make any sense to me.

> Korea would have been better keeping Chinese characters (Hanja) in use.

I can't speak for all, but I, for one, do regularly encounter machine translation failures in Korean contents due to homophones even with LLM-based ones in the ways that don't happen with Japanese. It manifests as either homonym errors[1] or the MTL resorting to phonetic transcripts that I have no idea about[2]. Both happens in formal writings like newspaper Web articles in addition to casual social media posts. Since it appears that there's no way this issue could happen with "our" system, it sometimes feel like reverting to that could fix it.

1: (like "plain/plane", had the source been English and this was somehow happening)

2: (like "That arm might be fukuzatukossetsushiteru" had the source been Japanese)

Re: Unicode 18.0.0 Beta

#55

Post by duskwuff »

sourcegrift wrote: post #28

Personally the whole emoji thing is an unmitigated disaster. I'm okay with smileys and gestures but everything else is pointless

Personally, I'm all for it. It's been an incredibly effective way of urging developers to support newer Unicode standards.

Re: Unicode 18.0.0 Beta

#56

Post by brikym »

uxellodunum wrote: post #49
brikym wrote: post #40

Earlier quoted context omitted.

This. But more work is needed. I tried a bunch of Discord alternatives like Matrix but very few have a fun experience with things custom emoji images that really make a chat server feel like a home.

There are clients on Matrix that support custom emojii, such as Sable and Commet. Neither are absolutely perfect, but I know people who daily-drive one of the other (or both, which is where I'm at depending on the device). For the most part, now that Matrix is merging those Matrix 2.0 specs finally, and the 2.0 features are already out in the wild with excellent results, it has a really good base, and as expected we'…

That's positive. I think the other feature I wanted was embeds. I know prefetching content can be a security risk but it's super convenient.

Re: Unicode 18.0.0 Beta

#57

Post by extraduder_ire »

Rp8yXmdmr wrote: post #8
orangepanda wrote: post #2

What everyone actually cares about — new emojis! * Cracking face * Left/Right thumb sign * Monarch butterfly * Pickle * Lighthouse * Meteor * Eraser * Net with handle

I would be more interested if they are ever going to cancel HAN unification. Looking at their "Factors for Exclusion" list it could be summarized by "we made some mistakes in past but are sticking to it" :D

Cancel how? There's documents encoded like that which would break it it were changed now.

Unicode takes backward compatibility like this very seriously.

Re: Unicode 18.0.0 Beta

#58

Post by Bolwin »

adzm wrote: post #37
sourcegrift wrote: post #28

Personally the whole emoji thing is an unmitigated disaster. I'm okay with smileys and gestures but everything else is pointless

Personally I think the whole emoji thing is a triumph of Unicode. Being able to convey more subtext through emoji makes communication so much easier especially across language boundaries.

I've never seen emoji used for subtext. Usually they just repeat or emphasize what's in the text

Re: Unicode 18.0.0 Beta

#59

Post by gschizas »

duskwuff wrote: post #55
sourcegrift wrote: post #28

Personally the whole emoji thing is an unmitigated disaster. I'm okay with smileys and gestures but everything else is pointless

Personally, I'm all for it. It's been an incredibly effective way of urging developers to support newer Unicode standards.

Yet Microsoft still refuses to do flags.

Re: Unicode 18.0.0 Beta

#60

Post by hulitu »

xigoi wrote: post #52
wongarsu wrote: post #10

Earlier quoted context omitted.

> if not is there really a point to include everything possible in unicode ? Needing to load three fonts to show a single document that mixes vastly different character sets is still infinitely better than not being able to have those different characters in the same .txt or .md file at all > how many space is remaining for new alphabet and smileys ? Unicode can encode about 1100k code points, and about 800k of those…

Also, the 1.1M limit is because of UTF-16. If UTF-16 was deprecated in favor of UTF-8, the limit could be much higher.

We need UTF-32. For the future.
Post reply on HN