Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Full Unicode Search at 50× ICU Speed with AVX‑512

ashvardanian.com

Post reply on HN
Print view View original

71–80 of 80 posts

Re: Full Unicode Search at 50× ICU Speed with AVX‑512

#71

Post by janc_ »

ashvardanian wrote: post #17
orthoxerox wrote: post #9

Is it possible to extend this to support additional transformation rules like Any-Latin;Latin-ASCII? To make it possible to find "Վարդանյան" in a haystack by searching for "vardanyan"?

Yes — fuzzy and phonetic matching across languages is part of the roadmap. That space is still poorly standardized, so I wanted to start with something widely understood and well-defined (ICU-style transforms) before layering on more advanced behavior. Also, as shown in the later tables, the Armenian and Georgian fast paths still have room for improvement. Before introducing higher-level APIs, I need to tighten the e…

Even when transliteration is somewhat de-facto standardised, it usually is dependent on the target/host language. So e.g. Arabic & Russian are transliterated differently in e.g. English, French, German, Dutch, etc.

Re: Full Unicode Search at 50× ICU Speed with AVX‑512

#72

Post by janc_ »

andersa wrote: post #11
b2ccb2 wrote: post #4

Earlier quoted context omitted.

The confusion likely stems from the relatively new introduction of the capitalized ẞ https://de.wikipedia.org/wiki/Gro%C3%9Fes_%C3%9F Maß capitalized (used to be) MASS. Funnily enough, Mass means one liter beer (think Oktoberfest).

It's strange, because I would expect "maß" as the case insensitive search to match "MASS" in the search text, but "mass" should not match "Maß".

I think all of those should be "tentative matches" for each other.

Re: Full Unicode Search at 50× ICU Speed with AVX‑512

#73

Post by ahoka »

bigwheels wrote: post #49
bee_rider wrote: post #46

Earlier quoted context omitted.

They seem to have (if I understand correctly) degree-Celsius and degree-Fahrenheit symbols. So maybe Kelvin is included for consistency, and it just happens to look identical to Latin K? IMO the confusing bit is giving it a lower case. It is a symbol that happens to look like an upper case, not an actual letter…

And why can't the symbol be a regular old uppercase "K"? Who is this helping?

Probably useful in a non-latin codeset?

Re: Full Unicode Search at 50× ICU Speed with AVX‑512

#74

Post by UltraSane »

bigwheels wrote: post #49
bee_rider wrote: post #46

Earlier quoted context omitted.

They seem to have (if I understand correctly) degree-Celsius and degree-Fahrenheit symbols. So maybe Kelvin is included for consistency, and it just happens to look identical to Latin K? IMO the confusing bit is giving it a lower case. It is a symbol that happens to look like an upper case, not an actual letter…

And why can't the symbol be a regular old uppercase "K"? Who is this helping?

having a dedicated Kelvin symbol preserves the semantics.

Re: Full Unicode Search at 50× ICU Speed with AVX‑512

#76

Post by oneshtein »

bigwheels wrote: post #49
bee_rider wrote: post #46

Earlier quoted context omitted.

They seem to have (if I understand correctly) degree-Celsius and degree-Fahrenheit symbols. So maybe Kelvin is included for consistency, and it just happens to look identical to Latin K? IMO the confusing bit is giving it a lower case. It is a symbol that happens to look like an upper case, not an actual letter…

And why can't the symbol be a regular old uppercase "K"? Who is this helping?

A symbol may look differently than original letter, for example N - №, € - E (Є), S - $, integral, с - ©, TM - ™, a - @, and so on.

However, those symbols doesn't have lower case variants. Moreover, lower case k means kilo-, not a «smaller Kelvin».

Re: Full Unicode Search at 50× ICU Speed with AVX‑512

#77

Post by zvr »

ashvardanian wrote: post #19

This article is about the ugliest — but arguably the most important — piece of open-source software I’ve written this year. The write-up ended up long and dense, so here’s a short TL;DR: I grouped all Unicode 17 case-folding rules and built ~3K lines of AVX-512 kernels around them to enable fully standards-compliant, case-insensitive substring search across the entire 1M+ Unicode range, operating directly on UTF-8 by…

Thank you for this, and congrats on your achievement!

Re: Full Unicode Search at 50× ICU Speed with AVX‑512

#78

Post by peterfirefly »

bbminner wrote: post #45

I was really confused about the case folding, this page explained the motivation well https://jean.abou-samra.fr/blog/unicode-misconceptions """ Continuing with the previous example of “ß”, one has lowercase("ss") != lowercase("ß") but uppercase("ss") == uppercase("ß"). Conversely, for legacy reasons (compatibility with encodings predating Unicode), there exists a Kelvin sign “K”, which is distinct from the Latin upp…

> One question I have is why have Kelvin sign that is distinct from Latin K and other indistinguishable symbols?

To allow round-tripping.

Unicode did not win by being better than all previously existing encodings, even though it clearly was.

It won by being able to coexist with all those other encodings for years (decades) while the world gradually transitioned. That required the ability to take text in any of those older encodings and transcode it to Unicode and back again without loss (or "gain"!).

Re: Full Unicode Search at 50× ICU Speed with AVX‑512

#79

Post by bee_rider »

zygentoma wrote: post #57
bee_rider wrote: post #50

Earlier quoted context omitted.

I think just using uppercase Latin K is the recommendation. But, I dunno. Why would anybody apply upper or lower case operators to a temperature measurement? It just seems like a nonsense thing to do.

Maybe not for text to be read again, but might be sensible e.g. for slug or file name generation and the like...

That’s an interesting thought.

IMO this is somewhere where if we were really doing something, we might as well go all the way and double check the relevant standards, right? The filesystem should accept some character set for use as names, and if we’re generating a name inside our program we should definitely find a character set that fits inside what the filesystem expects and that captures what we want to express… my gut says upper case Latin K would be the best pick if we needed to most portably represent Kelvin in a filename on a reasonably modern, consumer filesystem.

Re: Full Unicode Search at 50× ICU Speed with AVX‑512

#80

Post by bee_rider »

oneshtein wrote: post #76
bigwheels wrote: post #49

Earlier quoted context omitted.

And why can't the symbol be a regular old uppercase "K"? Who is this helping?

A symbol may look differently than original letter, for example N - №, € - E (Є), S - $, integral, с - ©, TM - ™, a - @, and so on. However, those symbols doesn't have lower case variants. Moreover, lower case k means kilo-, not a «smaller Kelvin».

Although it is a prefix in that case, so we should expect to see k alone.

To maximally confuse things, I suggest we start using little k alone to resolve another annoying unit issue: let’s call 1 kilocalorie “k.”

Post reply on HN