Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Don't stop early: Case-folding source code at memory speed

github.blog

Post reply on HN
Print view View original

11–20 of 40 posts

Re: Don't stop early: Case-folding source code at memory speed

#11

Post by codebje »

pixelesque wrote: post #3

> We deal mostly with source code, so the text we fold is overwhelmingly ASCII and making it run at memory speed is the single most important thing we can do. Everything else just has to keep the rare non-ASCII path from spoiling it. Semi-on-topic: I've noticed that many LLMs via coding agents (ChatGPT and Claude at work with my CoPilot account, and DeepSeek 4 and ChatGPT in pi.dev at home) really seem to like using…

Codebases written by native English speakers are almost exclusively ASCII, but codebases written by speakers of languages other than English frequently have non-ASCII content, even if only in the comments, but languages which support it often wind up with non-ASCII identifiers, too.

I do not believe that emoji like crosses and ticks are particularly common at all, for any language, but LLMs seem to have picked up heavy use of them from somewhere and inserted them into code (and everything else) they generate.

LLM training sets will very likely include the massive corpos of non-English open source code from sites like Gitee, but would be unlikely to generate responses heavily influenced by them unless you've done specific things to make that happen - prompt in Chinese, try to make use of a library only available with Chinese source and/or documentation, perhaps. I've not seen it happen, but I am a light user of LLMs.

Re: Don't stop early: Case-folding source code at memory speed

#12

Post by Groxx »

claudetard wrote: post #2

This is good technical content, but it's obvious that an AI wrote it.

Yea, it's terse and clipped in some sentences, and then changes tone abruptly and randomly, and paragraphs really don't flow together at all. It feels awkward to read, and there is a lot more to read here than there needs to be...

Re: Don't stop early: Case-folding source code at memory speed

#13

Post by Georgelemental »

inigyou wrote: post #5

TLDR: they implemented case folding with a lot more SIMD via autovectorization. > almost every fold preserves the UTF-8 length or shrinks it, but two outliers grow—U+023A (Ⱥ) and U+023E (Ɀ) are 2 bytes each yet fold to 3-byte characters (ⱥ, ɀ) Fix this by reversing it. Fold ⱥ to Ⱥ instead of the other way around. The search index won't only consist of lowercase characters any more, but that never mattered.

> The search index won't only consist of lowercase characters any more

This isn't the case anyway. Unicode case-folding has a few lowercase-to-uppercase mappings, e.g. Cherokee

Re: Don't stop early: Case-folding source code at memory speed

#15

Post by pyentropy »

pixelesque wrote: post #3

> We deal mostly with source code, so the text we fold is overwhelmingly ASCII and making it run at memory speed is the single most important thing we can do. Everything else just has to keep the rare non-ASCII path from spoiling it. Semi-on-topic: I've noticed that many LLMs via coding agents (ChatGPT and Claude at work with my CoPilot account, and DeepSeek 4 and ChatGPT in pi.dev at home) really seem to like using…

A lot of good repos (CLIs, frameworks) had 'tree' unicode directory structure with like ├──, └──, and │ , as well as emojis for passing/failed tests and README docs maybe a unicode arrow or two, but LLMs absolutely overuse it.

I don't know why chatbots prefer → over -> so much. It's becoming a countersignal compared to the old terminal customization era, where arrow ligatures were a signal of effort.

Re: Don't stop early: Case-folding source code at memory speed

#16

Post by nomel »

pixelesque wrote: post #3

> We deal mostly with source code, so the text we fold is overwhelmingly ASCII and making it run at memory speed is the single most important thing we can do. Everything else just has to keep the rare non-ASCII path from spoiling it. Semi-on-topic: I've noticed that many LLMs via coding agents (ChatGPT and Claude at work with my CoPilot account, and DeepSeek 4 and ChatGPT in pi.dev at home) really seem to like using…

Perhaps there was heavy weighting of swift code [1] ;)

I would just paste the example, but HN code block display appears to think it's as unreasonable as I do.

[1] https://wolfmcnally.com/121/programming-with-fruit-using-emo...

Re: Don't stop early: Case-folding source code at memory speed

#18

Post by gwking »

inigyou wrote: post #5

TLDR: they implemented case folding with a lot more SIMD via autovectorization. > almost every fold preserves the UTF-8 length or shrinks it, but two outliers grow—U+023A (Ⱥ) and U+023E (Ɀ) are 2 bytes each yet fold to 3-byte characters (ⱥ, ɀ) Fix this by reversing it. Fold ⱥ to Ⱥ instead of the other way around. The search index won't only consist of lowercase characters any more, but that never mattered.

I wonder if making the index uppercase is strictly better in this sense, or if both upper and lowercase have chars that take more bytes.

Re: Don't stop early: Case-folding source code at memory speed

#19

Post by nextaccountic »

claudetard wrote: post #2

This is good technical content, but it's obvious that an AI wrote it.

I actually came to the comments to share this snippet

> The hot operation isn’t really “fold this character,” it’s “does this character fold?” Almost always no.

Really that's distracting. If you must use LLMs, also do a rewrite pass that removes most LLMisms

Re: Don't stop early: Case-folding source code at memory speed

#20

Post by goalieca »

inigyou wrote: post #5

TLDR: they implemented case folding with a lot more SIMD via autovectorization. > almost every fold preserves the UTF-8 length or shrinks it, but two outliers grow—U+023A (Ⱥ) and U+023E (Ɀ) are 2 bytes each yet fold to 3-byte characters (ⱥ, ɀ) Fix this by reversing it. Fold ⱥ to Ⱥ instead of the other way around. The search index won't only consist of lowercase characters any more, but that never mattered.

Curious if his isn’t some Over-optimization since how often do those characters ever come up.
Post reply on HN