Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

The best – but not good – way to limit string length

adam-p.ca

Post reply on HN
Print view View original

31–40 of 48 posts

Re: The best – but not good – way to limit string length

#31

Post by cjs_ac »

fsckboy wrote: post #28
poincaredisk wrote: post #5

Earlier quoted context omitted.

Which is a reasonable and clean solution - I love simplicity of ASCII like every programmer does. Except ASCII is not enough to represent my language, or even my name. Unicode is complex, but I'm glad it's here. I'm old enough to remember the absolute nightmare that was multi-language support before Unicode and now the problem of encodings is... almost solved.

> ASCII is not enough to represent my language, or even my name. Hebrew and Arabic don't include vowels. While you think that writing your language needs vowels, we can tell from the existence of Hebrew and Arabic that you are probably wrong. It would take some getting used to, but just like that "scramble the letters in the middle of words, you can still read": https://www.sciencealert.com/word-jumble-meme-first-las…

Once again, I request that 95% of the world's population change the way it does almost everything, so that I can simplify my code.

Thank you for writing this comment; it's cleared up some self-esteem issues I've been having about whether I'm clever enough to start my own company.

Re: The best – but not good – way to limit string length

#32

Post by saagarjha »

wpollock wrote: post #24

Best advice I've heard is to never use the character type in your programming language. Instead, store characters in strings. An array of strings can be used as a string of characters. In this approach, characters become opaque blobs of bytes. This makes it easy to get the two numbers you care about: length in characters and size in bytes. There is some overhead for this, so maybe a technique more suited to backends.…

This is generally a bad idea, even if you ignore the obvious overhead from doing so. At some point you are going to create a "real" string out of the thing you have, and it is not going to behave like you expect if you just blindly use the array's properties to compute them. Nor will they really have well defined semantics unless you are careful about what the "characters" you're storing in strings are.

Re: The best – but not good – way to limit string length

#33

Post by saagarjha »

fsckboy wrote: post #28
poincaredisk wrote: post #5

Earlier quoted context omitted.

Which is a reasonable and clean solution - I love simplicity of ASCII like every programmer does. Except ASCII is not enough to represent my language, or even my name. Unicode is complex, but I'm glad it's here. I'm old enough to remember the absolute nightmare that was multi-language support before Unicode and now the problem of encodings is... almost solved.

> ASCII is not enough to represent my language, or even my name. Hebrew and Arabic don't include vowels. While you think that writing your language needs vowels, we can tell from the existence of Hebrew and Arabic that you are probably wrong. It would take some getting used to, but just like that "scramble the letters in the middle of words, you can still read": https://www.sciencealert.com/word-jumble-meme-first-las…

It's hilarious that the guy telling people to go back to ASCII is the one saying "stop clinging to the past".

Re: The best – but not good – way to limit string length

#34

Post by saagarjha »

Retr0id wrote: post #16
aidenn0 wrote: post #15

Earlier quoted context omitted.

They show a single Hindi character that is 15 bytes in UTF-8. That's enough over 10 that it would be believable that Hindi words could get uncomfortably close to the 10x limit.

A single hindi character, yes. But they also mention that only ~25% of hindi characters use combining marks.

Most of them are vowels. They're pretty common. (Also, I feel like you of all people would understand the issues with "only 25% of the time this happens, therefore surprising behavior at the edges is unlikely to happen".)

Re: The best – but not good – way to limit string length

#35

Post by DemocracyFTW2 »

o11c wrote: post #7

Note that normalization involves rearranging combining characters of different combining classes: > Array.from("\u{10FFff}\u0300\u0327".normalize('NFC')).map(x=>x.codePointAt().toString(16)) [ '10ffff', '327', '300' ] If a precombined character exists, the relevant accent will be pulled into the base regardless of where it is in the sequence. Note also that normalization can change the visual length (see below) under…

> * TODO what about Ideographic Description Characters?

I've never encountered them other than rendered with widths like any other CJK character, i.e. with (nominally) double width. There may be software that makes an effort to render IDSes (Ideographic Description Sequences) as existing or generated ideographs (or whacha may call those), but I have yet to see one. There may however, and IMO more likely, be situations where you want to grant the user an input of exactly one, or up to a certain number of CJK characters e.g. for the purpose of searching and grant them the ability to use IDSes for unencoded characters or incompletely known characters. But in that case you're clearly leaving the boundaries of what is Unicode and enter into the grammar of your search engine's customized search strings. Meaning that you probably don't need to handle IDC separately at all other than treating them like any other fullwidth CJK codepoint.

Re: The best – but not good – way to limit string length

#37

Post by chrismorgan »

aidenn0 wrote: post #15
Retr0id wrote: post #9

> The byte size allowed would need to be about 100x the length limit. That’s… kind of a lot? Would it need to be, though? ~10x ought to be enough for any realistic string that wasn't especially crafted to be annoying.

They show a single Hindi character that is 15 bytes in UTF-8. That's enough over 10 that it would be believable that Hindi words could get uncomfortably close to the 10x limit.

Triple conjuncts are very uncommon in Indic scripts, though there are a few in common use, like stri is a single-syllable word that means woman or wife in many languages. Pick your Indic script, and that’ll be LETTER SA, SIGN VIRAMA, LETTER TA, SIGN VIRAMA, LETTER RA, VOWEL SIGN I. Most Indic syllables/grapheme clusters are a single consonant and a single vowel sign, if not the inherent vowel -a. Conjuncts use their script’s SIGN VIRAMA to suppress the inherent vowel and normally graphically join the next consonant (an orthographic choice rarely broken, a little like ß being ss in German).

I’m not so confident about Hindi, though 25% seems very low if we’re talking frequency; but in Telugu writing it’s definitely a lot more than that that specify a vowel sign and thus take at least two Unicode scalar values to represent a syllable.

My feeling (as a white fellow moved to India, with well above average knowledge of Indian languages and Unicode for a place like HN, but not yet fluent in any Indian language) is that some four-bytes-per-code-point script might conceivably get realistic existing texts above an average of 10 bytes per syllable for at least twenty syllables, and that most Indic languages could sustain it indefinitely in specific deliberate styles of writing.

Re: The best – but not good – way to limit string length

#38

Post by chrismorgan »

Another problem is line breaks. Have a ? Line breaks are counted as \n on the client (affecting maxlength attribute and JavaScript calculations using textarea.value.length), but submitted as \r\n. This has bitten me on “2000 character maximum” feedback forms at least twice: client says it’s fine, server says it’s too long, and promptly throws everything away.

Re: The best – but not good – way to limit string length

#39

Post by Retr0id »

saagarjha wrote: post #34
Retr0id wrote: post #16

Earlier quoted context omitted.

A single hindi character, yes. But they also mention that only ~25% of hindi characters use combining marks.

Most of them are vowels. They're pretty common. (Also, I feel like you of all people would understand the issues with "only 25% of the time this happens, therefore surprising behavior at the edges is unlikely to happen".)

That's why you have a limit on both.

Re: The best – but not good – way to limit string length

#40

Post by adam-p »

aidenn0 wrote: post #14

This doesn't seem to cover truncation, but rather acceptance/rejection. If you are given something with "too many" codepoints, but need to use it anyways it seems like it would make sense to truncate it on a grapheme cluster boundary.

I added a section with brief discussion of rejection, truncation, and the perils therein.

https://adam-p.ca/blog/2025/04/string-length/#what-to-do-whe...

Post reply on HN