Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

The best – but not good – way to limit string length

adam-p.ca

Post reply on HN
Print view View original

21–30 of 48 posts

Re: The best – but not good – way to limit string length

#21

Post by jerf »

I had this problem recently, in logging email subjects into something that has a defined byte limit size. I went for iterating on graphemes and fitting as many complete graphemes into the bytes as I could, and then stopping. The idea is, don't show broken graphemes and fit as much as I can.

This approach probably solves most programmer problems with length. However if this has to be surfaced to an end-user who is not intimately familiar with the nature of Unicode encodings, which is, you know, basically everybody, it may be difficult to explain to them what the limits actually mean in any sensible way. About all you can do is maybe give vague hints about it being nearly too long and avoid being precise enough for there to be a problem. There doesn't seem to me to be a perfect solution here, the intrinsic problem of there being no easy to explain the lengths of these things to end-users and no reason to ever expect them to understand it seems fundamental to me.

Re: The best – but not good – way to limit string length

#22

Post by gcau »

wavemode wrote: post #12

In the age of unicode (and modern computing in general), all of this is more headache than it's worth. What is actually important is that you limit the size of an HTTP request to your server (perhaps making some exceptions for file upload endpoints). As long as the user's form entries fit within that, let them do what they want.

I don't it's practical or useful to just say "limit the size of entire requests" and just ignore all the real world reasons you'd want to actually validate/check data before putting it in your database. The logic you're using is how we have bugs and security holes. This persons write-up gives specific and detailed information that's genuinely useful.

Re: The best – but not good – way to limit string length

#23

Post by [deleted] »

gcau wrote: post #22
wavemode wrote: post #12

In the age of unicode (and modern computing in general), all of this is more headache than it's worth. What is actually important is that you limit the size of an HTTP request to your server (perhaps making some exceptions for file upload endpoints). As long as the user's form entries fit within that, let them do what they want.

I don't it's practical or useful to just say "limit the size of entire requests" and just ignore all the real world reasons you'd want to actually validate/check data before putting it in your database. The logic you're using is how we have bugs and security holes. This persons write-up gives specific and detailed information that's genuinely useful.

[deleted]

Re: The best – but not good – way to limit string length

#24

Post by wpollock »

Best advice I've heard is to never use the character type in your programming language. Instead, store characters in strings. An array of strings can be used as a string of characters. In this approach, characters become opaque blobs of bytes. This makes it easy to get the two numbers you care about: length in characters and size in bytes.

There is some overhead for this, so maybe a technique more suited to backends. Normalization, sanitation and validation steps are best performed in the frontend.

Also worth knowing is the ICU library, which is often the easiest way to work with Unicode consistently regardless of programming language.

Finally, punycode is a standard way to represent arbitrary Unicode strings as ASCII. It's reversible too (and built into every web browser). You can do size limits on the punycode representation.

BTW, you shouldn't store passwords in strings in the first place. Many programming languages have an alternative to hold secrets in memory safely.

Re: The best – but not good – way to limit string length

#25

Post by wild_egg »

wpollock wrote: post #24

Best advice I've heard is to never use the character type in your programming language. Instead, store characters in strings. An array of strings can be used as a string of characters. In this approach, characters become opaque blobs of bytes. This makes it easy to get the two numbers you care about: length in characters and size in bytes. There is some overhead for this, so maybe a technique more suited to backends.…

> validation steps are best performed in the frontend.

I'm really hoping we have very different definitions of "frontend"

Re: The best – but not good – way to limit string length

#26

Post by wpollock »

wild_egg wrote: post #25
wpollock wrote: post #24

Best advice I've heard is to never use the character type in your programming language. Instead, store characters in strings. An array of strings can be used as a string of characters. In this approach, characters become opaque blobs of bytes. This makes it easy to get the two numbers you care about: length in characters and size in bytes. There is some overhead for this, so maybe a technique more suited to backends.…

> validation steps are best performed in the frontend. I'm really hoping we have very different definitions of "frontend"

I meant the web server, not in the end user's browser! (So by backend, I meant the application and data layers.)

Re: The best – but not good – way to limit string length

#27

Post by frizlab »

wpollock wrote: post #24

Best advice I've heard is to never use the character type in your programming language. Instead, store characters in strings. An array of strings can be used as a string of characters. In this approach, characters become opaque blobs of bytes. This makes it easy to get the two numbers you care about: length in characters and size in bytes. There is some overhead for this, so maybe a technique more suited to backends.…

Swift’s Character type represents an extended grapheme cluster, which is the correct thing to do.

Re: The best – but not good – way to limit string length

#28

Post by fsckboy »

poincaredisk wrote: post #5
neuroelectron wrote: post #4

This is why my website is going to be ASCII only.

Which is a reasonable and clean solution - I love simplicity of ASCII like every programmer does. Except ASCII is not enough to represent my language, or even my name. Unicode is complex, but I'm glad it's here. I'm old enough to remember the absolute nightmare that was multi-language support before Unicode and now the problem of encodings is... almost solved.

>ASCII is not enough to represent my language, or even my name.

Hebrew and Arabic don't include vowels. While you think that writing your language needs vowels, we can tell from the existence of Hebrew and Arabic that you are probably wrong. It would take some getting used to, but just like that "scramble the letters in the middle of words, you can still read":

https://www.sciencealert.com/word-jumble-meme-first-last-let...

>Aocdrnig to a rscheearch at Cmabrigde Vinervtisy, it deosn't mttaer in waht oredr the Itteers in a wrod are, the olny iprmoetnt ting is taht the frist and Isat Itteer be at the rghit pclae. The rset can be a toatl mses and you can sitll raed it wouthit porbelm. Tihs is bcusee the huamn mnid deos not raed ervey teter by istlef, but the wrod as a wlohe.

your language, too, is redundant and could be modified to be simpler to write.

I'm not asking you to write your language with no vowels, I'm simply saying you could reduce to ASCII, get used to it, and civilization could move on. Stop clinging to the past, you are holding up the flying cars.

Re: The best – but not good – way to limit string length

#29

Post by fsckboy »

wpollock wrote: post #24

Best advice I've heard is to never use the character type in your programming language. Instead, store characters in strings. An array of strings can be used as a string of characters. In this approach, characters become opaque blobs of bytes. This makes it easy to get the two numbers you care about: length in characters and size in bytes. There is some overhead for this, so maybe a technique more suited to backends.…

>length in characters and size in bytes

you change the word you use as if those words have inherent meanings that we can draw upon. they don't.

it would be more clear to write "length in characters and length in bytes"

[linguistically speaking, words don't carry meanings, it is us who ascribe meaning to words. we use words to say what we want to say, but words don't limit us in what we can say]

Re: The best – but not good – way to limit string length

#30

Post by smrq »

fsckboy wrote: post #28
poincaredisk wrote: post #5

Earlier quoted context omitted.

Which is a reasonable and clean solution - I love simplicity of ASCII like every programmer does. Except ASCII is not enough to represent my language, or even my name. Unicode is complex, but I'm glad it's here. I'm old enough to remember the absolute nightmare that was multi-language support before Unicode and now the problem of encodings is... almost solved.

> ASCII is not enough to represent my language, or even my name. Hebrew and Arabic don't include vowels. While you think that writing your language needs vowels, we can tell from the existence of Hebrew and Arabic that you are probably wrong. It would take some getting used to, but just like that "scramble the letters in the middle of words, you can still read": https://www.sciencealert.com/word-jumble-meme-first-las…

English itself lost some lovely letters because of the printing press (RIP, þ), so I suppose simplifying writing systems in the name of technological simplicity isn't unprecedented.
Post reply on HN