Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Unicode Normalization Forms: When ö ≠ ö

blog.opencore.ch

Post reply on HN
Print view View original

141–144 of 144 posts

Re: Unicode Normalization Forms: When ö ≠ ö

#141

Post by rob_c »

jodrellblank wrote: post #103
rob_c wrote: post #81

Earlier quoted context omitted.

Have to agree. It's also usually only about 10 lines of code to support both insensitive and sensitive searching for those who can't read English that way.

You've done the same thing here as your other comment. Here suggesting that people "can't read English" and in your other comment suggesting that people "can't get their head around capslock and don't deserve support". What about people who CAN read English that way, but think having to match case when searching or referencing text hinders more than it helps?

Then the search tool should support not being case sensitive. I understand the efficiency of case insensitive search (otherwise Google's empire wouldn't exist). But having it enforced as the source of all truth is just broken.

Re: Unicode Normalization Forms: When ö ≠ ö

#142

Post by rob_c »

jodrellblank wrote: post #80
rob_c wrote: post #25

Earlier quoted context omitted.

WORD, Word WoRD.... Sorry to say I tend to use case sensitivity as a filter for me offering support to other developers. I'm not willing to find time for people who can't get their head around "turn on/off caps lock". You don't do it in professional writeups or applications (and I hope not in a CV) so don't pollute my filesystems or codebases with that madness.

I’m not talking about caps lock. I can get my head around case sensitivity, I can use it, it’s worse, I don’t want to have to use it anymore than I want to use filesystem permissions in octal even though I can. Having tools take chmod u+r is easier and doesn’t change the filesystem at all.

Sorry, not sure I see the point here other than computers provide human representation of binary data?

If the mapping is non trivial then unless you're careful you end up breaking basic consistency between input and stored data hence the weird issues with mangling the unicode chars. If the mapping is trivial theres almost nothing to discuss. If the mapping is many to many you're going to have a bad time unless your consistent with your use of the maps. Then the fun is broken mappings where you get data loss due to incorrect many to one and one to many mappings.

There are times when caps matter, I e. code and filesystems are human readable so should not be arbitrary, but searching these for instance makes sense to be insensitive when needed (perhaps even default)

Re: Unicode Normalization Forms: When ö ≠ ö

#143

Post by jrochkind1 »

nieve wrote: post #60
jrochkind1 wrote: post #43

Earlier quoted context omitted.

Well, precisely because if you don't normalize the filenames, ö ≠ ö. You could have two files with different filenames, `göteborg.txt` and `göteborg.txt`, and they are different files with different filenames. Or you could have one file `göteborg.txt`, and when you try to ask for it as `göteborg.txt`, the system tells you "no file by that name". Unicode normalization is the solution to this. And the unicode normal…

It looks like instead of the config option switching everything to use the same normalization it keeps a second copy of the name in a database to compare to. What a horrible kludge, I wonder how they even got into this situation of using different normalization in different parts of the system?

That seems an odd choice indeed, because even if you do have different normalizations in differnet parts of the system, you don't need to keep multiple copies -- you just need to apply the right normalization in the right place. All of the unicode normalization algorithms are both idempotent and of course completely deterministic. If you apply NFD to any legal input, you get the same thing every time -- there's no need to store the NFC version separately to compare it to NFC input when all you have is NFD otherwise, you can just normalize the input to NFD to compare it to what you have!

Unless it was meant to be for performance?

Post reply on HN