Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Vim 7.3.1000

ftp.vim.org

Post reply on HN
Print view View original

11–20 of 53 posts

Re: Vim 7.3.1000

#11

Post by meritt »

tsm wrote: post #7

I guess I should be thankful 98% of my programming life is in higher-level languages than C, but...am I the only one whose terrified of the magic numbers? In hex ? Can anyone provide context about what's so special about 0xfb20 and 0xfb4f?

http://www.unicode.org/versions/Unicode6.2.0/ch08.pdf page 250

Screenshot of relevant text: http://i.imgur.com/xMnHDg4.png

Basically certain Hebrew characters have descenders which can overlap with diatric marks in the next line of text. These variations are suitable replacements that alter it slightly so it won't overlap.

Re: Vim 7.3.1000

#12

Post by sltkr »

Macha wrote: post #8
tsm wrote: post #7

I guess I should be thankful 98% of my programming life is in higher-level languages than C, but...am I the only one whose terrified of the magic numbers? In hex ? Can anyone provide context about what's so special about 0xfb20 and 0xfb4f?

Given that it's in a regex source file, and the variable is named c, they're probably character codes for Unicode ranges.

Correct -- 0xfb20 through 0xfb4f are alternative glyphs for Hebrew characters. (I don't know what makes these so special that they need to be handled separately by the regexp processing code, though.)

Re: Vim 7.3.1000

#13

Post by sltkr »

jlgreco wrote: post #9
Macha wrote: post #8

Earlier quoted context omitted.

Given that it's in a regex source file, and the variable is named c, they're probably character codes for Unicode ranges.

This might be a good case for unicode character literals. Code dealing with unicode can be a nightmare to work with, even if you already know what those numbers are (arguably as this bug demonstrates).

You realize you're talking about a codebase which still uses pre-standard parameter type declarations, right?

Re: Vim 7.3.1000

#14

Post by jlgreco »

jng wrote: post #5

Pretty ugly bug. Where did it happen? Anyway, congrats & long life!

Apparently this is in the code that decomposes strings so that they can be compared (necessary with Hebrew and Arabic seemingly?).

This is an interesting snippet from that code...:

        /* decompose the character if necessary, into 'base' characters
        * because I don't care about Arabic, I will hard-code the Hebrew
        * which I *do* care about! So sue me... */
        if (c1 != c2 && (!ireg_ic || utf_fold(c1) != utf_fold(c2)))
        {
            /* decomposition necessary? */
            mb_decompose(c1, &c11, &junk, &junk);
            mb_decompose(c2, &c12, &junk, &junk);
            c1 = c11;
            c2 = c12;
            if (c11 != c12 && (!ireg_ic || utf_fold(c11) != utf_fold(c12)))
                break;
        }
Apparently string comparison is harder than I previously thought.

Re: Vim 7.3.1000

#15

Post by jlgreco »

sltkr wrote: post #13
jlgreco wrote: post #9

Earlier quoted context omitted.

This might be a good case for unicode character literals. Code dealing with unicode can be a nightmare to work with, even if you already know what those numbers are (arguably as this bug demonstrates).

You realize you're talking about a codebase which still uses pre-standard parameter type declarations, right?

Oh yes, Vim's code is anything but pleasant. I worked with it a decent amount several years ago when I was maintaining, for a short period of time, a patch that would add a terminal emulator to Vim windows.

Re: Vim 7.3.1000

#16

Post by plorkyeran »

jlgreco wrote: post #14
jng wrote: post #5

Pretty ugly bug. Where did it happen? Anyway, congrats & long life!

Apparently this is in the code that decomposes strings so that they can be compared (necessary with Hebrew and Arabic seemingly?). This is an interesting snippet from that code...: /* decompose the character if necessary, into 'base' characters * because I don't care about Arabic, I will hard-code the Hebrew * which I *do* care about! So sue me... */ if (c1 != c2 && (!ireg_ic || utf_fold(c1) != utf_fold(c2))) { /* de…

String comparison with Unicode is pretty astonishingly complex, partially because equality is not as well defined as it seems to be on the surface. Should e and é be equal? If you're dealing with user input from people who are unlikely to know how to type é, then they probably should, but in many cases they shouldn't. A more complex case is é and é (precomposed vs decomposed forms), which nearly always should be equal, but a simple byte comparison will say they're different.

Fortunately, there are ICU bindings for every non-toy language which solves these sorts of problems for you (although ICU has the drawback of being absolutely huge).

Re: Vim 7.3.1000

#19

Post by jlgreco »

plorkyeran wrote: post #16
jlgreco wrote: post #14

Earlier quoted context omitted.

Apparently this is in the code that decomposes strings so that they can be compared (necessary with Hebrew and Arabic seemingly?). This is an interesting snippet from that code...: /* decompose the character if necessary, into 'base' characters * because I don't care about Arabic, I will hard-code the Hebrew * which I *do* care about! So sue me... */ if (c1 != c2 && (!ireg_ic || utf_fold(c1) != utf_fold(c2))) { /* de…

String comparison with Unicode is pretty astonishingly complex, partially because equality is not as well defined as it seems to be on the surface. Should e and é be equal? If you're dealing with user input from people who are unlikely to know how to type é, then they probably should, but in many cases they shouldn't. A more complex case is é and é (precomposed vs decomposed forms), which nearly always should be equa…

There are a lot of things I have seen in Unicode that seem like they should not exist in the first place. MATHEMATICAL [BOLD|SANS-SERIF|DOUBLE-STRUCK|MONOSPACE] DIGIT for example... I guess those things potentially carry significant meaning in some mathematics texts though.

I guess the ICU stuff probably gives you an strtol equivalent that can handle that sort of stuff.

Re: Vim 7.3.1000

#20

Post by IvyMike »

meritt wrote: post #11
tsm wrote: post #7

I guess I should be thankful 98% of my programming life is in higher-level languages than C, but...am I the only one whose terrified of the magic numbers? In hex ? Can anyone provide context about what's so special about 0xfb20 and 0xfb4f?

http://www.unicode.org/versions/Unicode6.2.0/ch08.pdf page 250 Screenshot of relevant text: http://i.imgur.com/xMnHDg4.png Basically certain Hebrew characters have descenders which can overlap with diatric marks in the next line of text. These variations are suitable replacements that alter it slightly so it won't overlap.

I'd so prefer to see

  #define HEBREW_CHAR_MIN 0xfb20
  #define HEBREW_CHAR_MAX 0xfb4f
(Probably with even more descriptive names, but I don't know the problem domain.)
Post reply on HN