Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

So you think you know what a number is

chris.improbable.org

Post reply on HN
Print view View original

31–35 of 35 posts

Re: So you think you know what a number is

#32

Post by Tistron »

evincarofautumn wrote: post #26
Tistron wrote: post #20

Earlier quoted context omitted.

Hmm, interesting. What's up with the terminal here? The shell is doing something to the text. $ python2.7 -c "print(int(u'۲۶۷۹'))" Traceback (most recent call last): File " ", line 1, in ValueError: invalid literal for int() with base 10: '\xdb\xb2\xdb\xb6\xdb\xb7\xdb\xb9' $ python2.7 Python 2.7.13 (default, Jan 03 2017, 17:41:54) [GCC] on linux2 Type "help", "copyright", "credits" or "license" for more information.…

DB B2, etc. are the UTF-8 encodings of U+06F2, etc. So Python is seeing mojibake: U+00DB (Û), U+00B2 (²), etc. which are not digits. Well, one of them kinda is, but it’s No (“Number, other”), not Nd (“Number, decimal digit”).

Yeah I get that, but why is that happening?

Re: So you think you know what a number is

#33

Post by evincarofautumn »

Tistron wrote: post #32
evincarofautumn wrote: post #26

Earlier quoted context omitted.

DB B2, etc. are the UTF-8 encodings of U+06F2, etc. So Python is seeing mojibake: U+00DB (Û), U+00B2 (²), etc. which are not digits. Well, one of them kinda is, but it’s No (“Number, other”), not Nd (“Number, decimal digit”).

Yeah I get that, but why is that happening?

I’d guess because CPython is assuming the input to -c is ISO-8859-1 (Latin-1) when it decodes it using Py_DecodeLocale():

    main()
      …
      setlocale(LC_ALL, "")
      …
      argv_copy[i] = Py_DecodeLocale(argv[i], NULL)
        …
        mbstowcs() or mbrtowc()
        …
      setlocale(LC_ALL, oldloc)
      …
      Py_Main(argc, argv_copy)
While the REPL’s encoding (sys.stdin.encoding) is set to UTF-8 due to LANG/LC_CTYPE settings. You can get the same error when invoking the REPL as:

    LANG="en_US.iso8859-1" python2.7
So the shell isn’t doing anything to the text—it’s providing UTF-8 bytes in both cases, it’s just that Python is interpreting them differently.

Re: So you think you know what a number is

#34

Post by gumby »

At least AFAIK all the non-CJKV scripts (except perhaps Mongolian?) use right-to-left decimal characters with 0. So you should be able to "transliterate" (transnumerate) ۲۶۷۹ to 2979 with a simple look up table and no R-L confusion. In fact ۲9۷9 should be the same.

Re: So you think you know what a number is

#35

Post by inimino »

Grue3 wrote: post #23

int("一万三千二百六十九") Traceback (most recent call last): File "python", line 1, in ValueError: invalid literal for int() with base 10: '一万三千二百六十九' int("一三二六九") Traceback (most recent call last): File "python", line 1, in ValueError: invalid literal for int() with base 10: '一三二六九' Well, that was disappointing. For the record, my site http://ichi.moe/ can handle both (and Arabic numerals too).

But those are the names of numbers, like "one", and not numerals. "one two three" would be the English equivalent.
Post reply on HN