Python 3: Ten years later • Hacker News Board

Re: Python 3: Ten years later

#91

Post by the_mitsuhiko » Sun, Dec 23, 2018, 4:38 PM UTC

wbond wrote: post #66

the_mitsuhiko wrote: post #35

Earlier quoted context omitted.

I think this is largely a second system result. If you want correct unicode handling it's not harder in 2.x than in 3.x. The main reason unicode support was more complex in 2.x for the most part is that many APIs supported working with Bytes and on 3.x that burden is pushed to the API consumer. > Trying to guess encoding at runtime was pretty error prone, with Python 3 it just works. You should not guess on 2.x eithe…

At first blush you would think so, but I know for a fact that Unicode on Windows is not possible to properly handle below Python 3.4. Python 3’s Unicode handling has been a huge win for cross-platform deployments in my experience. We run a patched version on 3.3 with some backports from 3.4 on millions of machines, and it is actually possible to support non-Latin filesystem paths, where that was just not possible whe…

Click uses ctypes to retrofit some unicode support on 2.x for what it’s worth.

Re: Python 3: Ten years later

#92

Post by rurban » Sun, Dec 23, 2018, 8:34 PM UTC

avian wrote: post #49

rurban wrote: post #45

Earlier quoted context omitted.

The python nightmare is purely performance due to its use of the UCS-2 or 4 encoding (wchar_t without surrogate pairs). UTF-8 was ruled out for its inability for direct indexing, but this proved wrong. 4x longer string buffers trash the cache much more than the needed CPU cycles for utf-8 decoding. perl's problem is that they didn't make room for the original encoding, it's only latin-1 or utf-8 from everything else.…

> it's only latin-1 or utf-8 There’s no other way, is it? You basically have to have the complete string library re-implemented for each supported multi-byte encoding.

No. Foreign encodings are internally converted to utf-8 via the Encoding library and IO layers. The string library is only done for latin-1 and utf-8.

The only trouble is that you have to remember the original encoding, if you need that information. It's stored nowhere. E.g. to write to a file in the original encoding you have to open it with the correct IO layer. perl6 got that fixed.

Re: Python 3: Ten years later

#93

Post by slavik81 » Thu, Dec 27, 2018, 12:00 AM UTC

imtringued wrote: post #90

slavik81 wrote: post #53

Earlier quoted context omitted.

I saved this page from the English Wikipedia as input.txt: https://en.wikipedia.org/wiki/Shinz%C5%8D_Abe C:\Users\local\tmp>python --version Python 3.7.1 C:\Users\local\tmp>python foo.py Diet Parliament Traceback (most recent call last): File "foo.py", line 7, in text = f.read() File "C:\Users\local\AppData\Local\Programs\Python\Python37-32\lib\encodings\ cp1252.py", line 23, in decode return codecs.charmap_decode(in…

Well as you can see the problem is that open without encoding arguments uses the system default settings (in this case cp1252) which is always a terrible idea regardless of how a language represents strings internally. Honestly it surprises me how simple and easy it is to understand this trace and I'd agree with the statement that this is a good user experience.

It's acceptable for developers, but when I say "user friendly" I mean friendly to end users, who may not even know that the program is written in Python. A better message would tell users in plain language what the problem is and how they could fix it (e.g. by setting a flag or an environment variable).