Live data from Hacker News

Why Is the Migration to Python 3 Taking So Long?

stackoverflow.blog

331–340 of 355 posts

Re: Why Is the Migration to Python 3 Taking So Long?

#331
post #263
post #215

Earlier quoted context omitted.

> Actually that's the behavior of python 2, it works fine, until you send invalid characters then it blows up. Not always. As far as I can tell writing garbage bytes to various APIs works fine unless they explicitly try to handle encoding issues. First time I noticed encoding issues in my code was when writing an xml structure failed on windows, all because of an umlaut in an error message I couldn't care less about.…

What kind of text do you have to process at your job, that you never meet any unicode in it? Nowadays unicode is everywhere, especially with emojis. Even a simple IRC bot needs to handle that.

A lot of scientific/numeric work (up until quite recently, it's slowly, slowly changing) involves text processing of inputs and outputs of other programs, using Python as the glue language.

This is a lot of old code, and it's all ASCII, no matter what the locale of the system is. And even if the code was updated, all the messages would still be in some text == bytes encoding, because there's no "user data" involved, and the throughput desired is in many gigabytes of text processed per second.

So yeah, unicode is not "everywhere": it may be everywhere on the public internet, but there is a world beyond this.

Re: Why Is the Migration to Python 3 Taking So Long?

#332

Earlier quoted context omitted.

I meant the bugs in your dependencies that you'll have to backport fixes to. But hey, if you're writing bug-free code that'll be easy ;)

I understood what you meant. What I'm saying is that my existing code works fine, so whatever bugs are in the dependencies are ones that don't affect me. Should I make a code change that surfaces one, then I have the means to deal with it -- but that is almost certainly going to be a rare event, as my Python projects are stable and aren't going to see much change. I wasn't commenting on how buggy my own code is. Whic…

I get that. But the interesting thing about dependencies is how they surface vulnerabilities that can hurt code that works perfectly well. Your current code probably doesn't have many bugs, but includes an unknown number of vulnerabilities from your dependencies. The bad people probably won't bother examining your code for vulnerabilities, but they will be informed of vulnerabilities in popular libs, and then looking for projects that use those versions of those libs is a lot easier than scanning all those projects individually. So you end up having to backport a bunch of fixes to other people's code because that code was popular and came under intense scrutiny.

But I guess you know this, and are OK with the compromises involved. I'll stop here ;)

Re: Why Is the Migration to Python 3 Taking So Long?

#333

Earlier quoted context omitted.

I understood what you meant. What I'm saying is that my existing code works fine, so whatever bugs are in the dependencies are ones that don't affect me. Should I make a code change that surfaces one, then I have the means to deal with it -- but that is almost certainly going to be a rare event, as my Python projects are stable and aren't going to see much change. I wasn't commenting on how buggy my own code is. Whic…

I get that. But the interesting thing about dependencies is how they surface vulnerabilities that can hurt code that works perfectly well. Your current code probably doesn't have many bugs, but includes an unknown number of vulnerabilities from your dependencies. The bad people probably won't bother examining your code for vulnerabilities, but they will be informed of vulnerabilities in popular libs, and then looking…

> but they will be informed of vulnerabilities in popular libs, and then looking for projects that use those versions of those libs is a lot easier than scanning all those projects individually.

This is true, and if we were talking about code that is exposed to the world at large, then my stance might be different. However, the projects that I've used Python for are not exposed in that way.

Note that I'm talking about personal projects, not work-related ones. At work, I use whatever is required.

Re: Why Is the Migration to Python 3 Taking So Long?

#334
post #316
post #258

Earlier quoted context omitted.

Those issues are common when you're having python 2 code that uses unicode datatype and you have a task to migrate it to python 3. You run your python 2 code on python 3 and it fails, most people at that point will place encode() or decode() in place where you have a failure. When the correct fix would be to place encode/decode at I/O boundary (writing to files (and in python 3 even that is not needed if you open fil…

I've been struggling with this recently when trying to print stdout from subprocess.communicate with code that runs on both 2 and 3. Such a headache - got any recommended reading around this area?

I don't think this is exactly what you're asking but a good starting point:

https://sunscrapers.com/blog/python-best-practices-what-ever...

With 2 vs 3 code is easiest to write your code for python 3 and then in 2 import everything you have in __future__ package including unicode literals. That's still not enough and you still might need to do extra work. In python 3 there's argument encoding, which could do the encoding which doesn't look like it is available in python 2. So you probably shouldn't be use it and treat all input/output as bytes (i.e. call encode() when sending data to stdin, and decode() on what you get back from stdout and stderr).

Perhaps that might be enough for your case, although many things is hard to get right in python 2 even when you know what you should do, because of the implicit conversion.

Edit: this also might be useful: https://unicodebook.readthedocs.io/good_practices.html

Also this could help: https://unicodebook.readthedocs.io/programming_languages.htm...

Re: Why Is the Migration to Python 3 Taking So Long?

#335

Earlier quoted context omitted.

Yes python 3 was clearly a mistake. There could have been less hostile ways to make improvements in the language.

Probably the mistake was not dropping support much sooner. Python 3 came out in 2008 so say no backported features after 2009 no bug fixes after 2012. All announced in 2008 of course. Given 4 years to migrate most would have made the jump sooner.

Dropping support is about as user-hostile as it gets.

Once again, how can you ask/require users to expend precious limited energy to re-program just to stay in place? It's totally obnoxious and completely unnecessary.

Re: Why Is the Migration to Python 3 Taking So Long?

#336
post #335

Earlier quoted context omitted.

Probably the mistake was not dropping support much sooner. Python 3 came out in 2008 so say no backported features after 2009 no bug fixes after 2012. All announced in 2008 of course. Given 4 years to migrate most would have made the jump sooner.

Dropping support is about as user-hostile as it gets. Once again, how can you ask/require users to expend precious limited energy to re-program just to stay in place? It's totally obnoxious and completely unnecessary.

It's absolutely amazing to me that people can pay nothing for something and frame the party providing that software for free opting not to spend even more effort to provide bug fixes for an old version of their software as them effectively taxing them. This is especially true of python 2 which is was supported from the release of python 3 in 2008 until 2020 and further supported by red hat until 2024.

This is exactly backwards of reality. It's as if they were eating at someones home and had turned a cup off coffee into a week long stay during which they rudely complained when the host asked them to please do something about their pile of dishes, trash, laundry, and leavings.

Nobody is after all taking away your version of python 2 or ability to use and maintain it. It takes active effort to keep fixing bugs in software that may be network facing. If you want to do that maintenance you can of course but people it seems aren't going to be doing this for python 2 forever. If you disagree either take up the reigns or pool your funds to pay someone to do it.

The thing to do back in 2008 was to figure out when you wanted to switch and schedule a bit of time to learn python 3. Anyone who did this by oh 2009 or 2010 would have virtually no work to do now. Any work that has been created since based on something you were told 11 years ago was going away is most assuredly work that you have created for yourself and will be obliged to take up.

Anyone who did this in 2014 would have a decade of runway before they can no longer run their python 2 apps on rhel/centos. Anyone who switches TODAY 11 years late to the party can run python + redhat for another 4 years.

>completely unnecessary

It would be more work to do otherwise. Nobody wants to do that work. You don't and they don't.

Re: Why Is the Migration to Python 3 Taking So Long?

#337
post #204

Earlier quoted context omitted.

you don't have to do these things in python 3 either, your problem was that you had python 2 code that was already broken and you are started adding encode/decode to fix it, typically making the problem worse. If you write code in python 3 from the start you rarely need to use encode() and decode(). Typically what you always want is a text not bytes. Exception to it might be places where you want to serialize like IO…

The problem are external APIs returning whatever they want no matter what they should return. The world is messy. Example, I just had to write this return urllib.request .urlopen(url, timeout=60) .read() .decode("utf-8", errors="backslashreplace") (probably not valid code because of the newlines but you'll forgive me) Then I use that string in a regexp, etc. This is the only language where I have to explicitly deal w…

Why is that bad? The result returned from an URL is always binary. In certain situations it could be text but it doesn't have to be. If the result was an image and you would want to convert the data to image, if it was sound file same, you should think of text as another distinguished type.

Of course urllib could have method text() that would do such conversion, but then urllib is not requests. It never was user friendly.

Edit: personally I use aiohttp, the interface is much nicer: https://aiohttp.readthedocs.io/en/stable/client_reference.ht... if I can't use asyncio then would use requests.

Re: Why Is the Migration to Python 3 Taking So Long?

#338

Earlier quoted context omitted.

Python grammar is not LL(1) in general; just look at set and dict literals. This is really no different than ":" after an expression being legal inside a dict literal (and how you know that it is a dict literal). But also, there's no reason to make those legal only inside classes. All it needs to do is make "def foo.bar" produce a different type of function, that has the method descriptor-producing behavior that is c…

> Python grammar is not LL(1) in general; just look at set and dict literals. This is really no different than ":" after an expression being legal inside a dict literal (and how you know that it is a dict literal). Yes it is[0]. LL(1) Grammars can still be recursive, they just can't change the parsing rules based on distant context. > As far as less vs more common case - I think it's more important to optimize for ob…

Why is it an unwanted thing? By the same logic that demands "self" to be explicit, if you don't specify "self", then you explicitly don't want the method to be an instance method - seems very straightforward to me. And it's still invokable via instance member access, so the user of the class is none the wiser. Where's the footgun?

My complaint is of course more complicated than the syntax alone. I'm just saying that a distinctive syntax for self-as-receiver could be used to drive other changes that would make things more intuitive and self-consistent overall. And, of course, any discussion about "elegance" is going to be inherently subjective.

With respect to commonality of various constructs - this is all from personal experience writing Python code (for a project that is hosted on GitHub, by the way). It doesn't require particularly fancy code to trip that wire in general - it just requires code that tries to be generic, i.e. not make more assumptions that it needs to about the types of values that flow through it. Python classes break that genericity by treating functions, and only functions, in a special way whenever they flow through class attributes. This is particularly egregious in a language where every object is potentially callable, and non-function callables are very common; so functions really aren't all that special in general - except for that one case.

With partial() specifically, of course you can avoid that in this manner. But why would you, if it works with regular functions? I prefer it over defs, not just because it's more concise and avoids repeating things, but because it's also clearer - when you see partial(), it immediately tells you that it's a simple alias, nothing else.

But regardless, it's a function that has a certain documented behavior, and common sense would dictate that this behavior works the same for methods as well as functions. That it doesn't is not an intentional limitation of partial() - it's an unfortunate quirk of the design of methods themselves. And if you don't know exactly how functions become methods in Python, you wouldn't have any reason to expect the behavior that it exhibits. That's why the docs for partial() have to spell it out explicitly: "partial objects defined in classes behave like static methods and do not transform into bound methods during instance attribute look-up" - because that's not a reasonable default assumption!

The bigger problem is that every library that offers a generic wrapper callable has to add the same clause to its docs, because they're all affected in the same way. And if they don't document it, and you use, say, a decorator from a library - how do you know whether the fact that it returns a function and not some other callable is part of the contract that you can rely on, or an implementation detail? Conversely, whenever you implement a decorator, you have to be cognizant that changing the type of the return value from/to plain function can be a breaking change for your clients - and that is even less obvious.

Re: Why Is the Migration to Python 3 Taking So Long?

#339

Earlier quoted context omitted.

> Python grammar is not LL(1) in general; just look at set and dict literals. This is really no different than ":" after an expression being legal inside a dict literal (and how you know that it is a dict literal). Yes it is[0]. LL(1) Grammars can still be recursive, they just can't change the parsing rules based on distant context. > As far as less vs more common case - I think it's more important to optimize for ob…

Why is it an unwanted thing? By the same logic that demands "self" to be explicit, if you don't specify "self", then you explicitly don't want the method to be an instance method - seems very straightforward to me. And it's still invokable via instance member access, so the user of the class is none the wiser. Where's the footgun? My complaint is of course more complicated than the syntax alone. I'm just saying that…

> Why is it an unwanted thing?

Let's take the following code snippet as an example:

    class Foo:
      def bar(val):
        foo = val
I claim that most of the time this is a mistake, and the author would have preferred `def bar(self, val): self.foo = val`. In current python, this will raise an exception when called. In your proposed python, this will silently do nothing, possibly leaving the instance in an invalid state. This is a footgun. I admit the example is contrived, but forgetting `self` is a thing I've seen happen, and having it fail loudly is preferable to having it do something likely unintended. Again, if someone wants to do the unusual thing, `@staticmethod` is still around.

> With partial() specifically, of course you can avoid that in this manner. But why would you, if it works with regular functions?

Simply: because I'd prefer it if functions look like functions. Understanding that `def x` is a callable is easier than trying to discern if `x = foo(other_thing)` results in x being callable or not, where it does for some values of `foo`, but not for others. Which isn't to say that python shouldn't make this change, I think I mostly agree with your complaint, I just probably wouldn't take advantage of it.

> My complaint is of course more complicated than the syntax alone.

To be frank, I don't see any connection between your syntactical suggestions and your semantic ones. They seem to be entirely orthogonal.

Re: Why Is the Migration to Python 3 Taking So Long?

#340

Earlier quoted context omitted.

It may in fact be what every class-based OO language actually does operationally. But having the OO language do that for you is, to me, one of the lines that separates them from "a procedural language playing OO".

How would you describe OO languages which only have multimethods, then? Say, Dylan?

Don't know enough to say.
Post reply on HN