Python Is Eating the World

Back in the day when I was way more inexperienced than I'm now, I was a die-hard Python fanboy. To me, it seemed like a best option available due to it being way more concise and readable than JS/PHP, not to mention that it was way more powerful and dense. I didn't really bother with type safety at the time and I wasn't exposed to any real criticism of it at the time, as it happens often when you're just too new, and I was more than fine with it.

Time mercilessly marched forward since then. I worked with different codebases, changing tech stacks and languages a bit, learning new stuff and so on. With some experience, lots of self-education and a vast amount of different workflows to compare with, I've realized one important thing. While Python may be better than some languages that are considered trash by industry consensus, it's also a trashcan language to ones with sound and flexible type systems, not to mention that, across dynamically typed contenders, it's nowhere near as powerful as literally any solid Lisp.

That said, I feel bad for the industry that "dives into Python" more and more over the last years. While it may seem like a pretty easy and cool-ish language to write in, you will inevitably have large problems at scale, it just doesn't have what it takes to be easily scalable, and most importantly, it's just tedious to refactor when it's big, even though the language itself is mind-numbingly easy and even your grandma can jump right in.

In a conclusion I'd like to say that you'll be better off without using Python as your project's main driver. Yes, it's simple to write it and the developers supply is abundant. It's easy to start. It'll just be a major pain to scale it and support it later on.

Edit: Made a pun.

I agree, although a lot of it has to do that there's so much misinformation about the web, and many articles recommending bad solutions. This is because python went through many packaging solutions. IMO the setuptools one is the one that's most common and available by default. It has a weakness though, it started with people writing setup.py file and defining all parameters there. Because setup.py is actually a python program it encourages you to write it as a program and that creates issues, setuptools though for a wile had a declarative way to declare packages using setup.cfg file, you should use that and your setup.py should contain nothing more than a call to setup().

> Why do I need to make a "virtual environment" to have separate dependencies, and then source it my shell?

Because chances are that your application A uses different versions than application B. Yes this could be solved by allowing python to keep multiple versions of the same packages, but if virtualenv is bothering you you would like to count on system package manager to keep care of that, and rpm, deb don't offer this functionality by default. So you would once again have to use some kind of virtualenv like environment that's disconnected from the system packages.

> Why do I need to manually add version numbers to a file?

You don't have to, this is one of the things that there's a lot of misinformation about how to package application. You should create setup.py/cfg and declare your immediate dependencies, then you can optionally provide version _ranges_ that are acceptable.

I highly recommend to install pip-tools and use pip-compile to generate requirements.txt, that file then works like a lock file and it is essentially picking the latest versions within restrictions in setup.cfg

> Why isn't there any builtin way to automatically define a lock file (currently, most Python projects just don't even specify indirect dependency versions, many Python developers probably don't even realize this is an issue!!!!!)?

Because Python is old (it's older than Java) it wasn't a thing in the past.

> Why can't I parallelize dependency installation?

Not sure I understand this one. yum, apt-get etc don't parallelize either because it's prone to errors? TBH I never though of this as an issue, because python packages are relatively small and it installs quickly. The longest part was always downloading dependencies, but caching solves that.

> Why isn't there a builtin way to create a redistributable executable with all my dependencies?

Some people are claiming that python has a kitchen sink and that made it more complex, you're claiming it should have even more things built in, I don't see a problem, there are several solutions to package it as an executable. Also it is a difficult problem to solve, because Python also works on almost all platforms including Windows and OS X.

> Why do I need to have fresh copies of my dependencies, even if they are the same versions, in each virtual environment?

You don't you can install your dependencies in system directory and configure virtualenv to see these packages as well, I prefer though to have it completly isolated from the system.

> There is so much chaos, I've seen very few projects that actually have reproducible builds. Most people just cross their fingers and hope dependencies don't change, and they just "deal with" the horrible kludge that is a virtual environment.

Not sure what to say, it works predictable to me and I actually really like virtualenv

> We need official support for a modern package management system, from the Python org itself. Third party solutions don't cut it, because they just end up being incompatible with each other.

setuptools with declarative setup.cfg is IMO very close there.

> Example: if the Python interpreter knew just a little bit about dependencies, it could pull in the correct version from a global cache - no need to reinstall the same module over and over again, just use the shared copy. Imagine how many CPU cycles would be saved. No more need for special wrapper tools like "tox".

There is a global cache already and pip utilizes it even withing an virtualenv. I actually never needed to use tox myself. I think most of your problems is that there are a lot of bad information about how to package a python app. Sadly even the page from PPA belongs there.

I think people should start with this: https://setuptools.readthedocs.io/en/latest/setuptools.html#...

Yes it still has some of the problems you mentioned, but it fixes some others.

The tradeoff is that dynamic languages are faster to develop, more concise, but more expensive in maintenance exactly because of issues like this. The data that I base this opinion on is an unpublished internal report from nearly a decade ago at Google quantifying costs of projects of different size in their different languages. Which was Java, C++, and Python. Python had the lowest initial development, and the highest maintenance costs. That is why Google then moved to Go as a replacement for Python. It was good for the same things that they used Python for, but being statically typed, its maintenance costs were lower.