Earlier quoted context omitted.
How do you create the requirements.txt file - by hand, or by pip freeze? If by hand, how do you make sure to lock the versions of your transitive dependencies? If by pip freeze, how do you keep track of what you actually depend on as opposed to what your dependencies depend on? How do you update your dependencies? Do you modify requirements.txt directly? If so, how do you keep it in sync with setup.py? How do you fin…
Pip freeze is dead simple and I upgrade packages as needed. I've rarely manually updated requirements.txt. I literally make a venv, pip install what I need and I am off and running. My editor/ide automatically recognizes the environment. I can't imagine how much simpler it could be. I've been doing it for years without any problems. Dependcies, versioning etc are incredibly simple. A couple commands and I am up and r…
In my case, it doesn't. I'm maintaining libraries and also applications that use those libraries.
For libraries, you need your setup.py to be kept up to date, as requirements.txt doesn't do anything when you pip install a library package. Of course, requirements.txt is necessary when you want to run the libraries's tests, since few things are as frustrating as having the tests broken by some random dependency having a new version come out. But, then you have to keep setup.py and requirements.txt kinda synced - only kinda, because in setup.py you will list your test and dev dependencies separately, but in requirements.txt they all get mixed up together. In theory its possible to script keeping requirements.txt up to date - in practice, when working with a big team, its a tremendous pain. The first option is to tell everyone not to mess it up, but, that doesn't work. The 2nd option is to develop a bunch of scripts to do it, but, then you have to get everyone to install and use them, and thats quite frustrating since its not at all clear to my why the standard tools don't do it already.
We also have applications. And those applications depend on some libraries we wrote. Those libraries have their own dependencies. When a library is updated, it might gain or lose dependencies. So, when an application is updated to use a new version of that library, its requirements.txt should be updated accordingly. pip installing the new version of the library will bring in its new dependencies - but won't get rid of the old ones from the virtual environment. A subsequent pip freeze will freeze a bunch of dependencies that aren't needed anymore - and that get harder and harder to find an eliminate as the number of unused dependencies grows. Again, this could be scripted, but, its a pain to do, and, the tools should support it.
Whats so frustrating, is that these aren't unsolved problems in computer science. There are solutions. NPM/yarn does an OK job (I have minimal experience). Rust's cargo is fantastic. I've heard that Ruby's bundler is great. I fully appreciate that solving these problems probably requires volunteers - and I'm not volunteering, so, maybe there is only so much I can do to complain. But, looking at most of the work going into the ecosystem, it seems to be ignoring these problems.
Anyway, maybe I was a bit too harsh in my initial response. If your workflow is working for you, that is great. What I would suggest, however, is that if you find a need to add additional requirements to that workflow (such as an easy way to update a single dependency, and its transitive dependencies - both adding and removing them), you'll find that the available Python option quickly disappoint you. For your sake, I hope that doesn't happen, since, its unpleasant to deal with.