Freezing Python’s Dependency Hell

Mostly I've used plain `python -m venv venv` and it always worked well. A downside - you need to add a few bash scripts to automate typical workflow for your teammates.

Pipenv sounds great but there are some pitfalls as well. I've been going through this post recently and got a bit upset about Pipenv: https://chriswarrick.com/blog/2018/07/17/pipenv-promises-a-l...

Another point is that it does not work well with PyCharm and does not allow to put all dependencies into the project folder as I used to do with venv. (just like to keep everything in one folder to clean up it easily)

Are there any better practices to make life easier?

  $ pipenv install numpy scipy pandas matplotlib requests
  ....
  ....installs everything
  ....
  $ time pipenv sync
  Installing dependencies from Pipfile.lock (3f6ae1)…
        15/15 — 00:00:05
  All dependencies are now up-to-date!

  real	0m7.219s
  user	0m15.645s
  sys	0m1.406s

npm actually got this right, init helps, and it makes sense to traverse up directories to find a package.json.

You create a wrapper script around your application that calls a dev environment set-up script, that [if it wasn't done yet] sets up the environment from scratch for that project or application, and loads it before running your application. This does a couple things.

First, it removes the need to train anyone on using your best practices. The process is already enshrined in a version-controlled executable that anyone can run. You don't even need to 'install lore' or 'install pipenv' - you just run your app. If you need to add documentation, you add comments to the script.

Second, there's no need for anyone to set up an environment - the script does it for you. Either set up your scripts to go through all the hoops to set up a local environment with all dependencies, or track all your development in a Docker image or Dockerfile. The environment's state is tracked by committing both the process scripts and a file with pinned versions of dependencies (as well as the unpinned versions of the requirements so you can occasionally get just the latest dependencies).

Third, the pre-rolled dev environment and executable makes your CI-CD processes seamless. You don't need to "set up" a CI-CD environment to run your app. Just check out the code and run the application script. This also ensures your dev environment setup scripts are always working, because if they aren't, your CI-CD builds fail. Since you version controlled the process, your builds are now more reproducible.

All this can be language-agnostic and platform-agnostic. You can use a tool like Pipenv to save some steps, but you do not need to. A bash script that calls virtualenv and pip, and a file with frozen requires, does 99% of what most people need. You can also use pyenv to track and use the same python version.