Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

One does not simply 'pip install'

ianwootten.co.uk

Post reply on HN
Print view View original

61–70 of 118 posts

Re: One does not simply 'pip install'

#61

Post by __MatrixMan__ »

davidmurdoch wrote: post #24

I'm convinced that there are very few python libraries that Just Work if you follow their installation instructions. I've never found one that didn't come with issues myself. Complain about this to a Python dev and you'll be "Well actually"ied to oblivion and each and every one will have their own opinion-as-fact on the best practice for managing these -- totally unaware how antithetical Python development has become…

Yeah, the well-actually's are a problem. It's not all of us though.

Python dev's know we have a problem, it's just hard to fix because "people developing apps and worrying about dependencies" is a rather small part of the python community. It's not like Java or something where everybody writing the language is a developer. Most are scientists or business people or students working in places like anaconda or Jupyter. So it's really hard to get momentum behind an all-together-now solution.

I've slowly been gravitating toward Nix flakes so I can use it to pin to a project versions of all of the things you can't reliably install with pip alone (like python itself, or numpy, or postgres or whatever) and then have it read deps from poetry (via poetry2nix) for everything that "just works," but that's never gonna fly with the non-developer Python community. Hell, it probably won't even fly with half of the developers either, but it works well for me.

I think my situation is typical of python developers, which is why we have this problem. I think it'll stick around for a while because it's not like "just use a different language" is gonna fly with the non-dev crowd. They're going to expect somebody else to solve these problems for them.

(I may have a bias because my company offers OSS python apps in a SaaS form factor, so our support folk are the ones solving these problems--typically by either handling the virtualenv behind the scenes or by ensuring that users with conflicting dependencies are using different images).

Re: One does not simply 'pip install'

#62

Post by throwaway892238 »

A Makefile makes this trivial:

  # Makefile
  all: venv frozen test
  
  venv:
   python3 -m venv install venv
  
  frozen:
   [ -e frozen.txt ] || { echo "ERROR: run 'make update-frozen'"; exit 1 ; }
   ./venv/bin/pip install -r frozen.txt
  
  update-frozen: clean install-requirements freeze
  
  freeze:
   ./venv/bin/pip freeze > frozen.txt
  
  install-requirements:
   [ -e requirements.txt ] || { echo "ERROR: make a requirements.txt file"; exit 1 ; }
   ./venv/bin/pip install -r requirements.txt
  
  test:
   ./venv/bin/python3 run_tests.py
  
  clean:
   rm -rf venv
Put your package names in requirements.txt and run `make update-frozen`. To reinstall everything from frozen state, `make clean frozen`. (And replace the first space with a tab; HN is stripping my tabs out)

I know Pythonistas like to use Python for everything, but there are other tools out there that will make your life much simpler.

Re: One does not simply 'pip install'

#64

Post by qbasic_forever »

KptMarchewa wrote: post #7
jasonpeacock wrote: post #5

The article talks about installing Python packages for development, but if you find yourself using `pip` to install Python tools/scripts then you should use `pipx` - it will properly sandbox those tools so they don't break (or be broken by) the system or other Pythons: https://pypa.github.io/pipx/

The problem is that everyone has different problems with python packaging and everyone has different idea what you "should" do.

The tool specific usecase with pipx is unique though, it's laser focused and perfect at the job of getting a python tool to users regardless of whatever wacky state their Python install is in. It's kind of separate from the issues of managing dependencies. It's a fantastic tool I wish more python documentation and users would embrace.

Re: One does not simply 'pip install'

#65

Post by wheelerof4te »

qbasic_forever wrote: post #51
wheelerof4te wrote: post #17

Earlier quoted context omitted.

The way it's meant to be. On Linux, you either use the system packages via "apt install", or you use venvs. EDIT: For context, I've meant "managed" distros like Debian and Ubuntu.

Nowhere in the official Python documentation (where 99% of new python users are going to go) does it warn or even talk about Linux and Debian specific issues like only using apt packaged versions of dependencies. It wasn't even until recent years that pip gave a hint or warning something might break in those setups. The situation with Python on Debian has been pretty bad IMHO with a cloistered group of people saying…

"The situation with Python on Debian has been pretty bad IMHO"

The situation with anything has been pretty bad in Debian lately.

I'm all for the minimalistic approach in regards to Python. It's Ok to provide only the packages needed by applications and the core system. For everything else, there's pip.

EDIT: I've meant to say, there's pip inside a venv.

Re: One does not simply 'pip install'

#66

Post by vlunkr »

kkthxbb wrote: post #12

I'm not sure if I get the point of this article. So basically the author has learnt that there are a different ways of managing packages in Python? I'm aware that this might be a problem in Python, but let's be serious guys, you only need to spend 5 mins to learn about venv/conda and you will never face any problem in a basic Python project. You don't have to write an article about that.

As an outsider, the ecosystem looks like a mess. The point of the article is to illustrate some of those issues.

Re: One does not simply 'pip install'

#67

Post by qbasic_forever »

bentaber wrote: post #63

Is there a canonical example of how python projects should manage dependencies and sandboxing such that other developers can just clone, install, and get to work?

Put everything in a docker container/OCI image and have someone own managing and babysitting the build of that image for everyone else.

There really is no single tool or workflow for everything in the python world. What works for a simple source only python package can break horribly if you try using sophisticated scientific computing packages with numerous native dependencies (and then you realize you need conda or a whole other set of tools).

Re: One does not simply 'pip install'

#68

Post by kkielhofner »

KronisLV wrote: post #55
unxdfa wrote: post #21

Earlier quoted context omitted.

This makes me so happy. Back when we had Jenkins slaves, one of our devops guys set a pipeline up that pip installed different versions over the top of system packages causing weird intermittent failures everywhere. Different pipelines would be running in different requirements files. I revoked sudo privs immediately for Jenkins (I didn't add them in the first place) and reprovisioned the whole build cluster resultin…

> Back when we had Jenkins slaves, one of our devops guys set a pipeline up that pip installed different versions over the top of system packages causing weird intermittent failures everywhere. Not everyone might like containers, but using them for CI seems like a good way to avoid situations like this, at least when viable (e.g. web development). You get to choose what container images you need for your build, do wh…

Second containers (however controversial they may otherwise be). The official python Alpine container is < 20mb and the slim Debian variants are < 45mb. For many of my Python projects I end up needing to install various system dependency libs like CUDA, libav, libsndfile, etc. I tend to be a "fan" of containers generally and containers seem like the best way to handle situations like mine.

Re: One does not simply 'pip install'

#69

Post by georgyo »

michaelcampbell wrote: post #36
azornathogron wrote: post #29

Earlier quoted context omitted.

Various packages rely on node_modules existing as a directory with a particular layout, some rely on being able to write into it. Some of the npm alternatives are built to store and manage dependencies in other ways (e.g., keep packages as zip files or other archives and get node to load direct from the zip), and these other mechanisms do not use a node_modules directory, hence compatibility problems.

> these other mechanisms do not use a node_modules directory, Once you've landed on a package manager that does use it, wouldn't you continue to use that?

It doesn't matter if you decide to use a different package manager, if one of your dependencies depend on that layout, you must supply that layout.

Re: One does not simply 'pip install'

#70

Post by 404mm »

This post points out one of my struggles with python.

I am not a python developer but I use python heavily for some tooling. So all I need to do is to “distribute” my tools to other servers in a replicable and consistent matter, isolated from global packages.

Can you please help me understand two points?

1. If I use venv+pip to install some python app, do I have to “activate” that specific virtual environment before executing that tool or can I just simply call it by its path on the file system?

2. Are there any official guide rails for making venv-wrapped app accessible to other users on a server? Or just as simple as placing links to /usr/local/bin/ for example?

Post reply on HN