Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Use `Python -m Pip`

snarky.ca

Post reply on HN
Print view View original

61–70 of 141 posts

Re: Use `Python -m Pip`

#61

Post by jjoonathan »

xdfgh1112 wrote: post #37
bloblaw wrote: post #18

Earlier quoted context omitted.

I remember when Perl was more popular generally than Python (late 90's and early 00's), and Perl espoused the mantra of (There is more than one way to do it: TIMTOWTDI). As a tongue-in-cheek reaction, Python espoused "TOOWTDI", or There's Only One Way To Do It :) reference: https://wiki.python.org/moin/TOOWTDI The problem is, when it comes to the Python package management ecosystem, there are SO MANY ways to do it. A…

I'm relatively new to python. I use venv, pip and requirements.txt. It's dead simple. What am I missing?

Conda. Back in the day, before docker, it was a poor-man's docker combined with a repository of significantly newer and more compatible software than you could get from distro repositories. Python libraries with external dependencies used it heavily for this reason. Conda tended to Just Work in a bunch of cases where pip effectively forced you to go off and debug a bunch of autotools builds on your own. Yuck!

However, it was pushy. It would put its own header/linker paths in front of the system paths (this is how it made "environments," its wannabe containers), which tended to create inadvertent cross dependencies if you didn't understand or didn't remember that the semantics of a conda environment extended beyond python. These dependencies could get baked into binaries and break far down the road, or they could get sucked in as a transitive dependency and trip over the shoelaces of a different build of the same software installed outside conda. Unfortunate. However, around 2019, the problems started growing beyond mere foot-guns. Conda uses a full SAT solver to provide a highly featured versioning system, and this worked great until the big conda channels grew to the point that it started getting really slow. Installing packages went from taking seconds to minutes to hours to forever. They tried caching, they tried fragmenting channels, but it was all very not-seamless.

Eventually, people started migrating back to pip. It turns out that over the last decade distro repositories had gotten their shit together and now Docker existed to sweep up the last few use cases, so nobody needed conda's "poor-man's docker plus curated 3rd party repos" anymore. Now pip is the tool that Just Works, and it Just Works without any of conda's baggage. Virtualenv environments don't hook your system quite as aggressively, pip never stalls when resolving its version plans, and Docker can be used to reproducibly experiment and find the happy path.

Be glad that you missed out on pre-conda pip and the conda arc.

Re: Use `Python -m Pip`

#62

Post by ericholscher »

jwilk wrote: post #17

Beware that "python -m" is insecure in untrusted cwd: https://bugs.python.org/issue33053 E.g.: $ echo 'import os; os.execvp("cowsay", ["-", "pwned"])' > pip.py $ python -m pip --version _______ ------- \ ^__^ \ (oo)\_______ (__)\ )\/\ ||----w | || ||

Isn’t that true of command execution as well? I guess you could use a full path to the pip executable, but similar issue.

Re: Use `Python -m Pip`

#63

Post by jjnoakes »

ericholscher wrote: post #62
jwilk wrote: post #17

Beware that "python -m" is insecure in untrusted cwd: https://bugs.python.org/issue33053 E.g.: $ echo 'import os; os.execvp("cowsay", ["-", "pwned"])' > pip.py $ python -m pip --version _______ ------- \ ^__^ \ (oo)\_______ (__)\ )\/\ ||----w | || ||

Isn’t that true of command execution as well? I guess you could use a full path to the pip executable, but similar issue.

Do you mean there might be a 'pip' command in the local directory that might get called by accident, or do you mean the real python pip command might load a python module from the local directory by accident?

Re: Use `Python -m Pip`

#64

Post by singron »

ericholscher wrote: post #62
jwilk wrote: post #17

Beware that "python -m" is insecure in untrusted cwd: https://bugs.python.org/issue33053 E.g.: $ echo 'import os; os.execvp("cowsay", ["-", "pwned"])' > pip.py $ python -m pip --version _______ ------- \ ^__^ \ (oo)\_______ (__)\ )\/\ ||----w | || ||

Isn’t that true of command execution as well? I guess you could use a full path to the pip executable, but similar issue.

Only if PATH includes ".", which is why they say not to do that (or use windows. I'm not sure if you can fix that there).

Re: Use `Python -m Pip`

#65

Post by remram »

figbert wrote: post #38

If I'm understanding correctly, this basically just kicks the ball a little further down the road... You shouldn't use pip directly because you don't know which version is the one in your path. Ok: the same applies to the python command? Calling pip is version ambiguous, but so is calling python.

Yeah if your Linux distro comes with python, python2, python3.7, and python3.8... then you almost certainly have the matching pip, pip2, pip3.7, and pip3.8. If you activate a virtualenv, that will override Python and python3 but also pip and pip3.

The only situations where I've encountered breakage is pydoc (because your virtualenv does not necessarily have its own pydoc, contrary to having pip) and calling pip from a Jupyter notebook: the current kernel's virtualenv is not necessarily activated (the solution is the %pip magic or `!{sys.executable} -m pip` since `!python` would have the same issue).

Re: Use `Python -m Pip`

#66

Post by [deleted] »

behnamoh wrote: post #57
elbigbad wrote: post #53

Earlier quoted context omitted.

Yeah, I agree with this. I typically dockerize where appropriate to sidestep this a little, but very often rewrite everything into a more friendly language like Go if I am intending to ship software to other machines. The latter is obviously painful if I’m making heavy use of Python specific libraries that do a lot of heavy lifting like Numpy, if those features aren’t in something like GoNum.

I should also say: Unfortunately, Conda’s environment exports only work for similar OS’s. I don’t know why Python’s situation is like this, after so many years. Does docker fix all the above issues?

[deleted]

Re: Use `Python -m Pip`

#67

Post by colejohnson66 »

singron wrote: post #64
ericholscher wrote: post #62

Earlier quoted context omitted.

Isn’t that true of command execution as well? I guess you could use a full path to the pip executable, but similar issue.

Only if PATH includes ".", which is why they say not to do that (or use windows. I'm not sure if you can fix that there).

CMD includes the CWD in the PATH,[a] but PowerShell doesn’t; you’re forced to do the `./` (or `.\`) dance.

However, if you tab complete a file name that’s not "dot slash" prefixed, it’ll add it for you. Not sure if sh and friends do that.

[a]: Implicitly. Echoing the $PATH variable won’t include `.`, but you can execute as if it was

Re: Use `Python -m Pip`

#68

Post by Izkata »

maest wrote: post #32
bxparks wrote: post #29

What is the best way for an author of a Python package to develop and test on multiple versions of Python (e.g. 3.6, 3.7, 3.8, 3.9, 3.10), and be able to switch easily among the various Python versions? Every time I try to research this, I get lost in the chaos of Python packaging and environments. Currently I do a `pip3 install -e .`, which uses the default Python provided by the OS. Then I hope that my continuous i…

I think tox was designed to solve that problem. https://pypi.org/project/tox/

This, absolutely. If you're running tests for building a package, here's an entire "tox.ini" file for running all your tests on multiple python versions:

  [tox]
  envlist = py27,py36,py38

  [testenv]
  deps =
     pytest                 # Install extra packages for testing separate from what's in setup.py or whatever
  commands =
     pytest                 # Whatever test command your project uses
That's really it for bare minimum. If you have pytest configuration, they can also go into tox.ini under a [pytest] section to keep them in the same place.

Instead of installing tox globally you can even install it to a separate virtualenv, activate it, then run "tox". It figures out the right thing to do.

If you're not building a package:

  [tox]
  envlist = py27,py36,py38
  skipsdist = True          # Don't attempt to build a package / pull from setup.py

  [testenv]
  deps =
     -r requirements.txt    # Pull in project requirements
     pytest
  commands =
     pytest

Re: Use `Python -m Pip`

#69

Post by woodruffw »

newuser4321 wrote: post #15
gammalost wrote: post #8

Python is a fun language but the ecosystem around it is horrible. It's a shame. I just want to pip install like I would a package manager

I think you mean the package management rather than the ecosystem. For all its failings, the python ecosystem (for machine learning / data science in particular) is unmatched, which is why python is so popular. As a "professional" python user (who got there in a roundabout way), I'd say the biggest problem with package management is all the conflicting advice and different ways to accomplish the same thing (there are…

The conflicting advice is a serious problem.

I hope you'll forgive me for adding one additional piece of advice: for many Python packages, the only packaging metadata you need is `pyproject.toml`. You don't even need `setup.py` anymore, so long as you're using a build backend that supports editable installs with `pyproject.toml`.

Here's an example of a Python package that does everything in `pyproject.toml`[1]. You should be able to copy that into any of your projects, edit it to match your metadata, and everything will work exactly as if you have a `setup.cfg` or `setup.py`.

[1]: https://github.com/sigstore/sigstore-python

Re: Use `Python -m Pip`

#70

Post by cgriswald »

jwilk wrote: post #17

Beware that "python -m" is insecure in untrusted cwd: https://bugs.python.org/issue33053 E.g.: $ echo 'import os; os.execvp("cowsay", ["-", "pwned"])' > pip.py $ python -m pip --version _______ ------- \ ^__^ \ (oo)\_______ (__)\ )\/\ ||----w | || ||

You can use -I to have the interpreter not include the directory in sys.path. (This also eschews site-packages and any environment variables.)

$ echo 'import os; os.execvp("cowsay", ["-", "pwned"])' > pip.py

$ python -Im pip --version

pip 22.0.2 ...

Post reply on HN