Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Use `Python -m Pip`

snarky.ca

Post reply on HN
Print view View original

121–130 of 141 posts

Re: Use `Python -m Pip`

#121

Post by xigoi »

dmurray wrote: post #111
figbert wrote: post #38

If I'm understanding correctly, this basically just kicks the ball a little further down the road... You shouldn't use pip directly because you don't know which version is the one in your path. Ok: the same applies to the python command? Calling pip is version ambiguous, but so is calling python.

If there's an executable file named "python" in your current directory, typing "python" in your shell won't in general execute that file. You need to add the current directory to your PATH, or to run it explicitly with something like "./python". This is different from the behavior with "python -m modulename". So this security concern applies when you trust your shell and all the directories in your PATH, but you don'…

Any sane system package manager will ensure that pip corresponds to python, pip2 corresponds to python2, …

Re: Use `Python -m Pip`

#122

Post by gjvc »

jvolkman wrote: post #92
gjvc wrote: post #27

Earlier quoted context omitted.

"pip freeze" generates a versioned list of packages to install in the same format as requirements.txt -- in this example below i've called it versions.txt. I have a bash wrapper "bin/venv-create" which essentially does this python3 -m venv .venv/ if [[ -f versions.txt ]] && [[ versions.txt -nt requirements.txt ]]; then pip install --requirement versions.txt else pip install --requirement requirements.txt pip freeze >…

`pip freeze` doesn't generate hashes, so you can't be sure that the package contents haven't actually changed but maintained the same version string. Also, `pip freeze` doesn't include platform-specific dependencies for other platforms. So if you run the following on linux and again on macos, you'll get different results because `ipython` depends on `appnope` only when running on macos. python -m venv env && env/bin/…

these are legitimate weaknesses -- thank you for highlighting them. On the other hand, if one can accept these weaknesses, keeping separate the root requirements file and the versions file goes a long way to making the process manageable, and not as chaotic as some might have you believe.

The beef I have with so many explanations of pip is that they tell you to source ".venv/bin/activate" and then "just run pip install whatever" without a) separating the root requirements from the effective/complete requirements, and b) they don't suggest using a wrapper for the ".venv/bin/python3" binary so that execution is the same in all environments.

Re: Use `Python -m Pip`

#123

Post by jwilk »

romanows wrote: post #73
jjnoakes wrote: post #63

Earlier quoted context omitted.

Do you mean there might be a 'pip' command in the local directory that might get called by accident, or do you mean the real python pip command might load a python module from the local directory by accident?

The second thing. Python includes the current working directory first in the module search path. It leads to the above issue and occasionally some a tricky debugging problem.

If you run a script, such as "pip", sys.path[0] is the script's directory, not cwd:

https://docs.python.org/3/using/cmdline.html

> If the script name refers directly to a Python file, the directory containing that file is added to the start of sys.path

Re: Use `Python -m Pip`

#124

Post by Liquid_Fire »

josefx wrote: post #100
sidpatil wrote: post #11

Earlier quoted context omitted.

Contrast with https://xkcd.com/353/ .

They broke that with python 3. print "Hello World!" no longer works and import antigravity now only links to the comic.

> import antigravity now only links to the comic

"now"? Hasn't it always?

Re: Use `Python -m Pip`

#125

Post by mickeyk »

rat87 wrote: post #76
eslaught wrote: post #24

Earlier quoted context omitted.

To be honest, I don't see how you can give this advice. A couple of the most popular package management systems for Python are very obviously deficient (and sigh, these are usually the ones I get stuck working with, due to outside constraints). A classic example is version pinning. Rust has Cargo.lock, Ruby has Gemfile.lock. Python? It depends on which one of the multitude of options you pick. But at least a couple o…

python's poetry has poetry.lock I think it's slowly becoming semi-standard. But still slowly and still not fully ready(bugs). Also its odd that pyproject.toml (not poetry.toml which is the poetry settings for repo) is the dependency definition file and poetry.lock is the lock file for pip file I usually make a requirements-to-freeze.txt, make a fresh virtualenv, install requirements-to-freeze.txt and then pip freeze…

The pyproject.toml is telling you what should work in terms of a range of package versions and the poetry.lock is telling you what my environment is in terms of exactly what is installed. You don't need the lock file, but without it there is no guarantee of what it will install so it may not be completely compatible.

Re: Use `Python -m Pip`

#126

Post by mickeyk »

wombatpm wrote: post #22
jmcgough wrote: post #19

Earlier quoted context omitted.

As someone two months into their first python job - the ecosystem is solid for ML / data science, but there's a lot of places where it's painfully lacking. I hate the most popular ORM (sqlalchemy) and alembic has a lot of footguns: for instance, if you autogenerate a migration where you change a table name, it will try to drop the old table and create a new one. In web dev a lot of the OSS community has moved on to m…

Much prefer the Django ORM, which is a pain to use if you are not building web apps

I've used peewee in the past which worked well.

Re: Use `Python -m Pip`

#127

Post by bloblaw »

behnamoh wrote: post #57
elbigbad wrote: post #53

Earlier quoted context omitted.

Yeah, I agree with this. I typically dockerize where appropriate to sidestep this a little, but very often rewrite everything into a more friendly language like Go if I am intending to ship software to other machines. The latter is obviously painful if I’m making heavy use of Python specific libraries that do a lot of heavy lifting like Numpy, if those features aren’t in something like GoNum.

I should also say: Unfortunately, Conda’s environment exports only work for similar OS’s. I don’t know why Python’s situation is like this, after so many years. Does docker fix all the above issues?

Docker simultaneously solves and creates problems here.

Step 1: Docker must be installed on all targets. This is not a given and is a new piece of overhead.

Step 2: The Dockerfile hopefully doesn't source from just "ubuntu:latest" and bring in the whole kitchen sink for this SINGLE APPLICATION.

Effectively if you are using Docker and deploying your script to Windows or macOS, you are saying "Hey, this script requires you to install linux in a VM to run this. Docker makes it easy. Go download 500mb of Docker and a couple hundred more mb of images to run this 800k script.

Re: Use `Python -m Pip`

#128

Post by bloblaw »

okasaki wrote: post #106
bloblaw wrote: post #46

Earlier quoted context omitted.

The fact that the suggested solution in Python is to give every Python script a full copy of an entire specific Python runtime (via venv) is a mild annoyance as a design pattern... to me . Python scripting today requires shipping your development environment. Python is wonderful until you want to run that code on another machine. At that point, the target system has to venv their way into reproducing your environment…

Virtualenvs are no worse than static linking, or packaging dlls/sos, all very common practice (and even recommended). Python package management is VERY easy.

When I statically compile a C++ or Go binary, I end up with something I can copy and execute on any platform for which it was compiled for.

Single file. Copy and run.

Virtualenvs require every single target to reproduce your dev environment:

(1) have internet access and able to reach pypi (or artifactory, or whatever you use).

(2) the ability to install the required version of Python if it isn't already installed. That's another 30mb download.

Ever want to run a complex Python script on a bastion host or a host behind a bastion? Well now #1 and #2 above won't work (or haven't in my direct experience working for some of the big cloud companies). So you have to use something like PyInstaller and hope it works. It might. It might not. A statically compiled C++ binary or Go binary probably will.

This thread is full of pain points in python package management. The first step is admitting there is a problem. I don't think you've experienced the pain caused by Python package managers that others outline in this thread.

Re: Use `Python -m Pip`

#129

Post by eslaught »

rootusrootus wrote: post #80
eslaught wrote: post #24

Earlier quoted context omitted.

To be honest, I don't see how you can give this advice. A couple of the most popular package management systems for Python are very obviously deficient (and sigh, these are usually the ones I get stuck working with, due to outside constraints). A classic example is version pinning. Rust has Cargo.lock, Ruby has Gemfile.lock. Python? It depends on which one of the multitude of options you pick. But at least a couple o…

I must live a sheltered life. Everything I've run into so far uses requirements.txt to pin dependencies. Well supported and easy is probably why.

Yes, exactly. And how many of these projects go to any effort to pin their transitive dependencies?

In my experience, this is basically no one; in fact I've seen many (most?) projects not even pin the versions on their top-level dependencies. (Ask yourself how many times you've seen just "numpy" as a dependency without any version bound. Far too often in my experience.)

And this is exactly why stuff breaks: even when the root dependencies are pinned (and are they?), a transitive dependency could get upgraded and break something (or fail to build entirely).

Re: Use `Python -m Pip`

#130

Post by eslaught »

gjvc wrote: post #27
eslaught wrote: post #24

Earlier quoted context omitted.

To be honest, I don't see how you can give this advice. A couple of the most popular package management systems for Python are very obviously deficient (and sigh, these are usually the ones I get stuck working with, due to outside constraints). A classic example is version pinning. Rust has Cargo.lock, Ruby has Gemfile.lock. Python? It depends on which one of the multitude of options you pick. But at least a couple o…

"pip freeze" generates a versioned list of packages to install in the same format as requirements.txt -- in this example below i've called it versions.txt. I have a bash wrapper "bin/venv-create" which essentially does this python3 -m venv .venv/ if [[ -f versions.txt ]] && [[ versions.txt -nt requirements.txt ]]; then pip install --requirement versions.txt else pip install --requirement requirements.txt pip freeze >…

Thanks, I appreciate this.

I don't think this practice is widespread, which is exactly why I made a point about human engineering in my original post. But I do appreciate that solutions like this exist, and I should look into driving more of this sort of thing in my projects.

Post reply on HN