Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Python's New Package Landscape

andrewsforge.com

Post reply on HN
Print view View original

161–170 of 174 posts

Re: Python's New Package Landscape

#161

Post by mlthoughts2018 »

goerz wrote: post #159
mlthoughts2018 wrote: post #158

Earlier quoted context omitted.

This is deeply untrue for conda. Even very complex environments build in less than a minute. I can believe there are corner cases where conda is very slow, but claiming conda takes 5 minutes for trivial environments is flat out wrong. Perhaps it is issues with a firewall, VPN connection or something else, absolutely no chance that is from normally executing conda.

Well, I experience it on a daily basis, and I'm not the only one! It's officially an open issue: https://github.com/conda/conda/issues/7239

The slowest operation in the linked thread is still taking less than 2 minutes...

Edit: correction, there are two examples that take longer, one at 3.5 minutes, one around 8 minutes. I don’t think it changes any takeaways though.

Re: Python's New Package Landscape

#162

Post by jonafato »

Serow225 wrote: post #150
jonafato wrote: post #130

Earlier quoted context omitted.

The PyPA team has done a lot over the past five years. The changelog for pip ( https://pip.pypa.io/en/stable/news/ ) contains quite a bit, PyPI was migrated to Warehouse, and there have been several PEPs focused on improving the packaging situation. A lot of these ideas come from various people in the community and get formalized as official recommendations or tools, but these things take time, especially accounting…

Understood, I guess I'm wondering why it hasn't been possible to cull more of the less-successful attempts, or at least make it obvious to newer users what is legacy. As an outsider/newer person to Python, the number of package mgmt options to consider is vast and confusing, it would be helpful if there was one (or a few) more "blessed" solutions :)

There are some "blessed" recommendations at https://packaging.python.org/guides/tool-recommendations/ (the Python Packaging Authority is about as official as you're going to get), but this boils down to it being a large open source community. No one's going to cull other people's efforts, but tools do merge on occasion (e.g. the functionality of https://github.com/erikrose/peep has been merged into pip, so peep is deprecated now).

Re: Python's New Package Landscape

#163

Post by EdwardDiego »

toyg wrote: post #151
qwerty456127 wrote: post #140

Earlier quoted context omitted.

The only problem I know with requirements.txt is that many people would require particular versions there while later versions work perfectly fine. Every time I clone someone's Python project to work with I have to manually replace all the =s with >=s to avoid downloading obsolete versions of the dependencies and have never encountered a problem. Anyway, for me the most annoying thing about the Python projects archit…

> have never encountered a problem. Oh, so you weren't around when Requests went 2.0 backward-incompatible (because they changed .json() with .json, or the other way around, can't remember) and half of PyPI, with its happy-go-lucky ">=1.0", broke...? Since then, most people have learnt that you pin first and ask questions later.

I just encountered the fun fact that Pip 18.1 broke Pipenv whereas Pip 18.0 worked just fine.

Re: Python's New Package Landscape

#164

Post by johnlinvc »

erikb wrote: post #149

If you only think in Python then packaging really might be a painpoint. But honestly if we look at other languages it's not so bad actually, very specifically calling out Golang here because it claimed exactly this topic as an initial design goal and to this day has basically failed at delivering it. There are even languages like C++ where the community as a (w)hole has given up on that topic and instead opts for com…

Ruby's bundler is pretty neat. It's one of the first systems that introduced lock files.

Re: Python's New Package Landscape

#165

Post by _fbpt »

Pipenv is unusable for me, since launching your app only works when your current working directory is the Pipfile directory. If you want to launch an app via a shell script from another directory, you have to first cd to the Pipfile dir, pipenv shell (maybe you can pass in a second shell script as an argument).

The article mentions Pipsi is designed to make command-line apps globally accessible, and I'll try it out.

Additionally, adding git/src/package/module.py may be fine when you're using an IDE, but when browsing in a file manager, you must navigate 3 directories deep to even see any source files, which seems to be trending towards the inconvenience amd pain of Java projects.

Re: Python's New Package Landscape

#166

Post by goerz »

mlthoughts2018 wrote: post #161
goerz wrote: post #159

Earlier quoted context omitted.

Well, I experience it on a daily basis, and I'm not the only one! It's officially an open issue: https://github.com/conda/conda/issues/7239

The slowest operation in the linked thread is still taking less than 2 minutes... Edit: correction, there are two examples that take longer, one at 3.5 minutes, one around 8 minutes. I don’t think it changes any takeaways though.

The last one in that thread is 11 minutes. And, I just did `time conda create -n test -y anaconda pytest-cov pytest-xdist coverage sphinx_rtd_theme flake8` on my Macbook, and it clocked in at 36 minutes! Probably, this is affected by also having the condaforge channel active (which I need), but it's definitely not from some network/VPN issues. Everyone I work with has similar problems. So don't tell me there's "absolutely no chance that is from normally executing conda".

Re: Python's New Package Landscape

#167

Post by lmm »

sambe wrote: post #108
lmm wrote: post #88

Earlier quoted context omitted.

> I assume that someone working on the project would do: pip install -e . in a virtual environment. I thought this was quite well-established. Is there a problem with it that I'm not aware of? So ignoring your requirements.txt, and potentially working with different versions of dependencies from the ones you were working with and encountering different bugs? (Also managing your virtual environments "by hand" is tedio…

People working on your project have the choice of using the requirements.txt or not. I would think core developers use the loose dependencies, with the aim of testing the latest and fixing the bugs. Someone has to move dependencies forward at some point, and doing this locally for knowledgeable people seems reasonable. CI should definitely - and part time contributors should probably - just use the pinned dependencie…

> I would think core developers use the loose dependencies, with the aim of testing the latest and fixing the bugs. Someone has to move dependencies forward at some point, and doing this locally for knowledgeable people seems reasonable.

Agreed that developers should be moving the dependencies forward, but you want to do that as a deliberate action rather than by accident. E.g. if you want to consult another developer about a bug you're experiencing, you want them to be on the same versions of dependencies as you.

> This is why I would not worry about pip freeze being non-reproducible. It is a manual step: upgrade our dependencies. Testing should happen all the time.

It's a manual step, but you still want to be able to reproduce it. E.g. if a project is in maintenance mode, you want to be able to do an upgrade of one specific dependency without having to move onto new versions of everything else.

I don't work in Python any more so I don't know what the new tools do or don't do, I was just starting from your "I don't know why I would not have loose dependencies in setup.py and concrete, pinned dependencies in requirements.txt." and I know that workflow gave me a number of problems that I simply don't have when working in other languages. So I'm hoping that Python has caught up with the things that are known-working elsewhere, but maybe not.

Re: Python's New Package Landscape

#168

Post by sirn »

andersriutta wrote: post #153
sirn wrote: post #65

Earlier quoted context omitted.

I have been revisiting buildout recently and I wish there's something that merge ease of use of pipenv with buildout concept. Perhaps something similar to Nix throw in the mix, but more specific to a Python project. I heard that I can do this with Conda, but I never tried. Being able to define and install external dependencies (e.g. ImageMagick, libsodium, etc.) from a configuration file local to a project is somethi…

> Perhaps something similar to Nix throw in the mix, but more specific to a Python project. Any examples of how Nix itself doesn't do what you need? One example I can think of: Nix doesn't support Windows. https://nixos.org/nixpkgs/manual/#python

Nix does everything I want, but I find it hard to convince friends and coworkers to try out Nix. I think this is partly due to Nix itself not belonging to the Python's ecosystem, so the barrier is higher than say, "Yeah, Pipenv is just Virtualenv+Pip"

Re: Python's New Package Landscape

#169

Post by mlthoughts2018 »

goerz wrote: post #166
mlthoughts2018 wrote: post #161

Earlier quoted context omitted.

The slowest operation in the linked thread is still taking less than 2 minutes... Edit: correction, there are two examples that take longer, one at 3.5 minutes, one around 8 minutes. I don’t think it changes any takeaways though.

The last one in that thread is 11 minutes. And, I just did `time conda create -n test -y anaconda pytest-cov pytest-xdist coverage sphinx_rtd_theme flake8` on my Macbook, and it clocked in at 36 minutes! Probably, this is affected by also having the condaforge channel active (which I need), but it's definitely not from some network/VPN issues. Everyone I work with has similar problems. So don't tell me there's "absol…

The same command for me, also running on a Macbook, took just over 5 minutes, using conda 4.5.9. I also repeated inside a Ubuntu docker container with only miniconda installed and got the same.

It’s funny to me that you would expect operations involving “conda install anaconda” to be fast though. 11 minutes seems perfectly fast for that and would be comparable to pip or anything else if doing that huge of a set of package installs.

That is not at all a “trivial environment” like you said previously.

Re: Python's New Package Landscape

#170

Post by mixmastamyk »

jefurii wrote: post #73
patagonia wrote: post #60

Imagine I’m just getting started with Python, and I see this article. I think to myself, “Awesome, a primer!” Then I start reading (these comments)... mayyybe I should try Julia... or anything else, at least while I’m still getting started.

Just use pip and virtualenv. I've been using them on a cluster of related projects for years and they've been rock-solid.

And you can avoid virtualenvs as well until you need to ship a large app.

"pip install --user pkg" is good enough for beginners and writing libraries with few deps.

Post reply on HN