161–170 of 184 posts
After hitting some weird PyInstaller bugs, I gave up and started compiling Python myself. One interpreter for every project. Shell scripts to set the paths. All libraries go directly in site-packages, not some other layer. A little more complicated at the outset, but this approach has yet to let me down. And compared to the nightmares I was trying to fix, building Python is dead easy.
Earlier quoted context omitted.
Cargo is my ideal, but really anything that doesn't make me manage virtualenvs or take 30 minutes to resolve dependencies. Note that "managing my own virtualenvs" is tricky because you have to make sure everyone has all of the same versions of the same dependencies in their virtualenv across your entire team (including production). I'm sure there are workflows that allow for this (probably with some tradeoffs), but w…
Great experience report: thank you! Wanted to point out that the Pants project has been focusing on widening that happy path recently (...by narrowing its focus to Python-only in the short term), and is ramping up to ship a 2.0. This page covers some of the differences between v1 and v2 of the engine, and particularly its impact on Python: https://pants.readme.io/docs/pants-v1-vs-v2 ... We're using Rust and haven't b…
I’m going to dig into that 2.0 link though!
It's worth noting that on Linux it's slightly different because most of the popular libraries can be installed with the system package manager (no problem of dependency management, updates, ...), I rely on alternative solutions only when I want to use a version of a library different from the one shipped with the package manager (which is not that frequent with fast paced distros like Fedora) or when the library is n…
Earlier quoted context omitted.
Great experience report: thank you! Wanted to point out that the Pants project has been focusing on widening that happy path recently (...by narrowing its focus to Python-only in the short term), and is ramping up to ship a 2.0. This page covers some of the differences between v1 and v2 of the engine, and particularly its impact on Python: https://pants.readme.io/docs/pants-v1-vs-v2 ... We're using Rust and haven't b…
That’s great to hear. Is there any page that documents the architecture of pants? I understand build systems of various kinds quite well, but I can’t tease out the design philosophy behind pants, especially how the different target types / plugins end and the “core” begins. I’m going to dig into that 2.0 link though!
This file contains a few good examples of `@rule`s that collectively partition python targets to generate `setup.py` files for them automatically: https://github.com/pantsbuild/pants/blob/5e4f123a1dbc47313fe...
> Pipenv or poetry? If you used pipenv for a complex project with huge dependency tree, or used it for a long time, you definitely run into a blocker issue with it. That is the worst package manager of all, and probably the reason why Python has such a bad reputation in this area. It's because it's fundamentals are terrible. Just go with Poetry. It's very stable, easy to use, has a superior dependency resolver and wa…
Earlier quoted context omitted.
I agree with you that Docker should not be there, but the reality is that people us it to replace some other tools (like venv). I wonder why you prefer the former approach.
On the contrary, thanks for having included Docker in that list. It's the obvious answer to so many problems (developing, running and deploying apps, replicating deterministic Python environments, not installing linux dependencies required by Python packages directly on your machine, and so on). BTW, to comment one of the point you made in the article, it's not that hard to run CUDA inside a container. It's less stra…
Docker doesn't do anything Python specific on its own. It can be part of a pipeline but only with support from the Python specific tools which is what should be discussed in this kind of article.
Pip feels like an outdated package manager, lacking essential functionality that package managers of other languages have implemented for years. For example, credential redacting in pip was only introduced in 2019, 8 years after its initial release!
Not to mention the global-first nature of pip (package is installed globally unless the user explicitly requests for a local installation). You can still install packages locally, but this only shows that pip was not built with environment reproducibility in mind. As a consequence, the need for additional environment tooling (like venv) arose, which increased the complexity of the local python setup.
Tools wrapped around pip are also under par. I cannot see why Pipenv is that resource intensive, leading to long and noisy builds (my machine gets close to exploding on a pipenv lock), with very fragile lock files. Debugging an unsuccessful locking in the CI of an enterprise project is a mystery that could take an entire week to solve. Its javascript counter-part (npm) does the exact same thing, faster and with less CPU usage.
Trusting the OS community, I understand that there would be very good reasons for Pipenv to perform like this, but as the consumer of a package managing tool all I see is the same generation of file hashes I see on npm, but with npm doing it way more efficiently. I really see value in the principles that Pipenv is promoting, but to me the developer experience of using it is suboptimal.
Earlier quoted context omitted.
Python dependency management of packages using C or C++ behind the scenes is really problematic and sometimes, the installation may fail. In this case, a solution is to use Conda or mini conda which provide many pre-compiled packages and also Clang C++ compiler. An alternative way to allow people without software engineering background to play with Python data science and machine learning tool may be providing pre bu…
Docker is definitely an interesting tool for that, but my biggest problems is that I have to teach them Docker, which is a totally new layer of abstraction they haven't seen before. How do you approach this? How technical are people you prepare Docker images for?
> [Pipenv] loads packages from PyPI so it does not suffer from the same problem as Conda does. False. Conda manages packages installed from PyPI. This is discussed under the Conda section, so I'm surprised the quoted line wound up in the article.
Hey xapata, thanks for pointing this out. Any chance you could give me some reference so I can fix it in the original article?
Basically, use Conda to manage environments, use Pip to install packages. If you're using Conda to install anything, do that first.
People always get up in arms about this, but as someone who has used Python as her daily driver for years it's really... never been this serious of an issue for me? I have used virtualenv/venv and pip to install dependencies for years and years, since I was a teen hacking around with Python. Packaging files with setup.py doesn't really seem that hard. I've published a few packages on pypi for my own personal use and…
The only time I run into problems is when someone else is trying to use Conda. Then it can be hell trying to get their code running in standard pip/venv or vice versa. I'm sure Anacona filled a niche at some point, but we have wheels now, can we all just agree to stop using Conda? What value does it actually bring now that makes it worth screwing up the standard distribution tools?