For those new or unfamiliar with python, I think the best solution is the simplest: pip and virtualenv
I would add pip-tools to that so the requirements update process can be automated.
My Python Development Environment, 2020 Edition
171–180 of 241 posts
Re: My Python Development Environment, 2020 Edition
#172Does anyone else think this reflects badly on Python? The fact that the author has to use a bunch of different tools to manage Python versions/projects is intimidating. I don't say this out of negativity for the sake of negativity. Earlier today, I was trying to resurrect an old Python project that was using pipenv. "pipenv install" gave me an error about accepting 1 argument, but 3 were provided. Then I switched to…
This is why I manage every nontrivial project I do nowadays with Nix (Haskell, Go, Python, C & C++, bash ..anything) Everything is pinned to exact source revisions. You can be relatively sure to be able to git clone and nix-shell and be off to the races. You can even go the extra mile and provide working editor integration in your nix-shell (especially easy with emacs). So you can enable anyone to git clone, nix-shel…
> Everything is pinned to exact source revisions.
While you're here, how do you do this with nix-pkgs?
I looked into using nixos for nodejs deployments recently, and was amazed to find that the versions of node in the nix-pkgs repo are just pinned to X.Y.0 releases, with no discernable way to update to a bugfix release after .0 , so... I don't see how this could possibly be used for production deployments?
https://nixos.org/nixos/packages.html?channel=nixpkgs-unstab...
What I really want is to be able to tell nix to give me nodejs at version 10.16.3 and have it do the right thing every time.
I'm happy to be wrong on any of this.
Re: My Python Development Environment, 2020 Edition
#173python -m venv venv source venv/bin/activate pip install -U pip pip install whatever # deactivate no need any third-party tools, venv is built-in the above steps always worked for me out of the box.
Do you not consider pip third-party? It does not come with the standard Python distribution. Although `ensurepip` does.
Re: My Python Development Environment, 2020 Edition
#174Earlier quoted context omitted.
This is why I manage every nontrivial project I do nowadays with Nix (Haskell, Go, Python, C & C++, bash ..anything) Everything is pinned to exact source revisions. You can be relatively sure to be able to git clone and nix-shell and be off to the races. You can even go the extra mile and provide working editor integration in your nix-shell (especially easy with emacs). So you can enable anyone to git clone, nix-shel…
> Everything is pinned to exact source revisions. While you're here, how do you do this with nix-pkgs? I looked into using nixos for nodejs deployments recently, and was amazed to find that the versions of node in the nix-pkgs repo are just pinned to X.Y.0 releases, with no discernable way to update to a bugfix release after .0 , so... I don't see how this could possibly be used for production deployments? https://ni…
I think its a coincidence. Check nodejs commit history: https://github.com/NixOS/nixpkgs/commits/f3282c8d1e0ce6ba5d9...
there are minor versions also
Re: My Python Development Environment, 2020 Edition
#175Earlier quoted context omitted.
Granted, I'm just a scientific programmer, but my workplace has a full blown software team maintaining a multi million line codebase. That codebase is rebuilt every night, and as I understand it, you're not allowed to submit a change that breaks anything. And they have people whose job is to keep their tools working. What people casually think of as "Python" is really a huge dynamic ecosystem of packages. Imagine tha…
You should try Dockerizing your project. Then other people just have to type docker run and it’ll work everywhere.
Well in sceintific projects you are working in HPC platforms which docker doesn't exists and there are many reasons for it (it is not just technical).
Re: My Python Development Environment, 2020 Edition
#176Earlier quoted context omitted.
> Everything is pinned to exact source revisions. While you're here, how do you do this with nix-pkgs? I looked into using nixos for nodejs deployments recently, and was amazed to find that the versions of node in the nix-pkgs repo are just pinned to X.Y.0 releases, with no discernable way to update to a bugfix release after .0 , so... I don't see how this could possibly be used for production deployments? https://ni…
I think its a coincidence. Check nodejs commit history: https://github.com/NixOS/nixpkgs/commits/f3282c8d1e0ce6ba5d9... there are minor versions also
Right, so if I'm understanding correctly, I'd need to pin the whole nix-pkgs system to the particular git revision which contains the version I need?
Re: My Python Development Environment, 2020 Edition
#177Does anyone else think this reflects badly on Python? The fact that the author has to use a bunch of different tools to manage Python versions/projects is intimidating. I don't say this out of negativity for the sake of negativity. Earlier today, I was trying to resurrect an old Python project that was using pipenv. "pipenv install" gave me an error about accepting 1 argument, but 3 were provided. Then I switched to…
Granted, I'm just a scientific programmer, but my workplace has a full blown software team maintaining a multi million line codebase. That codebase is rebuilt every night, and as I understand it, you're not allowed to submit a change that breaks anything. And they have people whose job is to keep their tools working. What people casually think of as "Python" is really a huge dynamic ecosystem of packages. Imagine tha…
When I am working on large projects like this and the miryad of problems which python and its friends bring to us, I tend to see if projects like "nuitka" can solve those problems. For example if you need to scale you jobs ust imagine how many sysopen_at syscalls one single python script causes, and usually multiply that to the number of cores your job will have. Nuitka solves most of those problems by packaging everythin. It is almost as staticallly linking your code to run in the cluster.
Re: My Python Development Environment, 2020 Edition
#178Earlier quoted context omitted.
The big gap is management of the full dependency tree. With yarn I can get a package.lock which pretty well ensures I'll have the same exact version of everything, with no unexpected changes, every time I run yarn install. I get the same thing in the Rust world with Cargo. In Python it's a mess. Some packages specify their deps in setup.py; some in a requirements file, which may or may not be read in by their setup.p…
> With yarn I can get a package.lock pipenv generates a Pipfile.lock - if it can, I've used it primarily for Airflow, and some packages within Airflow have incompatible version ranges for the same dependency, which means it can't generate the lock file.
Oh, thanks - I'll give it a whirl!
Re: My Python Development Environment, 2020 Edition
#179Earlier quoted context omitted.
npm is equivalent to combining pip and virtualenv into a single tool. This gives better ergonomics when switching between projects since you never have to "activate" your environment, it's always activated when standing in the project directory.
Isn't this what Pipenv does? What has been a downer for me is that many of the cloud providers do not support pipfiles in their serverless app services (Elastic Beanstalk, App Engine etc.)
On second thought, at least on GCP I should be able to put the pipfiles into .gcloudignore and just update the requirements.txt file with each new commit using git hooks, build scripts or a ci/cd tool.
Re: My Python Development Environment, 2020 Edition
#180Earlier quoted context omitted.
I think its a coincidence. Check nodejs commit history: https://github.com/NixOS/nixpkgs/commits/f3282c8d1e0ce6ba5d9... there are minor versions also
Right, so if I'm understanding correctly, I'd need to pin the whole nix-pkgs system to the particular git revision which contains the version I need?
That's it, also you can cherry pick different software from different git revisions.