Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

My Python Development Environment, 2020 Edition

jacobian.org

Post reply on HN
Print view View original

231–240 of 241 posts

Re: My Python Development Environment, 2020 Edition

#231

Post by pvg »

fock wrote: post #223
pvg wrote: post #216

Earlier quoted context omitted.

I don't understand the point you're making, where does a 'large collection of binaries' come in? The thing I'm talking about is right in the name 'virtual environment'. Or are you simply answering the GP's comment about this being a potential wart in Python with the suggestion that people should just use something else?

the point is that the classic virtualenv does not take care of different base distributions (3.5,3.6,3.7...). You have to install them yourselves and switch them in your PATH before setting up a virtualenv. if this is too much "conceptual load", I wish the parent good luck with finding something not having this problem ;). If there are binaries involved, there is the additional problem that you also depend on compile…

I think we are talking about completely different things. Forget the actual virtualenv and Python for a sec.

Imagine you want to check out the hypothetical language Jagoust. This goes something like so - you install Jagoust and maybe some Jagoust tools. You fire up your interactive environment and set it up so it knows where Jagoust and its tools are. Using your enviornment you do things like tell Jagoust where your project is or which particular version of Jagoust to use, etc.. And that's pretty much it and while the details are different, the process is quite familiar to you since it just leverages your standard interactive environment in ways mechanically similar to Rugoja, another language you know.

Later you decide you want to try the language Snek. The first thing you notice in the Snek tutorial is that your standard interactive environment is somehow not good enough - you need a meta-environment for it and your project. Perplexed, you google some more docs and ask some Snek people and find out that meta-environment used to be how things were done but these days, you probably want a quasi- or para-environment. But why do you need an environment for your environment? You do not know. Such are the mysteries of the Snek.

Re: My Python Development Environment, 2020 Edition

#232

Post by marmada »

fiedzia wrote: post #166
marmada wrote: post #31

Does anyone else think this reflects badly on Python? The fact that the author has to use a bunch of different tools to manage Python versions/projects is intimidating. I don't say this out of negativity for the sake of negativity. Earlier today, I was trying to resurrect an old Python project that was using pipenv. "pipenv install" gave me an error about accepting 1 argument, but 3 were provided. Then I switched to…

> The fact that the author has to use a bunch of different tools to manage Python versions/projects is intimidating. It just shows that Python is used for a lot of purposes and there is no single tool that handles all usecases. > I gave up and started rewriting the project (web scraper) in Node.js with Puppeteer. And when you'll get to the point where you need to work with different node projects, you will also need…

The thing is, if I need to switch node versions. I can use nvm. The thing is, I don't need to manage different environments in node because the dependencies are contained in "node_modules" and not "attached" to a Python interpreter instance.

Re: My Python Development Environment, 2020 Edition

#233

Post by fock »

pvg wrote: post #231
fock wrote: post #223

Earlier quoted context omitted.

the point is that the classic virtualenv does not take care of different base distributions (3.5,3.6,3.7...). You have to install them yourselves and switch them in your PATH before setting up a virtualenv. if this is too much "conceptual load", I wish the parent good luck with finding something not having this problem ;). If there are binaries involved, there is the additional problem that you also depend on compile…

I think we are talking about completely different things. Forget the actual virtualenv and Python for a sec. Imagine you want to check out the hypothetical language Jagoust. This goes something like so - you install Jagoust and maybe some Jagoust tools. You fire up your interactive environment and set it up so it knows where Jagoust and its tools are. Using your enviornment you do things like tell Jagoust where your…

Well, you need some way to switch between different variants of Jagoust in the first setup too? And if everything is so well integrated (w/o bash-scripts and all...), just name Jagoust ;).

Noone forces people to use virtualenvs with "snek"/Python. They can always install python somewhere and "pip install" packages into the installation hierarchy, switching around different hierarchies with bash-scripts (and now have a look what all the other interactive environments do...).

Re: My Python Development Environment, 2020 Edition

#234

Post by soraminazuki »

shpx wrote: post #170
whateveracct wrote: post #93

Earlier quoted context omitted.

This is why I manage every nontrivial project I do nowadays with Nix (Haskell, Go, Python, C & C++, bash ..anything) Everything is pinned to exact source revisions. You can be relatively sure to be able to git clone and nix-shell and be off to the races. You can even go the extra mile and provide working editor integration in your nix-shell (especially easy with emacs). So you can enable anyone to git clone, nix-shel…

How do you specify Python dependencies and their versions in Nix?

A quick and dirty way would be:

$ nix run "(import {}).python37.withPackages([ pandas statsmodels ])"

This would drop you into a shell with Python 3.7, pandas, and statsmodels installed.

For more complex use cases, you should specify your dependencies in a shell.nix file or write your own nix package.

Re: My Python Development Environment, 2020 Edition

#235

Post by soraminazuki »

silviogutierrez wrote: post #221
huevo5050 wrote: post #180

Earlier quoted context omitted.

That's it, also you can cherry pick different software from different git revisions.

Do you have any public examples of doing this? Or docs? Closest I found was to pin the nix-channel entirely, but not different components to different versions.

You could do something like:

    let pinnedPkgs = import (builtins.fetchTarball {...}) {};
        node = pinnedPkgs.nodejs-10_x;
to refer to a specific version of node.

https://nixos.wiki/wiki/FAQ/Pinning_Nixpkgs

Re: My Python Development Environment, 2020 Edition

#236

Post by soraminazuki »

s_kilk wrote: post #172
whateveracct wrote: post #93

Earlier quoted context omitted.

This is why I manage every nontrivial project I do nowadays with Nix (Haskell, Go, Python, C & C++, bash ..anything) Everything is pinned to exact source revisions. You can be relatively sure to be able to git clone and nix-shell and be off to the races. You can even go the extra mile and provide working editor integration in your nix-shell (especially easy with emacs). So you can enable anyone to git clone, nix-shel…

> Everything is pinned to exact source revisions. While you're here, how do you do this with nix-pkgs? I looked into using nixos for nodejs deployments recently, and was amazed to find that the versions of node in the nix-pkgs repo are just pinned to X.Y.0 releases, with no discernable way to update to a bugfix release after .0 , so... I don't see how this could possibly be used for production deployments? https://ni…

While the nix package manager supports the coexistence of multiple versions of a package, the nixpkgs package collection does not contain every single versions of every package. However, it does make it very easy to refer to past versions of a package by importing package specifications from an older version of nixpkgs.

I think this is a reasonable choice, considering that the main purpose of nixpkgs is to provide packages for the NixOS distribution. It's impossible to actively maintain every single version of every package.

Re: My Python Development Environment, 2020 Edition

#237

Post by silviogutierrez »

soraminazuki wrote: post #235
silviogutierrez wrote: post #221

Earlier quoted context omitted.

Do you have any public examples of doing this? Or docs? Closest I found was to pin the nix-channel entirely, but not different components to different versions.

You could do something like: let pinnedPkgs = import (builtins.fetchTarball {...}) {}; node = pinnedPkgs.nodejs-10_x; to refer to a specific version of node. https://nixos.wiki/wiki/FAQ/Pinning_Nixpkgs

Thanks that helps.

I've mostly stuck to nix-shell -p foo bar baz to setup my environments. It's holding me back and I should pull the trigger on a shell.nix file, just gotta learn the language.

Re: My Python Development Environment, 2020 Edition

#238

Post by raihansaputra »

Twirrim wrote: post #83
Daishiman wrote: post #58

Earlier quoted context omitted.

Did you even read the article?

I did, certainly don't see what specific value poetry is providing from what he said.

AFAIK, pipenv and poetry also pins down the dependencies requirements to the same versions, while pip don't (?).

Re: My Python Development Environment, 2020 Edition

#239

Post by whateveracct »

soraminazuki wrote: post #234
shpx wrote: post #170

Earlier quoted context omitted.

How do you specify Python dependencies and their versions in Nix?

A quick and dirty way would be: $ nix run "(import {}).python37.withPackages([ pandas statsmodels ])" This would drop you into a shell with Python 3.7, pandas, and statsmodels installed. For more complex use cases, you should specify your dependencies in a shell.nix file or write your own nix package.

And then to expand, if you wish to pin one of those packages to a specific rev, the easiest way is to create an "overlay" (it's a Nix design pattern) that you apply to nixpkgs to override whatever version is in your nixpkgs version.

Re: My Python Development Environment, 2020 Edition

#240

Post by theptip »

luhn wrote: post #227
theptip wrote: post #96

Earlier quoted context omitted.

Last time I looked, it was very tedious to set up `pip` to be secure and pin your dependencies to hashes. Without this, a compromise of a library's pypa account would allow them to execute arbitrary code on your system, assuming you didn't notice the change. You can use `pip-tools` to get something like a Gemfile/package.json, but there are a few restrictions that are suboptimal. So Pipenv/Poetry are the current best…

What do you find restrictive about pip-tools? It's been working excellent for me.

I was going to say that pinning hashes seems to be painful, but since I last looked it now has a --generate-hashes flag.

Also if you want to link directly to a git repo, you can only install it in editable mode, you can't just install it with the rest of your packages (means you get a `src` directory where you ran pip, which makes Dockerizing slightly annoying, and probably impacts performance slightly).

Maybe this has also been fixed since I last upgraded.

Post reply on HN