Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

My Python Development Environment, 2018 Edition

jacobian.org

Post reply on HN
Print view View original

161–170 of 224 posts

Re: My Python Development Environment, 2018 Edition

#161

Post by CMCDragonkai »

MarcScott wrote: post #34

Since switching to nixos, my Python development environment couldn't be more satisfying. I use a default.nix file and a requirements.txt file and then with a single command I'm into a shell and virtual environment with all dependencies and packages installed, that I can easily transfer between machines. That is unless I want to use PyQt5.

My understanding is that you should be using shell.nix for your dev environment, and default.nix if you want to actually build the final "nix-package". I have a gist that talks about this along with python and other language environments: https://gist.github.com/CMCDragonkai/dcc1b538352624ea690d695... However it's not finished, there are some things that still need work. However package/input isolation is not the only isolation you need sometimes (for example, network isolation, fs isolation...etc). When you do need this, Docker provides all these, but it's not compositional. There's a discussion on the Nix issues about this (potentially working towards a new nix-shell development environment can can compose different kinds of isolation): https://github.com/NixOS/nix/issues/903#issuecomment-3647447...

Re: My Python Development Environment, 2018 Edition

#162

Post by jsmeaton »

mikewhy wrote: post #152
Spiritus wrote: post #131

Earlier quoted context omitted.

My problem with using Docker (only) is that it doesn't translate well to editors. Like, using jedi-vim[1] with a virtualenv constructed by a Docker container doesn't work at all. Unless I actually run vim itself inside said container. So unless your dependences build on macOS (like in my case), everything goes out the door. [1] https://github.com/davidhalter/jedi-vim

Lots of people mentioning you can maybe debug a remote process, but none suggesting what to do about completions. Maybe you can use bind mounts, but I'm not sure how well that works in practice, or with eg a package with a native component.

See my comment a few above. Setup a virtualenv locally that your dev env points to for completions, but actually run the app within the container (with its own virtualenv).

Docker is great for dependencies like databases and queues. I find it totally unnecessary for developing python.

Re: My Python Development Environment, 2018 Edition

#163

Post by wirrbel »

amenod wrote: post #55
pjmlp wrote: post #14

I just have latest Python installed and the nice support on Visual Studio. https://www.visualstudio.com/vs/python/

This, me too. I never understood the need for virtualenv and similar. Do people really encounter trouble with conflicting packages that often? I try to write scripts so they run on different versions of python anyway, unless there is a very specific reason why that is not possible; and even then you can run python versions in parallel on a Debian/Ubuntu box, with different pip installs for each of them. As for produc…

Docker containers and virtual envs are in some way both capsules that can serve a similar purpose. virtual envs are in general more lightweight, but limited to the Python world, whereas Docker containers are heavier, but enable you to also include other dependencies in that capsule.

I sometimes feel that Docker containers are a bit misused if they are just used as environment capsules. They provide more isolation (process space, etc.). But of course, that would be still fine.

The software I write mostly does not require databases and other supporting services to run on my laptop. So virtualenvs are all I need (most of the time). There is a docker tax that I don't want to pay, unless I heavily benefit from it.

I guess, if I could deploy docker images, the return of invest would be much better and I might use it more.

Re: My Python Development Environment, 2018 Edition

#164

Post by sheraz »

_drFaust wrote: post #144
Spiritus wrote: post #131

Earlier quoted context omitted.

My problem with using Docker (only) is that it doesn't translate well to editors. Like, using jedi-vim[1] with a virtualenv constructed by a Docker container doesn't work at all. Unless I actually run vim itself inside said container. So unless your dependences build on macOS (like in my case), everything goes out the door. [1] https://github.com/davidhalter/jedi-vim

We've got about 30+ backends in python all wrapped in docker containers. Majority of the team was pure vim before I joined and they're slowly converting to pycharm after seeing how nicely you can setup a remote interpreter against a docker container. And it has vim bindings so you don't have to re-learn new hotkeys. I've also been following this VS Code issue on adding remote docker support for python https://github.…

yup. happy pycharm and docker user here. even have it working with debugging and breakpoints.

Re: My Python Development Environment, 2018 Edition

#166

Post by james_the_cat »

wirrbel wrote: post #158
james_the_cat wrote: post #62

Earlier quoted context omitted.

Anaconda does most of the stuff mentioned, and also makes it much easier to install packages based on C/C++ libraries (which most deep learning things are). So you're better off staying with anaconda. It's widely used in commercial data science projects so the idea that noone "takes it seriously" as someone else suggests is a bit silly. I assume they're thinking about a different context to data science projects. Tha…

Anaconda was the competition pip needed to become good. When Anaconda was introduced, I (a pip person), was impressed. However, at work we had pip workflows that were working okayish, so I never made the switch. Today, pip et. al. has so dramatically improved, that I hardly see a reason to use anaconda. I am not using deep learning stuff, so I cannot comment on this, but for most scientific python stuff (scikit learn…

Tensorflow has a pip wheel, so probably pip works just as well as conda. Maybe mkl is easier with conda, though then again I constantly install the "nomkl" meta package in conda because it adds hundreds of megabytes to Docker images and doesn't seem to have any positive effect for anything I've worked on.

As you say, they're now sufficiently similar that there's no big reason to switch from one to the other...

Re: My Python Development Environment, 2018 Edition

#167

Post by stared »

guitarbill wrote: post #154
stared wrote: post #69

Earlier quoted context omitted.

Why not seriously? Many data science environments are pretty much based on Anaconda installations. (See e.g. these dockers: https://github.com/jupyter/docker-stacks/tree/master/scipy-n... ) I mean, all packages I know DONT require Anaconda. But if you need the whole environment, sometimes Anaconda is the easiest tool too install all dependencies.

> Anaconda is the easiest tool too install all dependencies. For hobbyist stuff that’s fine, and I applaud lowering the entry barrier. My issue would be if I needed Anaconda to deploy the project/dependencies into “prod” in some professional capacity, instead of standard Python build tools. Having said that, I’m not terribly familiar with Anaconda, and it seems to leverage virtualenv under the hood, possibly with pre…

"Hobbyist"? It's widely used in research in data science & machine learning.

If you want a setup which works on various systems, and uses Python numeric packages, usually it is the most failsafe way to use with various OS. (Unless you want to put everything in docker.)

Unless by "professional" you mean "building", well - then you have a point.

Re: My Python Development Environment, 2018 Edition

#168

Post by Rotareti »

_coveredInBees wrote: post #106
Rotareti wrote: post #86

Earlier quoted context omitted.

> ... and also makes it much easier to install packages based on C/C++ libraries I hear this often, though I cannot remember ever running into a pip package where this was an issue. Out of curiosity, could someone point me to a pip package and its conda equivalent where this is the case?

Try pip installing scipy or Numpy and you'll see the value of conda.

Just ran `pipenv install scipy numpy`. Installs like any other package..

Re: My Python Development Environment, 2018 Edition

#169

Post by jonwayne »

twexler wrote: post #84

What seems to be missing in the comments here is the `--user` option to pip. Lets you install modules on a per-user basis, doesn't mess with system python. All you need to do is add the bin folder this creates to your path.

pipsi installs into isolated virtualenvs, then symlinks into `~/.local/bin` just like `pip install --user`. Combining pipsi for cross-project tools (tox, twine, nox, etc.) with pipenv for project-specific packages is all you need. You don't need to `pip install --user` some package at that point.

Re: My Python Development Environment, 2018 Edition

#170

Post by jessaustin »

jrs95 wrote: post #83
erokar wrote: post #12

Every time I use Python I miss NPM and package.json.

Hell, prior to pipenv being a thing, Python package management made me miss Maven of all things!

Not having done anything in python in a few years, it is nice to learn that it's catching up. It may be hoped that the npm-bashers will learn to appreciate the new features available for python.
Post reply on HN