My Python Development Environment, 2018 Edition • Hacker News Board

Re: My Python Development Environment, 2018 Edition

#151

Post by nickjj » Thu, Feb 22, 2018, 9:57 PM UTC

_drFaust wrote: post #144

Spiritus wrote: post #131

Earlier quoted context omitted.

My problem with using Docker (only) is that it doesn't translate well to editors. Like, using jedi-vim[1] with a virtualenv constructed by a Docker container doesn't work at all. Unless I actually run vim itself inside said container. So unless your dependences build on macOS (like in my case), everything goes out the door. [1] https://github.com/davidhalter/jedi-vim

We've got about 30+ backends in python all wrapped in docker containers. Majority of the team was pure vim before I joined and they're slowly converting to pycharm after seeing how nicely you can setup a remote interpreter against a docker container. And it has vim bindings so you don't have to re-learn new hotkeys. I've also been following this VS Code issue on adding remote docker support for python https://github.…

Funny you mentioned that issue. I'm eagerly awaiting that feature too.

I commented in the issue a few weeks ago: https://github.com/Microsoft/vscode-python/issues/79#issueco....

Once that's implemented, oh man, development nirvana.

Re: My Python Development Environment, 2018 Edition

#152

Post by mikewhy » Thu, Feb 22, 2018, 9:59 PM UTC

Spiritus wrote: post #131

nickjj wrote: post #113

I'm a big fan of using Docker because for real world web app development, your app is often more than just getting Python and a virtualenv set up. Earlier this week I wrote about the pains of setting up a Python development experience without Docker, and then compared it to Docker as well. If anyone is curious, that's located at https://nickjanetakis.com/blog/setting-up-a-python-developme... . By the way, I would say…

My problem with using Docker (only) is that it doesn't translate well to editors. Like, using jedi-vim[1] with a virtualenv constructed by a Docker container doesn't work at all. Unless I actually run vim itself inside said container. So unless your dependences build on macOS (like in my case), everything goes out the door. [1] https://github.com/davidhalter/jedi-vim

Lots of people mentioning you can maybe debug a remote process, but none suggesting what to do about completions.

Maybe you can use bind mounts, but I'm not sure how well that works in practice, or with eg a package with a native component.

Re: My Python Development Environment, 2018 Edition

#153

Post by wirrbel » Thu, Feb 22, 2018, 9:59 PM UTC

_drFaust wrote: post #144

Spiritus wrote: post #131

Earlier quoted context omitted.

My problem with using Docker (only) is that it doesn't translate well to editors. Like, using jedi-vim[1] with a virtualenv constructed by a Docker container doesn't work at all. Unless I actually run vim itself inside said container. So unless your dependences build on macOS (like in my case), everything goes out the door. [1] https://github.com/davidhalter/jedi-vim

We've got about 30+ backends in python all wrapped in docker containers. Majority of the team was pure vim before I joined and they're slowly converting to pycharm after seeing how nicely you can setup a remote interpreter against a docker container. And it has vim bindings so you don't have to re-learn new hotkeys. I've also been following this VS Code issue on adding remote docker support for python https://github.…

I remember, on my laptop with pycharm and docker (and the virtual machine docker lived in), the RAM usage was just excessive. I am by no means a minimalist, but data-science stuff was barely impossible with 16 GB RAM.

Also, I strongly dislike PyCharm. I am a vim guy by heart, but I am generally not against IDEs. VS Code is okayish. For C++ development, I really loved Visual Studio. But PyCharm just feels wrong, bloated, slow and baroque

Re: My Python Development Environment, 2018 Edition

#154

Post by guitarbill » Thu, Feb 22, 2018, 10:02 PM UTC

stared wrote: post #69

guitarbill wrote: post #50

Earlier quoted context omitted.

I think you could probably use both, side-by-side (but not at the same time/for the same project). Anaconda might be nicer if the packages you need are C-based and would need compiling on your platform. Depends on your use-case. For experimentation/playing around, stick with what works for you. But if a project required me to install Anaconda, I wouldn’t take it seriously. And pipenv is pretty easy to use, too. So if…

Why not seriously? Many data science environments are pretty much based on Anaconda installations. (See e.g. these dockers: https://github.com/jupyter/docker-stacks/tree/master/scipy-n... ) I mean, all packages I know DONT require Anaconda. But if you need the whole environment, sometimes Anaconda is the easiest tool too install all dependencies.

> Anaconda is the easiest tool too install all dependencies.

For hobbyist stuff that’s fine, and I applaud lowering the entry barrier. My issue would be if I needed Anaconda to deploy the project/dependencies into “prod” in some professional capacity, instead of standard Python build tools. Having said that, I’m not terribly familiar with Anaconda, and it seems to leverage virtualenv under the hood, possibly with pre-compiled packages (like wheels?).

Re: My Python Development Environment, 2018 Edition

#155

Post by wirrbel » Thu, Feb 22, 2018, 10:05 PM UTC

Rotareti wrote: post #134

foobarandgrill wrote: post #2

>Why? pipenv handles dependency- and virtual-environment-management in a way that’s very intuitive (to me), and fits perfectly with my desired workflow. Why specifically do you use it instead of virtualenv (+virtualenvwrapper)?

Pipenv combines package management and virtualenv managment in one tool. You can create a new project as simple as this: $ mkdir myproj $ cd myproj $ pipenv --python 3.6 # This creates a virtualenv with Python 3.6 for you project. $ pipenv install flask # This installs flask in your virtualenv. $ pipenv run flask # This runs flask in your virtualenv. $ pipenv run python # This runs a REPL with the interpreter of your…

let me add, that if you do a virtualenv-based workflow, you will end up having some kind of a requirements list (requirements.txt), then you will realize that you want to also version the configuration of that virtualenv with all dependencies resolved (think `pip freeze`). You'll start to sort development dependencies like pytest from production requirements like `six`, by this time you will have written a few scripts to deal with this stuff.

This is where pipenv delivers. It is a destillation of best practices for virtualenv-configuration.

In your overview, I'd just add an example for a dev installation

    pipenv install --dev pytest

Re: My Python Development Environment, 2018 Edition

#156

Post by wirrbel » Thu, Feb 22, 2018, 10:06 PM UTC

Lxr wrote: post #117

acdha wrote: post #24

Earlier quoted context omitted.

I was recently explaining this here — you still end up with a virtualenv so it's not a difference in capabilities but rather ease of use: 1. It transparently creates the virtualenv for you 2. The pipfile format handles dependencies and version locking (including hashes of packages), including updates. That means that the versions won't change without your knowledge but upgrading to the latest versions of everything i…

To further elaborate on 2, it solves the problem of maintaining loose version ranges in your requirements.txt file, but keeping the versions pinned when you deploy. For example if you put `foo>=2` in your requirements.txt, this is dangerous without some way of pinning e.g. `foo==2.18.2` and running your tests against that before you deploy. But you obviously don't want to manually edit requirements.txt with minor ver…

Don't forget that usually you'll start to sort your requirements into dev requirements and production requirements which makes these packaging scripts much more complicated.

https://github.com/jazzband/pip-tools would be what I used before pipenv came to be.

Re: My Python Development Environment, 2018 Edition

#157

Post by wirrbel » Thu, Feb 22, 2018, 10:08 PM UTC

_coveredInBees wrote: post #106

Rotareti wrote: post #86

Earlier quoted context omitted.

> ... and also makes it much easier to install packages based on C/C++ libraries I hear this often, though I cannot remember ever running into a pip package where this was an issue. Out of curiosity, could someone point me to a pip package and its conda equivalent where this is the case?

Try pip installing scipy or Numpy and you'll see the value of conda.

As if that still was a problem, now that we have wheels, and especially manylinux wheels.

Re: My Python Development Environment, 2018 Edition

#158

Post by wirrbel » Thu, Feb 22, 2018, 10:14 PM UTC

james_the_cat wrote: post #62

natch wrote: post #27

I’m using Anaconda because it was recommended in a step by step tutorial for playing with deep learning. What would be involved in removing it from my system and moving instead to this set of tools? Not necessarily looking for s step by step answer, just for general suggestions. My guess is: find out which python the deep learning tools are using, remove Anaconda, and reinstall the python version needed, using the to…

Anaconda does most of the stuff mentioned, and also makes it much easier to install packages based on C/C++ libraries (which most deep learning things are). So you're better off staying with anaconda. It's widely used in commercial data science projects so the idea that noone "takes it seriously" as someone else suggests is a bit silly. I assume they're thinking about a different context to data science projects. Tha…

Anaconda was the competition pip needed to become good. When Anaconda was introduced, I (a pip person), was impressed. However, at work we had pip workflows that were working okayish, so I never made the switch.

Today, pip et. al. has so dramatically improved, that I hardly see a reason to use anaconda. I am not using deep learning stuff, so I cannot comment on this, but for most scientific python stuff (scikit learn, pandas, numpy, etc.). Pip and pre-built wheels work very well.

On my new job, I was given a Windows laptop that I happen to use now for my development work (because I am too stupid/lazy to properly configure and maintain a separate virtual linux machine). And I started with anaconda, assuming it was less trouble, but quickly ran into trouble. pip worked like a charm [].

[] Ironically pip is installed via the anaconda base installation if I am not mistaken :)

Re: My Python Development Environment, 2018 Edition

#159

Post by jsmeaton » Thu, Feb 22, 2018, 10:17 PM UTC

_drFaust wrote: post #144

Spiritus wrote: post #131

Earlier quoted context omitted.

My problem with using Docker (only) is that it doesn't translate well to editors. Like, using jedi-vim[1] with a virtualenv constructed by a Docker container doesn't work at all. Unless I actually run vim itself inside said container. So unless your dependences build on macOS (like in my case), everything goes out the door. [1] https://github.com/davidhalter/jedi-vim

We've got about 30+ backends in python all wrapped in docker containers. Majority of the team was pure vim before I joined and they're slowly converting to pycharm after seeing how nicely you can setup a remote interpreter against a docker container. And it has vim bindings so you don't have to re-learn new hotkeys. I've also been following this VS Code issue on adding remote docker support for python https://github.…

What I do is create the virtualenv locally that docker would also create, and point my editor to the local virtualenv. It gives me all of the intellisense locally. I still need to debug within the container/shell for the moment but I'm hoping there's a pathway coming for vscode that pycharm already has.

Re: My Python Development Environment, 2018 Edition

#160

Post by raduhek » Thu, Feb 22, 2018, 10:18 PM UTC

I didn't manage to get a run at pyenv, pipsi or pipenv, but for virtualenvwrapper and tmux work great together.

I'm running linux at work and usually keep it alive for days, so tmux is used for session keeping and remote work. I've created a few bash scripts which run some tmux commands for setting up the layout as I want it and also execute "workon" for the specific virtualenv.

For working on a new project, I've another bash script which I run with the repo url and automatically clones it, creates the virtualenv with the same name as the project, installs dependencies and starts a new tmux sessions with two panes in first window and a second window for other stuff.

The great thing about tmux for me is it's low memory footprint so I can have 10-15 sessions running at a time, without worrying about the computer slowing down. What takes a bit too long is setting it up again upon a restart.