Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

My Python Development Environment, 2018 Edition

jacobian.org

Post reply on HN
Print view View original

181–190 of 224 posts

Re: My Python Development Environment, 2018 Edition

#181

Post by acdha »

falcolas wrote: post #145
acdha wrote: post #92

Earlier quoted context omitted.

pipenv is basically the next level of this process: it records every install and locks only the things you installed rather than their dependencies which often change over time.

I'm a bit confused by this statement, do you mean: I install foo 1.0, which depends on frob >= 1.0 (which happens to be frob 1.1 when I installed it). Foo 1.1 comes out, as does frob 1.2 and 1.3. If I reinstall from the pipfile, do I get foo 1.0 with frob 1.3? I ask because that sounds like a bug waiting to happen. IMO, frozen requirements should remain frozen.

Here's what happens:

You type `pipenv install foo`. It will create the virtualenv if necessary and adds `foo = "​ * "` to the packages section of the Pipfile. The Pipfile.lock file will add a section for the package you installed _and_ all of its dependencies, including the hashes of the downloaded packages.

That avoids accidental breakage if the package you depend on doesn't set their versions correctly but it also means that your main Pipfile documents the things which you intentionally installed so a year from now you're not wondering why all of your servers have frob 1.2 installed, which only works on Python 2.7, even though nothing you're using now depends on it.

As a concrete example, here's what `pipenv install requests` looks like in a clean project:

Pipfile:

    [[source]]
    
    url = "https://pypi.python.org/simple"
    verify_ssl = true
    name = "pypi"
    
    
    [packages]
    
    requests = " * "
    
    
    [dev-packages]
(had I used `--python $(which python2.7)` it'd have recorded that as well)

Pipfile.lock:

    {
        "_meta": {
            "hash": {
                "sha256": "a0e63f8a0d1e3df046dc19b3ffbaaedfa151afc12af5a5b960ae7393952f8679"
            },
            "host-environment-markers": {
                "implementation_name": "cpython",
                "implementation_version": "3.6.4",
                "os_name": "posix",
                "platform_machine": "x86_64",
                "platform_python_implementation": "CPython",
                "platform_release": "17.4.0",
                "platform_system": "Darwin",
                "platform_version": "Darwin Kernel Version 17.4.0: Sun Dec 17 09:19:54 PST 2017; root:xnu-4570.41.2~1/RELEASE_X86_64",
                "python_full_version": "3.6.4",
                "python_version": "3.6",
                "sys_platform": "darwin"
            },
            "pipfile-spec": 6,
            "requires": {},
            "sources": [
                {
                    "name": "pypi",
                    "url": "https://pypi.python.org/simple",
                    "verify_ssl": true
                }
            ]
        },
        "default": {
            "certifi": {
                "hashes": [
                    "sha256:14131608ad2fd56836d33a71ee60fa1c82bc9d2c8d98b7bdbc631fe1b3cd1296",
                    "sha256:edbc3f203427eef571f79a7692bb160a2b0f7ccaa31953e99bd17e307cf63f7d"
                ],
                "version": "==2018.1.18"
            },
            "chardet": {
                "hashes": [
                    "sha256:fc323ffcaeaed0e0a02bf4d117757b98aed530d9ed4531e3e15460124c106691",
                    "sha256:84ab92ed1c4d4f16916e05906b6b75a6c0fb5db821cc65e70cbd64a3e2a5eaae"
                ],
                "version": "==3.0.4"
            },
            "idna": {
                "hashes": [
                    "sha256:8c7309c718f94b3a625cb648ace320157ad16ff131ae0af362c9f21b80ef6ec4",
                    "sha256:2c6a5de3089009e3da7c5dde64a141dbc8551d5b7f6cf4ed7c2568d0cc520a8f"
                ],
                "version": "==2.6"
            },
            "requests": {
                "hashes": [
                    "sha256:6a1b267aa90cac58ac3a765d067950e7dbbf75b1da07e895d1f594193a40a38b",
                    "sha256:9c443e7324ba5b85070c4a818ade28bfabedf16ea10206da1132edaa6dda237e"
                ],
                "version": "==2.18.4"
            },
            "urllib3": {
                "hashes": [
                    "sha256:06330f386d6e4b195fbfc736b297f58c5a892e4440e54d294d7004e3a9bbea1b",
                    "sha256:cc44da8e1145637334317feebd728bd869a35285b93cbb4cca2577da7e62db4f"
                ],
                "version": "==1.22"
            }
        },
        "develop": {}
    }
(EDITED: the HN Markdown parser appears to be a simple regex match and breaks formatting with a * and uses only the ASCII definition of whitespace so I couldn't use a zero-width space. The real output doesn't have spaces around the asterisks).

Re: My Python Development Environment, 2018 Edition

#182

Post by odonnellryan »

falcolas wrote: post #146
scaryclam wrote: post #122

Earlier quoted context omitted.

No, it's pretty much the same as you remember it. Using docker instead of just a normal virtualenv is overkill.

Overkill - Yes and No. Yes, because you are spinning up a full VM to run a docker container locally. No, because the container is also the container run in prod, with no opportunity for some other process to come along and hose your otherwise clean install. The process and FS isolation also make sysadmin-me all tingly inside. That way you can't hose up anybody else's clean install either (even if you're compromised).…

I'd never deploy more than 1 client to a machine anyway, so isolation in a security sense does not make much sense to me if I'm being honest.

But I understand. If the workflow works for you and/or your team, well, what's the problem? It's working!

Re: My Python Development Environment, 2018 Edition

#183

Post by odonnellryan »

amenod wrote: post #124
odonnellryan wrote: post #102

Earlier quoted context omitted.

Do you just use one environment for every project? That gets messy. What if you want to setup a new machine? Or a new employee's machine? Etc.. It isn't only about conflicting packages.

Yes - but then again, I don't have that many projects and I tend to use the same libraries across them anyway. Installing on another machine is just a matter of pip install -r requirements.txt , if it's for a dev. other wise it's just a docker run ... . I do know that most Python devs use virtualenv , I just never understood what all the fuss is about. Of course I don't mind if the others in team use it, I just never…

Yeah I understand what you mean. I have around ~12 projects, for different clients, that are just recent enough that I might need to access the environments.

If that was one environment (forget some of them being stuck on 2.x) the number of damn dependencies would be monstrous!

When I need to bring a partner onto the project, I can't give them a requirements file that's 5x what it should be!

> I just never understood what all the fuss is about.

It's to keep environments clean + easy to maintain. Virtual is probably a bad word for it. It isn't like Docker or a VM.

I don't think too many Python projects that are not libraries support many versions, that isn't the point of virtualenv or conda anyway.

Re: My Python Development Environment, 2018 Edition

#184

Post by adamson »

natch wrote: post #27

I’m using Anaconda because it was recommended in a step by step tutorial for playing with deep learning. What would be involved in removing it from my system and moving instead to this set of tools? Not necessarily looking for s step by step answer, just for general suggestions. My guess is: find out which python the deep learning tools are using, remove Anaconda, and reinstall the python version needed, using the to…

None of these should conflict with Anaconda (or at least Continuum conda, I'm not that familiar with what else is out there) unless you put the conda root bin directory on your path.

Re: My Python Development Environment, 2018 Edition

#185

Post by jorge-fundido »

erokar wrote: post #12

Every time I use Python I miss NPM and package.json.

For that flavor, maybe try:

    python3 -m pip install -t .pip ...
    export PYTHONPATH=".pip:$PYTHONPATH"
Use whatever directory name you prefer instead of .pip. Mucking with PYTHONPATH is a bit dirty.

Re: My Python Development Environment, 2018 Edition

#186

Post by natch »

natch wrote: post #27

I’m using Anaconda because it was recommended in a step by step tutorial for playing with deep learning. What would be involved in removing it from my system and moving instead to this set of tools? Not necessarily looking for s step by step answer, just for general suggestions. My guess is: find out which python the deep learning tools are using, remove Anaconda, and reinstall the python version needed, using the to…

Thanks for the replies everybody. One thing that's still confusing to Python tourists (my word for myself since I am usually programming in a different language, but come to Python occasionally to do something) is that everyone talks about pip, when actually it seems pip3 is required to install when using Python 3. Is this no longer the case? Or do people just say "pip" when they mean "pip3"? Or are people actually still living in 2.x land? I'm talking about pip the command line command/binary executable file, not pip the concept / tool name. Similar to the distinction between the capitalized "Python" (name of the language) versus "python" (command entered on the command line/name of the binary on the system).

Re: My Python Development Environment, 2018 Edition

#187

Post by gonvaled »

wirrbel wrote: post #158
james_the_cat wrote: post #62

Earlier quoted context omitted.

Anaconda does most of the stuff mentioned, and also makes it much easier to install packages based on C/C++ libraries (which most deep learning things are). So you're better off staying with anaconda. It's widely used in commercial data science projects so the idea that noone "takes it seriously" as someone else suggests is a bit silly. I assume they're thinking about a different context to data science projects. Tha…

Anaconda was the competition pip needed to become good. When Anaconda was introduced, I (a pip person), was impressed. However, at work we had pip workflows that were working okayish, so I never made the switch. Today, pip et. al. has so dramatically improved, that I hardly see a reason to use anaconda. I am not using deep learning stuff, so I cannot comment on this, but for most scientific python stuff (scikit learn…

I could provide a vagrant based repo that lets you spawn your ubuntu VM for dev in minutes. I'll try to get it in github.

Re: My Python Development Environment, 2018 Edition

#188

Post by [deleted] »

Rotareti wrote: post #134
foobarandgrill wrote: post #2

>Why? pipenv handles dependency- and virtual-environment-management in a way that’s very intuitive (to me), and fits perfectly with my desired workflow. Why specifically do you use it instead of virtualenv (+virtualenvwrapper)?

Pipenv combines package management and virtualenv managment in one tool. You can create a new project as simple as this: $ mkdir myproj $ cd myproj $ pipenv --python 3.6 # This creates a virtualenv with Python 3.6 for you project. $ pipenv install flask # This installs flask in your virtualenv. $ pipenv run flask # This runs flask in your virtualenv. $ pipenv run python # This runs a REPL with the interpreter of your…

[deleted]

Re: My Python Development Environment, 2018 Edition

#189

Post by bobwaycott »

tinymollusk wrote: post #18
symmitchry wrote: post #7

Earlier quoted context omitted.

Pyenv is great. On our Macs we've had zero issues installing older, specific versions of Python, every time. Highly recommended. (Getting it working properly with zsh was a bit frustrating, but that's my own fault.)

Do you know if it's easy to start using pyenv with existing projects, or should I wait until my next de novo project?

It’s pretty easy if you have a set of requirements handy and have `pyenv-virtualenv`:

    $ pyenv virtualenv 3.6 some-name
    $ echo some-name > /path/to/project/.python-version
    $ cd /path/to/project
    $ pip install -r requirements.txt
That’s it.

Re: My Python Development Environment, 2018 Edition

#190

Post by gonvaled »

sametmax wrote: post #9
foobarandgrill wrote: post #2

>Why? pipenv handles dependency- and virtual-environment-management in a way that’s very intuitive (to me), and fits perfectly with my desired workflow. Why specifically do you use it instead of virtualenv (+virtualenvwrapper)?

pipenv combine pip and venv. It's not just about activating. If you install, it will create the virtualenv if it's missing. It also, like pew, opens the virtualenv in a new shell instead of activating the current shell. A much saner approach. The UI is also more user friendly: one entry point for everything, pretty colors and icons, auto-correct of package name, and so on. Using Pipfiles, instead of requirements, are…

Does it install and manages different python versions? Pyenv does
Post reply on HN