It it considered best practice or advisable to run production deployments in a virtualenv? I have always considered it to be a tool for managing multiple development efforts on the same machine, not as production environment management.
My Python Development Environment, 2018 Edition
171–180 of 224 posts
Re: My Python Development Environment, 2018 Edition
#172Earlier quoted context omitted.
My problem with using Docker (only) is that it doesn't translate well to editors. Like, using jedi-vim[1] with a virtualenv constructed by a Docker container doesn't work at all. Unless I actually run vim itself inside said container. So unless your dependences build on macOS (like in my case), everything goes out the door. [1] https://github.com/davidhalter/jedi-vim
Ever heard about bind mounting? My current docker image for a Flask app doesn't even contains my code.
Are you mounting your virtualenv into the container somehow? How does your editor see the code/docs for your pip dependencies?
Re: My Python Development Environment, 2018 Edition
#173I'm a big fan of using Docker because for real world web app development, your app is often more than just getting Python and a virtualenv set up. Earlier this week I wrote about the pains of setting up a Python development experience without Docker, and then compared it to Docker as well. If anyone is curious, that's located at https://nickjanetakis.com/blog/setting-up-a-python-developme... . By the way, I would say…
>These are pretty big Flask apps too, which have thousands of lines of code, dozens of packages, tons of assets and require running Celery, Postgres, Redis, etc..
I was trying to create a new project last month trying to put all of these together. One thing that I was stuck in was putting these together:
- non-single-thread-Flask
- Redis pub-sub, subscribe to listen to a channel
- and websocket (such as socket.io) upon receiving redis sub message, pushing the message to client side.
I found out that this was a non-trivial thing to do, as redis subscribe is a blocking function, and I was not able to push websocket message from redis subscribe callback. I wonder how did they do that, if any?
Re: My Python Development Environment, 2018 Edition
#174Earlier quoted context omitted.
Why not seriously? Many data science environments are pretty much based on Anaconda installations. (See e.g. these dockers: https://github.com/jupyter/docker-stacks/tree/master/scipy-n... ) I mean, all packages I know DONT require Anaconda. But if you need the whole environment, sometimes Anaconda is the easiest tool too install all dependencies.
> Anaconda is the easiest tool too install all dependencies. For hobbyist stuff that’s fine, and I applaud lowering the entry barrier. My issue would be if I needed Anaconda to deploy the project/dependencies into “prod” in some professional capacity, instead of standard Python build tools. Having said that, I’m not terribly familiar with Anaconda, and it seems to leverage virtualenv under the hood, possibly with pre…
Anaconda does not leverage virtualenv under the hood. Anaconda does not use wheels as part of that. However, you can use pip and virtualenvs within Anaconda, but you'll probably get some inconsistent results.
The reason that Anaconda exists is to make it easier for people to have consistency between their dev environment, the build environment, and the prod environment. If you're seriously going to build all of the scientific Python stack from scratch, correctly and optimally, for your prod environment (and then also do the same for every dev environment you need to support), then you have way too much time and probably haven't actually tried doing it over any real period of time.
The problem isn't that it's impossible. The problem is that there are dozens of different ways to almost succeed, and you don't run into the problems until way too late.
Re: My Python Development Environment, 2018 Edition
#175Earlier quoted context omitted.
Anaconda does most of the stuff mentioned, and also makes it much easier to install packages based on C/C++ libraries (which most deep learning things are). So you're better off staying with anaconda. It's widely used in commercial data science projects so the idea that noone "takes it seriously" as someone else suggests is a bit silly. I assume they're thinking about a different context to data science projects. Tha…
> ... and also makes it much easier to install packages based on C/C++ libraries I hear this often, though I cannot remember ever running into a pip package where this was an issue. Out of curiosity, could someone point me to a pip package and its conda equivalent where this is the case?
This is also a useful read:
http://jakevdp.github.io/blog/2016/08/25/conda-myths-and-mis...
Re: My Python Development Environment, 2018 Edition
#176Earlier quoted context omitted.
Try pip installing scipy or Numpy and you'll see the value of conda.
As if that still was a problem, now that we have wheels, and especially manylinux wheels.
See Myth #6 here:
http://jakevdp.github.io/blog/2016/08/25/conda-myths-and-mis...
Re: My Python Development Environment, 2018 Edition
#177as someone who has to manage projects for multiple clients, I find docker + docker-compose to be the best solution. The overhead of docker is totally worth it because the container separation makes life so much easier. all I need to work on a client project is basically: cd /path/to/project docker-compose build (just once) docker-compose up
Absolutely. Though i have yet to have the chance to use it at work (large enterprise), it has totally been a game changer for me in terms of creating side projects in dev and seamless deployment. There is definitely some overhead and not everything plays nice all the time, but the thought of going back to running 5+ terminal windows / installing databases globally makes me shudder.
Re: My Python Development Environment, 2018 Edition
#178Earlier quoted context omitted.
> Anaconda is the easiest tool too install all dependencies. For hobbyist stuff that’s fine, and I applaud lowering the entry barrier. My issue would be if I needed Anaconda to deploy the project/dependencies into “prod” in some professional capacity, instead of standard Python build tools. Having said that, I’m not terribly familiar with Anaconda, and it seems to leverage virtualenv under the hood, possibly with pre…
"Hobbyist"? It's widely used in research in data science & machine learning. If you want a setup which works on various systems, and uses Python numeric packages, usually it is the most failsafe way to use with various OS. (Unless you want to put everything in docker.) Unless by "professional" you mean "building", well - then you have a point.
Yeah. Unless "Hobbyist" now also means my research university's 104 node, 2548 core HPC cluster...
Re: My Python Development Environment, 2018 Edition
#179This seems more like the 2008 Edition. You have to be pretty stubborn to not adopt Docker in 2018.
Re: My Python Development Environment, 2018 Edition
#180This seems more like the 2008 Edition. You have to be pretty stubborn to not adopt Docker in 2018.
If you're building a native python application, there's no reason to adopt yet another technology.
A native python app in "Docker/Containers" is going to look like a dockerfile that includes a copy of the app and runs three commands. Either you build that on the server (what's the point) or use a registry (additional complexity for little benefit).