From Python to Go and Back Again • Hacker News Board

Re: From Python to Go and Back Again

#31

Post by DasIch » Sat, Oct 17, 2015, 12:10 AM UTC

stock_toaster wrote: post #20

The claim that pypy uses less memory than Go seems...rather extraordinary. I worked with a fairly complex http api app that ran as a rather svelte wsgi framework under gunicorn, and we saw at least 10x or more increase in memory usage than cpython when we switched to pypy, once the jit was fully warmed up (memory usage seemed to hit steady state after about an hour). pypy has also historically (in my experience) been…

PyPy's JIT requires quite a bit of memory. CPython doesn't have a JIT so it doesn't share that problem. However PyPy also has optimizations specifically to keep memory usage down. Especially if you have a lot of objects, it's certainly possible to consume less memory than CPython.

PyPy using less memory than Go does seem weird but depending on how much the GCs differ and how they are configured it could simply be that Go's GC doesn't give up memory as freely to the OS.

Re: From Python to Go and Back Again

#32

Post by DasIch » Sat, Oct 17, 2015, 12:15 AM UTC

scott_karana wrote: post #7

zeckalpha wrote: post #6

Yes, it is time for Rust.

Yeah, Rust sounds like a great Python replacement. /s I do like Rust, but must we clutter every Go-related thread with comments about it? The zealousness is annoying.

If you're problem is that you're memory bound, using a language that allows precise control over memory allocation and layout is certainly a good idea.

The constant conflict might be annoying but using Rust or even C++ would certainly be a very reasonable choice.

Re: From Python to Go and Back Again

#33

Post by thristian » Sat, Oct 17, 2015, 12:15 AM UTC

falcolas wrote: post #12

I'll be interested in coming back to Python when it isn't a headache to deploy into production. I'm tired of an install requiring a GCC compiler on the target node. I'm also tired of having to work around the language and ecosystem to avoid dependency hell.

The way I deploy Python apps at $EMPLOYER:

- CI system detects a commit and checks out the latest code

- CI system makes a virtualenv and sets up the project and its dependencies into it with "pip install --editable path/to/checkout"

- CI system runs tests, computes coverage, etc.

- CI system makes a output directory and populates it with "pip wheel --wheel-dir path/to/output path/to/checkout"

- Deployment system downloads wheels to a temporary location

- Deployment system makes a virtualenv in the right location

- Deployment system populates virtualenv with "pip install --no-deps path/to/temp/location/*.whl"

The target node only needs a compatible build of python and the virtualenv package installed; it doesn't need a compiler and only needs a network connection if you want to transfer wheel files that way.

Re: From Python to Go and Back Again

#34

Post by kawera » Sat, Oct 17, 2015, 12:18 AM UTC

CatDevURandom wrote: post #2

I use and like both python and go. The presentation mirrors my own experiences, though I haven't come back again yet.

I use and like both too; webapps in python (django/flask) and microservices in go. My systems also use some large programs written in go but more like black boxes (nsq, heka, influxdb).

Re: From Python to Go and Back Again

#35

Post by JulianMorrison » Sat, Oct 17, 2015, 12:22 AM UTC

Huge interfaces are not how you do it. Little ones, each expressing a conceptual unit of the functionality. Loggable. MessageConsumer. Stuff like that, even if it only wraps one method. Make each major subsystem into an object, connect to it through the mini-interfaces it provides, and test its consumers by stubbing those mini-interfaces, not the whole object.

Re: From Python to Go and Back Again

#36

Post by JulianMorrison » Sat, Oct 17, 2015, 12:24 AM UTC

pjmlp wrote: post #28

flogic wrote: post #24

Earlier quoted context omitted.

I've tried a couple of times to play with Ocaml. In seem like something that should be great but it just falls short. Some of that is tooling that isn't fully baked. Another issue is the syntax. The weird split between the interface description file and the code file. The lack of a unified DB interface. The lack of proper Windows support.

That weird split is how most languages with modules work.

In languages that don't fully parse the dependencies (which Go does), that split uncouples "the implementation needs recompiling" from "all the dependencies need recompiling."

Re: From Python to Go and Back Again

#37

Post by ntrepid8 » Sat, Oct 17, 2015, 12:27 AM UTC

This echoes some of my own experiences. Python (even CPython) waits on the network just as fast as Go (or anything else).

I've had good luck writing applications in Python, then profiling them and implementing critical sections that are CPU bound in C as modules. Any sections that are memory hogs can be converted to stream processors.

Lately I've started implementing the modules with Rust and the results are promising. It seems like a nice balance of developer productivity and application performance.

Re: From Python to Go and Back Again

#38

Post by bmurphy1976 » Sat, Oct 17, 2015, 12:28 AM UTC

falcolas wrote: post #26

rattray wrote: post #25

Earlier quoted context omitted.

docker?

Doesn't solve the issue of needing a C compiler for third party extensions, and definitely qualifies as a work-around for the existing toolset. Yes, it helps. But you can use Docker with Go programs as well (and drop a lot more of the base image in the process).

Actually, nice thing about docker is you can build a compilation container (pre-built with all your C/C++ apps ready to go and shared amongst your coworkers), compile your extensions using that, and then only install them into your target container (sans compilation tools). It's a little more grunt work that way, but you get better control and reproducibility without the explosion in image sizes.

Re: From Python to Go and Back Again

#39

Post by eugenekolo2 » Sat, Oct 17, 2015, 12:35 AM UTC

Does anybody have some comments on the closing statements on Google's 10KB secret?

Re: From Python to Go and Back Again

#40

Post by zweiterlinde » Sat, Oct 17, 2015, 12:36 AM UTC

falcolas wrote: post #26

rattray wrote: post #25

Earlier quoted context omitted.

docker?

Doesn't solve the issue of needing a C compiler for third party extensions, and definitely qualifies as a work-around for the existing toolset. Yes, it helps. But you can use Docker with Go programs as well (and drop a lot more of the base image in the process).

The way we do this is to have a base image that has already yum installed or pip installed all modules (non trivial, anyway) that our package needs. Then the docker image that needs to be rebuilt (that depends on the first one) is just a minimal pip install away.