I am surprised the GIL hasn't been mentioned once in the presentation or any of the comments.
From Python to Go and Back Again
61–70 of 167 posts
Re: From Python to Go and Back Again
#62Earlier quoted context omitted.
The way I deploy Python apps at $EMPLOYER: - CI system detects a commit and checks out the latest code - CI system makes a virtualenv and sets up the project and its dependencies into it with "pip install --editable path/to/checkout" - CI system runs tests, computes coverage, etc. - CI system makes a output directory and populates it with "pip wheel --wheel-dir path/to/output path/to/checkout" - Deployment system dow…
The way I deploy Go apps at $EMPLOYER2: - go get - go test - go build - copy to target It's possible with Python, it's easier with Go. It's a place where we could use a lot of progress.
It seems weird how you can't easily package python into an executable without Docker.
Re: From Python to Go and Back Again
#63I am surprised the GIL hasn't been mentioned once in the presentation or any of the comments.
Why would it? Using twisted (or py3's asyncio), the event loop is spinning, there is no thread for Python to be switching to. Except that we're using a thread-pool of course (to make external blocking network calls), and those threads are.... making network requests, which is I/O, and the GIL releases during that.
On a network bound daemon, which is not CPU-bound, the GIL is really not an issue, so it never came up.
Re: From Python to Go and Back Again
#64I am surprised the GIL hasn't been mentioned once in the presentation or any of the comments.
node.js has a GIL and nobody talks about that either. The GIL is not relevant because Python is so slow that running it in two threads is hardly an improvement.
Re: From Python to Go and Back Again
#65I am surprised the GIL hasn't been mentioned once in the presentation or any of the comments.
Why would it? Using twisted (or py3's asyncio), the event loop is spinning, there is no thread for Python to be switching to. Except that we're using a thread-pool of course (to make external blocking network calls), and those threads are.... making network requests, which is I/O, and the GIL releases during that. On a network bound daemon, which is not CPU-bound, the GIL is really not an issue, so it never came up.
"On a network bound daemon, which is not CPU-bound" - I think that's a fairly unique constraint which makes your case special. For me, the GIL is unfortunately where Python is unable to compete with Go.
Re: From Python to Go and Back Again
#66Most people who rewrote their apps from X to Go and saw improved performance and readability benefited much more from the rewrite than they did from Go. At best, the fact that Go has a relatively weak ecosystem, means that they had to write from scratch a lot of things they were getting for free in X. But, because in X is was a library, they only used 5% of the features, but paid a high performance cost and had a com…
> Most people who rewrote their apps from X to Go and saw improved performance and readability benefited much more from the rewrite than they did from Go Precisely, and isn't this presentation the perfect example of this phenomenon? * Initially we had an implementation in language X * We then rewrote it in language Y - the lessons learned by making the system anew (this time knowing the exact problematic spots, what…
Regardless of how good you are, its easier to write code that has a predictable runtime memory footprint if you can actually predict how things will behave. Predicting an M:N scheduler is.... not easy.
This is part of why Rust removed the M:N scheduler and light-weight threads before Rust 1.0. It's hard to predict your memory use if the run-time is going to be creating/destroying OS threads, and juggling your lightweight threads (goroutines/etc) between real OS threads.
I agree entirely that the next iteration you write to solve the same problem is going to be better than the one before. The problem-scope is well defined, and you're already familiar with it and where the prior implementation was lacking. In this case the extra predictability of knowing what was going to be occurring at once did help.
Re: From Python to Go and Back Again
#67I am surprised the GIL hasn't been mentioned once in the presentation or any of the comments.
node.js has a GIL and nobody talks about that either. The GIL is not relevant because Python is so slow that running it in two threads is hardly an improvement.
But don't you need to run a bunch of node.js instances in production on a multi-cpu system behind nginx or haproxy? Would it be necessary had it not been for the GIL?
Re: From Python to Go and Back Again
#68I appreciate these war stories more than the "look at this great new thing that will take over the world" posts (those have a place as well). We need more war stories in this industry because everything has pros and cons, and our job as software engineers is to be able to make decisions based on limited information. Case studies are great way to glean real-world experience from others without having to implement ever…
Just wanted to share my own very small case study. I had a homework assignment to build a polite crawler. I initially built it in Python, and it was awfully slow. I rewrote same thing in Go, it turned out to very very fast (10x at least IIRC). I liked the fact how quickly I was able to write something so quickly (with not so shabby design) in Go with so much less experience in it. Go is definitely awesome for writing concurrent code quickly. Its not big industry story, but as a busy student I still feel great about using Go. Reason being, we had to use same crawler for doing other stuff, for which a fast crawler was really handy and saved me hours.
The problems I observed with Go was that its regex seemed to be slower than Python, and memory usage was way higher. I explicitly added some GC requests.
Re: From Python to Go and Back Again
#69btw, python with pypy gonna be a bright future.
Re: From Python to Go and Back Again
#70"Go experts still can't write leak-free code"
Big statement, anyone confirm this?