Live data from Hacker News

From Python to Go and Back Again

docs.google.com

91–100 of 167 posts

Re: From Python to Go and Back Again

#91

Earlier quoted context omitted.

The difference is that Python 3 is far more backwards incompatible than an update to Ruby, Go or Java. If there's any component of your Python project that was written for Python 2 and it hasn't been ported, your project as a whole is probably staying in Python 2. I certainly wouldn't call the Python 3 path "dead", as starting a new project you'd be silly not to use Python 3. It's just a very slow process. Also, port…

I understand this in its abstract sense - but I think (IMHO) the fundamental problem is that every framework out there is using Python 2.7.... which means nobody is using it. I'm not sure if you have worked in the Ruby or Go ecosystem... but if you are starting a new project, you are using latest Ruby+latest Rails. this is not the case in Python. I asked a question here and on the /r/python forums. What framework sho…

> every framework out there is using Python 2.7....

That's just FUD, it might have been true many years ago but not any longer. http://python3wos.appspot.com/, there are a few red holes but from what I could tell some of those have substitutes that in many cases are better even. It might also be true for in house libraries but at least there you have the chance to upgrade yourself.

Re: From Python to Go and Back Again

#92
post #52

I have to note this, because I think it deserves quite a bit more attention. SSL is extremely expensive, on RAM (perhaps the implementations have optimized for throughput over RAM). I have yet to benchmark any SSL implementation in any language, with any binding, that can use less than 20kb per SSL connection. I mentioned in my talk here that SSL is very expensive, here's my benchmark suite that others may add to: ht…

Do you have any insight about the mention in the slides of Google having a 10kb solution?

Re: From Python to Go and Back Again

#93
post #28

Earlier quoted context omitted.

That weird split is how most languages with modules work.

In languages that don't fully parse the dependencies (which Go does), that split uncouples "the implementation needs recompiling" from "all the dependencies need recompiling."

Not all languages are C and C++.

That split doesn't forbid reading the module metada.

Re: From Python to Go and Back Again

#94
post #91

Earlier quoted context omitted.

I understand this in its abstract sense - but I think (IMHO) the fundamental problem is that every framework out there is using Python 2.7.... which means nobody is using it. I'm not sure if you have worked in the Ruby or Go ecosystem... but if you are starting a new project, you are using latest Ruby+latest Rails. this is not the case in Python. I asked a question here and on the /r/python forums. What framework sho…

> every framework out there is using Python 2.7.... That's just FUD, it might have been true many years ago but not any longer. http://python3wos.appspot.com/ , there are a few red holes but from what I could tell some of those have substitutes that in many cases are better even. It might also be true for in house libraries but at least there you have the chance to upgrade yourself.

I know that. But I'm hard pressed to find anyone who is using this stuff in production.

the ecosystem is still running 2.7. I cant wait to switch to Py3.

Re: From Python to Go and Back Again

#95

Earlier quoted context omitted.

Just wanted to share my own very small case study. I had a homework assignment to build a polite crawler. I initially built it in Python, and it was awfully slow. I rewrote same thing in Go, it turned out to very very fast (10x at least IIRC). I liked the fact how quickly I was able to write something so quickly (with not so shabby design) in Go with so much less experience in it. Go is definitely awesome for writing…

Did you try pypy before rewriting?

Didn't solve my problem. PyPy still retains multi-threading and GIL stuff if I remember correctly.

Re: From Python to Go and Back Again

#97
post #52

I have to note this, because I think it deserves quite a bit more attention. SSL is extremely expensive, on RAM (perhaps the implementations have optimized for throughput over RAM). I have yet to benchmark any SSL implementation in any language, with any binding, that can use less than 20kb per SSL connection. I mentioned in my talk here that SSL is very expensive, here's my benchmark suite that others may add to: ht…

Doesn't a TLS terminator proxy solve this? E.g. I usually put my application services behind HTTPS-enabled nginx and it works wonderfully.

Re: From Python to Go and Back Again

#98
post #42

Earlier quoted context omitted.

I should reach out to our team that took python/twisted dealing with sockets and lots of concurrency and ported to Go and see if they would put together a similar presentation. Our case is a bit different, but we saw over 130x improvement in throughput going to Go. While they were in there, they increased monitoring, stability, and maintainability. More case studies to help others make informed choices. Sending that…

I should note, we don't care about throughput for the most part. Our constraint is purely the memory use of holding open the connections. The aim is to hold as many connections as possible within 10-20% of the machines RAM, and not exceed it. As such, we need to be careful about resource usage and spikes. Goroutines feel cheap, but if you're holding 140k connections, and just 20k of them do something that spins up a…

Typically if you wish to limit the number of goroutines you would spawn N workers and have them read from a single channel. If 20k of your incoming connections want to do something they send on the channel, without spawning a goroutine themselves.

Did you try something like that?

Re: From Python to Go and Back Again

#99
post #9

Most people who rewrote their apps from X to Go and saw improved performance and readability benefited much more from the rewrite than they did from Go. At best, the fact that Go has a relatively weak ecosystem, means that they had to write from scratch a lot of things they were getting for free in X. But, because in X is was a library, they only used 5% of the features, but paid a high performance cost and had a com…

While there is a nugget of truth in your statements, the generalisations are so excessive that the overall point is completely wrong.

Lets take a few statements:

> Most people who rewrote their apps from X to Go and saw improved performance and readability benefited much more from the rewrite than they did from Go.

Yes, the rewrite would certainly have helped. However some compilers and runtimes are just faster than others. You wouldn't say that rewriting C code in Perl would result in faster code would you? Of course not. But that's the implication of your post. Go does outperform some languages. Granted it's still slower than some others out there, but not all languages are equal in terms of performance so it's pretty naive to imply they are.

> At best, the fact that Go has a relatively weak ecosystem, means that they had to write from scratch a lot of things they were getting for free in X. But, because in X is was a library, they only used 5% of the features, but paid a high performance cost and had a complex API to work with.

That's completely rubbish. Go is a young language, that much is true. But it's ecosystem is actually very impressive given it's age. Since you're talking about web development, lets look at all the libraries you might want:

  x SQL / no-SQL databases:    check;
  x compression:               check;
  x hashing / encryption:      check;
  x image manipulation:        check;
  x monitoring (eg New Relic): check;
  x httpd frameworks:          check;
  x html templating:           check;
  x smtp (sending e-mails):    check;
  x JSON / XML:                check;
  x web sockets:               check;
I've no doubt missed some stuff that you probably use, but that's not a reflection of stuff missing from Go either. The fact is, in all practicality, there isn't much you need to rewrite in Go aside the web application itself.

> Go's a good language for some things. But it does nothing special or significant to close the massive productivity gap between dynamic and static language. Yes, it's terse compared to many other static language and it has stuff like implicit interfaces, but those are superficial (but nice) things when it comes to what and how you do things in dynamic land.

This sounds more like a rant about how much you hate statically typed languages than it does about how poor Go is compared to [insert preferred language]. For what it's worth, statically typed languages do also offer some productivity bonuses over dynamically typed languages: a big one being the fussier compiler / runtime checking can pick up subtle bugs (eg 0 / "0" / false) that might otherwise take a little while to trace. And I am aware that some dynamic languages have typed checking operators, eg ===, but you can't always guarantee what you're libraries are going to handle / return so you're still sometimes left tracing values up the code base to find the origin of the problem rather than having the compiler explicitly tell you at the first point where your uncleansed data arrives.

So there are also advantages with going down the statically typed route.

> But for a traditional web app / service? It's horrible. At least as horrible as most static languages.

Sorry, but now you're just descending into unashamed language bigotry. A large proportion of the worlds cloud services are supported by statically typed platforms such as ASP.NET, Java, and Go. This forum, HN, is written in Haskell, which is also statically typed. Saying "most static languages [are] cumbersome, error prone, slow, inflexible and difficult to test." is so far off the mark that it's just plain ignorant.

Which is a real pity as there would have been a few good points raised in your rant if you hadn't have jumped off in the deep end with your ridiculous generalisations.

Re: From Python to Go and Back Again

#100

Earlier quoted context omitted.

Did you try pypy before rewriting?

Didn't solve my problem. PyPy still retains multi-threading and GIL stuff if I remember correctly.

Did you try requests_futures lib? It's all about async and network speed with crawling. Not so much cpu.
Post reply on HN