Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

From Python to Go and Back Again

docs.google.com

Post reply on HN
Print view View original

41–50 of 167 posts

Re: From Python to Go and Back Again

#41

Post by sethammons »

hvs wrote: post #4

I appreciate these war stories more than the "look at this great new thing that will take over the world" posts (those have a place as well). We need more war stories in this industry because everything has pros and cons, and our job as software engineers is to be able to make decisions based on limited information. Case studies are great way to glean real-world experience from others without having to implement ever…

I should reach out to our team that took python/twisted dealing with sockets and lots of concurrency and ported to Go and see if they would put together a similar presentation. Our case is a bit different, but we saw over 130x improvement in throughput going to Go. While they were in there, they increased monitoring, stability, and maintainability. More case studies to help others make informed choices. Sending that email now :)

[edit: grammar]

Re: From Python to Go and Back Again

#42

Post by windlep »

sethammons wrote: post #41
hvs wrote: post #4

I appreciate these war stories more than the "look at this great new thing that will take over the world" posts (those have a place as well). We need more war stories in this industry because everything has pros and cons, and our job as software engineers is to be able to make decisions based on limited information. Case studies are great way to glean real-world experience from others without having to implement ever…

I should reach out to our team that took python/twisted dealing with sockets and lots of concurrency and ported to Go and see if they would put together a similar presentation. Our case is a bit different, but we saw over 130x improvement in throughput going to Go. While they were in there, they increased monitoring, stability, and maintainability. More case studies to help others make informed choices. Sending that…

I should note, we don't care about throughput for the most part. Our constraint is purely the memory use of holding open the connections. The aim is to hold as many connections as possible within 10-20% of the machines RAM, and not exceed it. As such, we need to be careful about resource usage and spikes.

Goroutines feel cheap, but if you're holding 140k connections, and just 20k of them do something that spins up a goroutine each... you can easily exceed the memory constraint. As such, we had to put goroutine pools in place, careful select statements around them from connections to ensure we didn't overwhelm external resources, etc. It was a huge pain. It has been drastically easier to control resource usage with these constraints under python/twisted.

YMMV, of course, this is just our experience. Part of the reason for putting it out there is that there are already many people who have talked/blogged about going from Python -> Go. I thought maybe the world could handle just one story about going the other direction.

Re: From Python to Go and Back Again

#43

Post by ylh »

FraaJad wrote: post #18
zem wrote: post #10

i've seen a lot of these posts ending along the lines of "it's time for rust". two languages that are always conspicuous by their absence are D and ocaml. D in particular seems like it would be the logical upgrade path from python or ruby. it has a comfortably familiar C lineage, supports a variety of programming paradigms, and has good concurrency support. i wonder why people don't at least give it a look. (personal…

D has replaced Python for my regular programming. D is the sweet spot for Python programmers to upgrade to without going backwards to Go (Programming language design wise) nor weighed down by all the new (and very good) stuff in Rust. D has everything from a nice IDE(Xamarin Studio), debugger, package management (Dub), statically compiled binaries, pretty decent std lib (not as good as python or Go, but very good non…

People saying "backwards to Go" instantly reminds me of the following quote:

Are you quite sure that all those bells and whistles, all those wonderful facilities of your so called powerful programming languages, belong to the solution set rather than the problem set?

        — Edsger W. Dijkstra

Re: From Python to Go and Back Again

#44

Post by eliben »

latch wrote: post #9

Most people who rewrote their apps from X to Go and saw improved performance and readability benefited much more from the rewrite than they did from Go. At best, the fact that Go has a relatively weak ecosystem, means that they had to write from scratch a lot of things they were getting for free in X. But, because in X is was a library, they only used 5% of the features, but paid a high performance cost and had a com…

> Most people who rewrote their apps from X to Go and saw improved performance and readability benefited much more from the rewrite than they did from Go

Precisely, and isn't this presentation the perfect example of this phenomenon?

* Initially we had an implementation in language X

* We then rewrote it in language Y - the lessons learned by making the system anew (this time knowing the exact problematic spots, what really to optimize for, etc) - we got a better system. Long live language Y

* We then rewrote it in language X - the lessons learned [...] - we got a better system. Long live language X

Good programmers can productively write good and fast code in C, Python, Java, Go, or whatever. The skill of the developer and the understanding of the problem matters much more than the programming language

Re: From Python to Go and Back Again

#45

Post by anacrolix »

mratzloff wrote: post #15

There are people who have been working in Go for years, successfully, but who don't post comments with the same frequency and dogged determination as the middlebrow dismissers. Both Python and Go are fine. They both have their strengths and weaknesses. I personally wouldn't write a web app in Go (at least, anything beyond the most basic admin interface). I also personally wouldn't write a very large and complex Pytho…

> You can't write Java enterprise software in Go, and I really appreciate that.

You can, and I've seen it, unfortunately.

Re: From Python to Go and Back Again

#46

Post by FraaJad »

ylh wrote: post #43
FraaJad wrote: post #18

Earlier quoted context omitted.

D has replaced Python for my regular programming. D is the sweet spot for Python programmers to upgrade to without going backwards to Go (Programming language design wise) nor weighed down by all the new (and very good) stuff in Rust. D has everything from a nice IDE(Xamarin Studio), debugger, package management (Dub), statically compiled binaries, pretty decent std lib (not as good as python or Go, but very good non…

People saying "backwards to Go" instantly reminds me of the following quote: Are you quite sure that all those bells and whistles, all those wonderful facilities of your so called powerful programming languages, belong to the solution set rather than the problem set? — Edsger W. Dijkstra

Interestingly, ALGOL failed because it was too complicated to implement.. And Djikstra played a huge role in the formulation of that language.

Why aren't you using ALGOL?

Re: From Python to Go and Back Again

#47

Post by methodover »

windlep wrote: post #8
tinco wrote: post #5

I've been porting a project from Ruby to Go in search of making it a bit lighter. The project is about a 1000 lines now so I'm not allowed to criticize Go yet, but so far it's very nice. I picked the language up in just a few hours as I went and the whole project took just about a week and a half. So as someone who is clearly in no position to be criticizing other projects yet, isn't Heka exactly the sort of project…

For heka, one of the things we wanted besides performance, was a small, easy-to-distribute binary you could drop on a system and 'just run'. Go is fantastic for distribution thanks to the small staticly linked binaries you can get out of it. To accommodate the pluggable filters and such later, a Lua sandbox system was added... and oddly some of the other Go pieces ended up being faster in Lua which is why Lua decoder…

I don't have much to add, other than I went to the talk this past Wednesday (Python meetup in SF) and really enjoyed it. You did a great job. I agree with the currently-most upvoted comment here that I really value these kinds of war stories. Making technology choices is a huge part of what I have to do on a week to week basis and it was great to hear your experience.

Re: From Python to Go and Back Again

#48

Post by mrits »

windlep wrote: post #42
sethammons wrote: post #41

Earlier quoted context omitted.

I should reach out to our team that took python/twisted dealing with sockets and lots of concurrency and ported to Go and see if they would put together a similar presentation. Our case is a bit different, but we saw over 130x improvement in throughput going to Go. While they were in there, they increased monitoring, stability, and maintainability. More case studies to help others make informed choices. Sending that…

I should note, we don't care about throughput for the most part. Our constraint is purely the memory use of holding open the connections. The aim is to hold as many connections as possible within 10-20% of the machines RAM, and not exceed it. As such, we need to be careful about resource usage and spikes. Goroutines feel cheap, but if you're holding 140k connections, and just 20k of them do something that spins up a…

I miss the time when you bragged about increasing performance without resulting in having to switch frameworks or languages.

Re: From Python to Go and Back Again

#49

Post by windlep »

mrits wrote: post #48
windlep wrote: post #42

Earlier quoted context omitted.

I should note, we don't care about throughput for the most part. Our constraint is purely the memory use of holding open the connections. The aim is to hold as many connections as possible within 10-20% of the machines RAM, and not exceed it. As such, we need to be careful about resource usage and spikes. Goroutines feel cheap, but if you're holding 140k connections, and just 20k of them do something that spins up a…

I miss the time when you bragged about increasing performance without resulting in having to switch frameworks or languages.

So do I! Eventually you 'top-out' in a language/framework though... and then its all tears.

Re: From Python to Go and Back Again

#50

Post by themartorana »

latch wrote: post #9

Most people who rewrote their apps from X to Go and saw improved performance and readability benefited much more from the rewrite than they did from Go. At best, the fact that Go has a relatively weak ecosystem, means that they had to write from scratch a lot of things they were getting for free in X. But, because in X is was a library, they only used 5% of the features, but paid a high performance cost and had a com…

This is patently absurd. It's mind boggling that people still hop on here and declare how something is "amazing" or "horrible" for a particular problem-set as fact.

You want my anecdote? Go is brilliant for web services. We've decreased server costs significantly while decreasing response times by orders of magnitude for write-heavy APIs. Concurrency primitives that do bleed into parallelism have made a mockery of interpreted dynamic languages.

But don't believe me. Ask Cloudflare, or Google, or Dropbox, or any other number of companies how horrible Go has been.

But just for shits and grins, I'll bet it'd take me moments to find people in situations where Go didn't meet their needs or domain requirements.

Please stop with the absolutes. They're absolutely ridiculous.

Post reply on HN