Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Why we switched from Python to Go

getstream.io

Post reply on HN
Print view View original

351–360 of 406 posts

Re: Why we switched from Python to Go

#351

Post by vorg »

saghm wrote: post #344
oblio wrote: post #158

Earlier quoted context omitted.

I'd say that simplest "production-ready" criterion is this: does it have an officially supported and stable release of a IntelliJ IDE (there is a slight Java-ecosystem bias, but I think it's only slight). So the mainstream languages are...: C/C++, C#, F#, Go, Groovy, Java, JavaScript, TypeScript, Kotlin, Objective-C, PHP, Python, Ruby, Scala, SQL, Swift, VB.NET (source: https://www.jetbrains.com/products.html ). Soun…

Slight nit, but I don't think it's accurate to say that Go has an "officially supported and stable release of an Intellij IDE". While Gogland is excellent (I use it as my main Go editor at work), I think it's still technically in pre-release. On a more unrelated note, is Groovy widely used? As someone who doesn't really ever use JVM languages, my impression was that it didn't have the similar usage to Kotlin and Scal…

Apache Groovy's used for scripting on the JVM, like Bash and Perl on Linux. It's OK for glue code, mock testing, and 10-line build scripts in Gradle. But 5 years ago, some of Groovy's backers tried to re-purpose it as a competitor to statically typed languages like Java, Kotlin, and Scala, which is when things went awry. If it had stuck to its knitting, it could have been used widely for scripting classes written in Kotlin and Scala as well as Java. But it didn't even keep up with Java -- it doesn't even have Java 8's lambdas on the eve of Java 9's release.

Re: Why we switched from Python to Go

#352

Post by jamiesonbecker »

steveklabnik wrote: post #320
_hyn3 wrote: post #273

Earlier quoted context omitted.

For that project, we had very specific requirements: easily handle SSL/TLS with contexts and control over self-signed vs certificate checking, JSON processing, speed, nice syntax, and one of the most challenging requirements: statically compiled, linkable against musl and libressl, while still supporting mingw_64 for windows. Only a few languages have flexible compilers that can do this; for example, rust can't (afai…

Rust can do what you asked, as far as I know.

Excellent-- thank you for the correction!

Re: Why we switched from Python to Go

#353

Post by steveklabnik »

tychver wrote: post #331
hasenj wrote: post #287

Earlier quoted context omitted.

Contrived benchmarks are not useful. Specially when you have tiny datasets. If you want to do anything interesting, Python/Ruby are slow as hell, which is why you cannot do anything interesting in them. For example, while in Go, you can load say 1000 rows from the db and perform some data manipulation on it in the code to get a desired result, you cannot do this in Python because it will very very slow. So what you d…

Simple benchmarks are a useful yardstick. I recently wrote a service in Rust/Iron which only has 4.7x the throughput of the same Ruby/Rails service. That was rather disappointing considering how much more effort is required to do it in a lower level language. Is Python/Django performance significantly worse than Ruby/Rails? The situations you describe are things I do every day in Ruby. Getting 1000 rows from the DB a…

Iron is still using sync IO, and while Ruby isn't great at parallel stuff, it at least does async IO, IIRC. That's going to be a huge difference.

Re: Why we switched from Python to Go

#354

Post by viraptor »

steveklabnik wrote: post #353
tychver wrote: post #331

Earlier quoted context omitted.

Simple benchmarks are a useful yardstick. I recently wrote a service in Rust/Iron which only has 4.7x the throughput of the same Ruby/Rails service. That was rather disappointing considering how much more effort is required to do it in a lower level language. Is Python/Django performance significantly worse than Ruby/Rails? The situations you describe are things I do every day in Ruby. Getting 1000 rows from the DB a…

Iron is still using sync IO, and while Ruby isn't great at parallel stuff, it at least does async IO, IIRC. That's going to be a huge difference.

It depends what you're doing. If you're running a ruby web app server and talking to the database, all the io you're doing is most likely synchronous. In the one-server-per-cpu model, anyway.

Re: Why we switched from Python to Go

#355

Post by steveklabnik »

viraptor wrote: post #354
steveklabnik wrote: post #353

Earlier quoted context omitted.

Iron is still using sync IO, and while Ruby isn't great at parallel stuff, it at least does async IO, IIRC. That's going to be a huge difference.

It depends what you're doing. If you're running a ruby web app server and talking to the database, all the io you're doing is most likely synchronous. In the one-server-per-cpu model, anyway.

It's been a while, but I thought that MRI basically slept during IO and released the GIL so that other threads could do work. In that case, you'd still be handling more requests while the IO was performed. I could be wrong. This kind of thing: http://ablogaboutcode.com/2012/02/06/the-ruby-global-interpr...

Re: Why we switched from Python to Go

#356

Post by viraptor »

hasenj wrote: post #287
tychver wrote: post #251

Earlier quoted context omitted.

Twitter's issue was more with concurrent IO than anything. This problem was solved about 10 years ago with the release of Ruby 1.9 and since then only one Ruby process per core is required, just like NodeJS. In that time most CPU intensive parts of the web stack have also been re-written as native extensions. Recent benchmarks of a properly setup Ruby environment vs Go/Gin are showing Ruby/Sinatra as having 50% of th…

Contrived benchmarks are not useful. Specially when you have tiny datasets. If you want to do anything interesting, Python/Ruby are slow as hell, which is why you cannot do anything interesting in them. For example, while in Go, you can load say 1000 rows from the db and perform some data manipulation on it in the code to get a desired result, you cannot do this in Python because it will very very slow. So what you d…

Every bigger web application will start caching at some point. This is nothing new in either python or ruby. It's even well integrated into SQL access libs: python sqlalchemy http://docs.sqlalchemy.org/en/latest/orm/examples.html#modul... or just using the in memory memcache.

Re: Why we switched from Python to Go

#357

Post by viraptor »

steveklabnik wrote: post #355
viraptor wrote: post #354

Earlier quoted context omitted.

It depends what you're doing. If you're running a ruby web app server and talking to the database, all the io you're doing is most likely synchronous. In the one-server-per-cpu model, anyway.

It's been a while, but I thought that MRI basically slept during IO and released the GIL so that other threads could do work. In that case, you'd still be handling more requests while the IO was performed. I could be wrong. This kind of thing: http://ablogaboutcode.com/2012/02/06/the-ruby-global-interpr...

You're right that it can do async io. But you compared a framework to a language. Rust has Tokio for async - Iron is just not using it.

It's similar with Ruby/RoR - yeah, they can do async. But not on their own. With unicorn server, you still get no threading and just a bunch of processes. With puma you can do threading (async cooperative really) - as long as you keep the configuration/code within the limits of what's allowed.

And due to the extra care needed whenever you do caching/storage things, I expect unicorn is still the king in RoR deployments. (GH uses that for example)

Re: Why we switched from Python to Go

#358

Post by Jach »

slackingoff2017 wrote: post #347
Jach wrote: post #342

Earlier quoted context omitted.

If you start getting into decently designed systems territory, you're still going to have trouble beating some of the stuff that comes out of the Python/Lisp/Node communities. For instance: https://magic.io/blog/asyncpg-1m-rows-from-postgres-to-pytho... Anyway, if you're locally looping for hundreds/thousands of queries to the database, instead of writing one query or calling a stored procedure, you're probably doing…

I'm talking about API calls where you make a few database queries, not local or batched. Java/Go with pretty much any database can manage about 50,000 individual queries+REST calls a second.

Ah I see. I still think if you're measuring that way, there's no reason languages like Python et al. can't accomplish similar too. And if they can't, well, there's always adding more machines. Horizontal scaling tends to occur no matter what you're using, so is the argument just that with less productive (probably a matter of taste for many at this point) languages you'll have to scale out later? That's a tradeoff to consider, but there are other tradeoffs too.

https://www.techempower.com/benchmarks/#section=data-r14&hw=... has some interesting benchmarks, apparently we should be using Dart or C++. Go does slightly better than JS but not a lot. Some newer Python frameworks aren't on there yet. None of them reach near 50k, but I don't know the details of the benchmark and they aren't all using the same DB. Certainly you can get crazy numbers depending on what you're testing. e.g. https://github.com/squeaky-pl/japronto gets one million requests per second through async and pipelining but those requests aren't doing much.

Re: Why we switched from Python to Go

#359

Post by faceplanted »

jokoon wrote: post #277
dundercoder wrote: post #3

Python was my entry into the programming world, and I've been an evangelist ever since... Or I was until I ran into distribution and parallelism. Since then, Nim has been my go-to language of choice. It is all that Python was, plus unbelievable speed, compiling to shippable binaries, and some other cool language features that admittedly, are still beyond my scope of abilities. Still quite lacking in libraries compare…

There is a huge need for python to be compiled or made faster. I wonder if python 4 could make this happen.

Pypy exists.

Re: Why we switched from Python to Go

#360

Post by steveklabnik »

viraptor wrote: post #357
steveklabnik wrote: post #355

Earlier quoted context omitted.

It's been a while, but I thought that MRI basically slept during IO and released the GIL so that other threads could do work. In that case, you'd still be handling more requests while the IO was performed. I could be wrong. This kind of thing: http://ablogaboutcode.com/2012/02/06/the-ruby-global-interpr...

You're right that it can do async io. But you compared a framework to a language. Rust has Tokio for async - Iron is just not using it. It's similar with Ruby/RoR - yeah, they can do async. But not on their own. With unicorn server, you still get no threading and just a bunch of processes. With puma you can do threading (async cooperative really) - as long as you keep the configuration/code within the limits of what'…

Yeah, I guess I moved to Puma long enough ago that I forgot about this. Good points, thank you.
Post reply on HN