Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Grumpy: Go running Python

opensource.googleblog.com

Post reply on HN
Print view View original

11–20 of 463 posts

Re: Grumpy: Go running Python

#11

Post by ericfrederich »

Twirrim wrote: post #5

This seems like an odd engineering choice. Presumably the effort to create a python->go translator would be non-trivial. Why not just start rewriting components into Go, and migrating them out of Python, leaving python as essentially the presentation layer at most?

It's not just a matter of changing the implementation language from C to Go, they wanted to remove the GIL as well. For that you can't even use the existing CPython design.

Re: Grumpy: Go running Python

#12

Post by aibottle »

This actually pretty great. Even if you will never run your code in grumpy, it underlines the status of Python. Having large corps investing in Python will help pulling in new talent into the Python environment also it strengthens the ecosystem of Python.

Re: Grumpy: Go running Python

#13

Post by MBCook »

brian_herman wrote: post #2

No C extensions boo.

But isn't that really what lets them do this? In following PyPy (and other alternate Python discussions) it seems like eliminating the GIL and getting better performance out of Python, even in C, isn't that hard if you drop the C extensions.

As soon as something can see into the guts of the interpreter you have to maintain compatibility which is a pain/waste.

Worse than that is that view wasn't designed for multi-threading which is why the GIL exists. The C extensions were t designed to be multi-threaded because that wasn't a thing in Python so they're not safe. You either have to drop them, define a new interface layer that would be safe, or I suppose somehow sandbox their little view of the world but keep it coherent between threads.

If you have a codebase where you can make the choice to drop C extensions and you're trying to accelerate Python it seems like a very smart choice.

Re: Grumpy: Go running Python

#14

Post by jerf »

Twirrim wrote: post #5

This seems like an odd engineering choice. Presumably the effort to create a python->go translator would be non-trivial. Why not just start rewriting components into Go, and migrating them out of Python, leaving python as essentially the presentation layer at most?

Ask yourself how much Python code you think Google has. Then multiply it by some large single-digit number, minimum. Then compare the effort of spending 30 seconds per line on that code vs. writing a new Python interpreter/compiler/runtime, especially when you can trivially get people who are capable of doing that.

There's a reason why the large companies often end up working on new runtimes/interpreters/compilers like HipHop for PHP, Hack, and so on, rather than working on the code bases written in those languages. It is very easy for it to not just be easier to leverage in at that level, but an order of magnitude or two easier. Or three.

Re: Grumpy: Go running Python

#15

Post by martincmartin »

ericfrederich wrote: post #11
Twirrim wrote: post #5

This seems like an odd engineering choice. Presumably the effort to create a python->go translator would be non-trivial. Why not just start rewriting components into Go, and migrating them out of Python, leaving python as essentially the presentation layer at most?

It's not just a matter of changing the implementation language from C to Go, they wanted to remove the GIL as well. For that you can't even use the existing CPython design.

I think he's talking about moving their Python codebase to Go, so they don't have to run Python anymore.

Re: Grumpy: Go running Python

#16

Post by MBCook »

matt_wulfeck wrote: post #8
Twirrim wrote: post #5

This seems like an odd engineering choice. Presumably the effort to create a python->go translator would be non-trivial. Why not just start rewriting components into Go, and migrating them out of Python, leaving python as essentially the presentation layer at most?

That's a question I have as well. I'm guessing they have a very massive cpython codebase, and the trade-off was worth it.

I'd assume that's true, but they also didn't know how well this would work. A 'little' project to make Grumpy is probably much easier to sell to management/throw away if to slow than "Let's rewrite all our code in language X to see if it goes faster".

Re: Grumpy: Go running Python

#17

Post by sgift »

naasking wrote: post #4

I wonder why the Grumpy Fibonacci is so much slower than CPython for 1 thread. Seems weird given Grumpy is compiled.

From the post it looks like it is optimized for multithreading from the start, so they possibly used the multithreaded version with thread count = 1. When you care for parallel workloads its not so important if your single-threaded code is the fastest.

Re: Grumpy: Go running Python

#19

Post by MBCook »

scrollaway wrote: post #10

That looks like a super interesting runtime. Seems to target 2.7 only, I hope they're open to supporting 3.x as well.

I wonder if they intend to maintain it long term or if they're just going to use it as a bridge while they rewrite things natively in Go.

Re: Grumpy: Go running Python

#20

Post by ohitsdom »

Very interesting project and technical solution. Is this in use at Google? The described example problem is Youtube with millions of requests per second, but the post doesn't say if it Grumpy was put in production (and what performance gains were then achieved).
Post reply on HN