Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Grumpy: Go running Python

opensource.googleblog.com

Post reply on HN
Print view View original

61–70 of 463 posts

Re: Grumpy: Go running Python

#61

Post by trotterdylan »

Animats wrote: post #48

- Amusingly, it runs Python 2.7, even though this project started long after Python 3.x came out. - It's a hard-code compiler, not an interpreter written in Go. That implies some restrictions, but the documentation doesn't say much about what they are. PyPy jumps through hoops to make all of Python's self modification at run-time features work, complicating PyPy enormously. Nobody uses that stuff in production code,…

> Amusingly, it runs Python 2.7, even though this project started long after Python 3.x came out.

Basically, we needed to support a large existing Python 2.7 codebase. See discussion here: https://github.com/google/grumpy/issues/1

> It's a hard-code compiler, not an interpreter written in Go. That implies some restrictions, but the documentation doesn't say much about what they are. PyPy jumps through hoops to make all of Python's self modification at run-time features work, complicating PyPy enormously. Nobody uses that stuff in production code, and Google apparently dumped it.

There are restrictions. I'll update the README to make note of them. Basically, exec and eval don't work. Since we don't use those in production code at Google, this seemed acceptable.

> If Grumpy doesn't have a Global Interpreter Lock, it must have lower-level locking. Does every built-in data structure have a lock, or does the compiler have enough smarts to figure out what's shared across thread boundaries, or what?

It does fine grained locking. Mutable data structures like lists and dicts do their own locking. Incidentally, this is one reason why supporting C extensions would be complicated.

Re: Grumpy: Go running Python

#62

Post by ericfrederich »

epanastasi wrote: post #33

As someone who works on both python and go day to day, I find this to be quite interesting. Just tried this out on a reasonably complex project to see what it outputs. Looks like it only handles individual files and not any python imports in those files. So for now you have to manually convert each file in the project and put them into the correct location within the build/src/grumpy/lib directory to get your depende…

I hope this is a well thought out solution that can evolve into something great... and not just something built for a single purpose.

I question the transpiler. I think I'd much rather prefer a solution like Jython.

Re: Grumpy: Go running Python

#63

Post by falcolas »

jerf wrote: post #14
Twirrim wrote: post #5

This seems like an odd engineering choice. Presumably the effort to create a python->go translator would be non-trivial. Why not just start rewriting components into Go, and migrating them out of Python, leaving python as essentially the presentation layer at most?

Ask yourself how much Python code you think Google has. Then multiply it by some large single-digit number, minimum. Then compare the effort of spending 30 seconds per line on that code vs. writing a new Python interpreter/compiler/runtime, especially when you can trivially get people who are capable of doing that. There's a reason why the large companies often end up working on new runtimes/interpreters/compilers li…

I wouldn't think that a tool for transpiling Python to Go would be harder than creating a whole new runtime. Most of the work is already done; you can get the compiled ASTs directly out of Python itself.

Of course, if it started out as a "this seems like a cool project", that skews the "a is more efficient than b" ratio significantly.

Re: Grumpy: Go running Python

#64

Post by joshuata »

Animats wrote: post #48

- Amusingly, it runs Python 2.7, even though this project started long after Python 3.x came out. - It's a hard-code compiler, not an interpreter written in Go. That implies some restrictions, but the documentation doesn't say much about what they are. PyPy jumps through hoops to make all of Python's self modification at run-time features work, complicating PyPy enormously. Nobody uses that stuff in production code,…

In regards to the third point: The global interpreter lock protects the fact that python's GC scheme is not thread safe. It does not coordinate accesses across threads, and therefore grumpy's would not either. In grumpy the GIL is replaced by Go's GC implementation that is specifically tuned for multithreaded execution. Any additional synchronization would need to be done with individual locks etc...

Re: Grumpy: Go running Python

#65

Post by trotterdylan »

ot wrote: post #60

There's no mention of whether Grumpy passes the CPython test suite. Until it doesn't, it's not a Python runtime, it's a compiler for a language with Python-like syntax. Compilers like this, from almost-Python to say C/C++, have existed for a while: Cython, Shedskin, Nuitka are some examples.

It does not, yet. The standard library is very incomplete at this point.

Re: Grumpy: Go running Python

#66

Post by TillE »

Animats wrote: post #48

- Amusingly, it runs Python 2.7, even though this project started long after Python 3.x came out. - It's a hard-code compiler, not an interpreter written in Go. That implies some restrictions, but the documentation doesn't say much about what they are. PyPy jumps through hoops to make all of Python's self modification at run-time features work, complicating PyPy enormously. Nobody uses that stuff in production code,…

> Nobody uses that stuff in production code

Nobody uses the features of Python which make it a dynamic language? Google must write some really weird Python if their compiler is that strict.

Re: Grumpy: Go running Python

#67

Post by gigatexal »

on a related note: given my affection for python and having, until just now, no idea that YouTube runs python (albeit 2.7) at Google scale (a language not known for being able to scale very well given the GIL and such)-- I now more than ever would love to work on something like that. Welp, back to learning and hacking with python.

Re: Grumpy: Go running Python

#68

Post by jerf »

the_duke wrote: post #29

Haha, now that's what I call an interesting project. The biggest surprise for me is that the Go runtime would be a good fit for Python, performance wise, considering the very different object and dispatch model. The post also mentions runtime reflection, which used to be painfully slow last time I used it. (Go 1.5, i think). Has this improved in the latest releases?

A huge part of the reason Python (and Perl, PHP, etc. in their original interpreters) is so slow is that it is like running a Go program that does everything through runtime reflection, even just to add two integers. If your code is already in Python, this level of performance is apparently not a problem for you.

If you know Go or are willing to learn about Go and reflection, you can learn a lot about how dynamic languages work under the hood by implementing:

    func Add(interface{}, interface{}) interface{} { ... }
using the reflect module to accept all types of numbers, including for a bit of extra fun the math.Big* number types, and returning upgraded numbers as appropriate, or panicking on types you can't Add with. That's not all there is to writing a dynamic language interpreter, but I'd say you can learn the core idea this way, shorn away from a lot of accidental complexity and with a lot of the grunt work plumbing of setting up (type, value) pairs already done for you.

Re: Grumpy: Go running Python

#69

Post by AnkhMorporkian »

trotterdylan wrote: post #61
Animats wrote: post #48

- Amusingly, it runs Python 2.7, even though this project started long after Python 3.x came out. - It's a hard-code compiler, not an interpreter written in Go. That implies some restrictions, but the documentation doesn't say much about what they are. PyPy jumps through hoops to make all of Python's self modification at run-time features work, complicating PyPy enormously. Nobody uses that stuff in production code,…

> Amusingly, it runs Python 2.7, even though this project started long after Python 3.x came out. Basically, we needed to support a large existing Python 2.7 codebase. See discussion here: https://github.com/google/grumpy/issues/1 > It's a hard-code compiler, not an interpreter written in Go. That implies some restrictions, but the documentation doesn't say much about what they are. PyPy jumps through hoops to make a…

> There are restrictions. I'll update the README to make note of them. Basically, exec and eval don't work. Since we don't use those in production code at Google, this seemed acceptable.

I'm guessing pretty much the entire AST module is a no-go?

Re: Grumpy: Go running Python

#70

Post by zardeh »

thedjinn wrote: post #47

I'm curious to why they started this project when there already is low hanging fruit that can speed up Python (e.g. pypy). What makes this better other than to satisfy the inner go-fanboy?

Pypy still has a GIL. GIL-free seems to be one of the explicit goals of this project.
Post reply on HN