Grumpy: Go running Python • Hacker News Board

Re: Grumpy: Go running Python

#91

Post by quotemstr » Wed, Jan 04, 2017, 6:22 PM UTC

poooogles wrote: post #59

Animats wrote: post #48

- Amusingly, it runs Python 2.7, even though this project started long after Python 3.x came out. - It's a hard-code compiler, not an interpreter written in Go. That implies some restrictions, but the documentation doesn't say much about what they are. PyPy jumps through hoops to make all of Python's self modification at run-time features work, complicating PyPy enormously. Nobody uses that stuff in production code,…

>- Amusingly, it runs Python 2.7, even though this project started long after Python 3.x came out. Python 2.7 is what's running at Google. Not really surprising they're looking at this considering the fast approaching end of (core dev) support for Python 2.7. Write an interpreter in another language and programatically port modules to Go. Seems pretty sensible to me.

> Python 2.7 is what's running at Google. Not really surprising they're looking at this considering the fast approaching end of (core dev) support for Python 2.7.

I'd prefer that all new Python tools that need to support 2.x also support 3.x. It's an additional development cost, but IMHO, a worthwhile investment in the future.

Re: Grumpy: Go running Python

#92

Post by moosingin3space » Wed, Jan 04, 2017, 6:22 PM UTC

vram22 wrote: post #51

sfifs wrote: post #27

The main reason I moved from coding in Python to Go as my main language many years back is because concurrency was such a pain in standard Python (the other was compile time error checking). It's interesting to see the same pain has now made caused the runtime itself to be implemented in Go. It's a pity C extensions (often used in scientific computing) are not supported but Go does have support via CGO, so maybe some…

Does anyone know the technical reasons why C extensions are not (at least easily, apparently) supported by Go? Is it to do with Go's being a GC'd language? I would have thought that should not be a reason per se, since Python also has GC, but has plenty of extensions written in C. But I'm not a language internals expert. Also, further signs that GC may not be the reason, is that D also has GC, but can link to C libra…

It's because Go uses a different stack structure, called "segmented stacks", in order to enable cheap goroutines. Basically, Go stacks start tiny (8 KiB, as opposed to much larger C stacks), then it grows them in small segments. Additionally, Go code runs inside an event loop, which enables excellent I/O performance without kernel context-switches, and ordinary C function calls conflict with this event loop.

Re: Grumpy: Go running Python

#93

Post by ProAm » Wed, Jan 04, 2017, 6:25 PM UTC

the_duke wrote: post #31

oblio wrote: post #6

It seems interesting but my concern is that it's just another Unladen Swallow.

Well, if Google is pushing it and actually using it internally... They also have the benefit of being able to push features into the Go core that they might need for this.

> Well, if Google is pushing it and actually using it internally...

It will be silently abandoned in 18 months....

Re: Grumpy: Go running Python

#94

Post by ericfrederich » Wed, Jan 04, 2017, 6:25 PM UTC

martincmartin wrote: post #15

ericfrederich wrote: post #11

Earlier quoted context omitted.

It's not just a matter of changing the implementation language from C to Go, they wanted to remove the GIL as well. For that you can't even use the existing CPython design.

I think he's talking about moving their Python codebase to Go, so they don't have to run Python anymore.

You're probably right. He seemed to be questioning the translator so I thought he was suggesting replacing CPython piece by piece with a Go implementation.

Re: Grumpy: Go running Python

#95

Post by theatrus2 » Wed, Jan 04, 2017, 6:25 PM UTC

vram22 wrote: post #51

sfifs wrote: post #27

The main reason I moved from coding in Python to Go as my main language many years back is because concurrency was such a pain in standard Python (the other was compile time error checking). It's interesting to see the same pain has now made caused the runtime itself to be implemented in Go. It's a pity C extensions (often used in scientific computing) are not supported but Go does have support via CGO, so maybe some…

Does anyone know the technical reasons why C extensions are not (at least easily, apparently) supported by Go? Is it to do with Go's being a GC'd language? I would have thought that should not be a reason per se, since Python also has GC, but has plenty of extensions written in C. But I'm not a language internals expert. Also, further signs that GC may not be the reason, is that D also has GC, but can link to C libra…

Go has cgo, which works fine for most purposes of fine; native code interop is not an issue for Go.

Grumpy likely doesn't support the C extensions due to time, and complexity of having to actually emulate the GIL since Python does not have fine grained locking for structures. C extensions that work with Python data structures need to first hold the GIL.

Re: Grumpy: Go running Python

#96

Post by kodablah » Wed, Jan 04, 2017, 6:25 PM UTC

stiff wrote: post #78

It's actually a transpiler written in Python that generates Go code.

Nice. I've been toying with doing similar for the JVM [0]. I think people are coming to realize that while Go is not a great/powerful language (IMO), the runtime is great. I would not be surprised if more and more people start targetting Go if they want a GC and cross platform static compilation w/out LLVM complications and get a nice stdlib for free.

0 - https://github.com/cretz/goahead

Re: Grumpy: Go running Python

#97

Post by BuckRogers » Wed, Jan 04, 2017, 6:27 PM UTC

Is this the final nail in the coffin for Python3? Seems like it.

Who would use 3 if you could have CPython2 for existing code and write new code in Grumpy Python? This is the dream language for me. Python on the Go runtime.

Re: Grumpy: Go running Python

#98

Post by trotterdylan » Wed, Jan 04, 2017, 6:27 PM UTC

Twirrim wrote: post #5

This seems like an odd engineering choice. Presumably the effort to create a python->go translator would be non-trivial. Why not just start rewriting components into Go, and migrating them out of Python, leaving python as essentially the presentation layer at most?

It's been said a couple times in this thread already but basically, the YouTube Python codebase is very large and the cost of a rewrite is prohibitive.

Re: Grumpy: Go running Python

#99

Post by ericfrederich » Wed, Jan 04, 2017, 6:27 PM UTC

dijit wrote: post #37

ericfrederich wrote: post #25

Python needs a new runtime. This talk shows how bad of shape it's really in. https://www.youtube.com/watch?v=qCGofLIzX6g&list=PLRdS-n5seL... Basically, the language doesn't have a "spec" per-se. The language is whatever the defacto CPython implementation happens to do within it's giant eval loop. Another great talk about CPython internals: http://pgbovine.net/cpython-internals.htm

cpython is the reference implementation, so it makes sense that it's; A) Not well optimised. B) Touting features before the spec/standard. EDIT: people really dislike that I said this, and I'm having trouble finding my original citation- it was on one of the many python books I own. Most likely "Learn Python The Hard Way" but I'll dig out the exact chapter where they compare pypy to cpython and mention that because c…

Please watch that first video, it's a good one. It explains how CPython essentially _is_ the spec because its internals leak into the spec when they have no business being there.

Re: Grumpy: Go running Python

#100

Post by lelandbatey » Wed, Jan 04, 2017, 6:27 PM UTC

sfifs wrote: post #27

The main reason I moved from coding in Python to Go as my main language many years back is because concurrency was such a pain in standard Python (the other was compile time error checking). It's interesting to see the same pain has now made caused the runtime itself to be implemented in Go. It's a pity C extensions (often used in scientific computing) are not supported but Go does have support via CGO, so maybe some…

I "grew up" on Python, then wrote a whole bunch of Go for my job. Then this past Autumn re-visited Python to implement a networked terminal based game[1].

With what I learned about Go and concurrency, I would say that currently in Python, writing concurrent code is not very hard, and is as close to Go as you can get without actually just writing Go.

Now, you may be saying "but Python has the GIL, how can concurrency be easy in Python?" I'd say, you're definitely not wrong that the GIL is a problem, but it's not much of a problem for concurrency.

This goes back to the heart of Rob Pike's classic talk, "Concurrency Is Not Parallelism"[2]. To quote Wikipedia:

        In computer science, concurrency is the decomposability property of a
        program, algorithm, or problem into order-independent or partially-ordered
        components or units.

In Python, you can pretty easily emulate the conceptual properties of Goroutines and Go channels with Python threads and queues. The problem is that doing this in Python won't net you the performance increases you get with Go. And I believe that is an important distinction. There are plenty of cases where you don't care so much about the performance benefits of parallelism, but you want the conceptual and implementation benefits of concurrency.

In closing, concurrency in Python is pretty easy to work with, it just performs very poorly.

[1] - https://github.com/lelandbatey/defuse_division

[2] - https://blog.golang.org/concurrency-is-not-parallelism