Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Grumpy: Go running Python

opensource.googleblog.com

Post reply on HN
Print view View original

181–190 of 463 posts

Re: Grumpy: Go running Python

#181

Post by pjmlp »

jszymborski wrote: post #139
pjmlp wrote: post #114

Earlier quoted context omitted.

I never saw Python as anything more than a scripting language to portably automate tasks across UNIX and Windows environment, even back in the Zope days. My experience with Tcl teached me to stay away from languages that don't have either a JIT or AOT compiler on their reference implementation.

I think that that's rather dismissive of a language that runs huge web, scientific, and general purpose applications daily.

It might be, but that it is usually a consequence of not knowing any better or making use of existing libraries.

Just like people learned 8-bit BASIC and went on to do business applications and games on it. I went Z80 ASM instead.

Personally I would only use Python for shell scripting and advise for using Julia instead.

Of course, others see it differently.

Re: Grumpy: Go running Python

#182

Post by dochtman »

curtis wrote: post #171
coldtea wrote: post #134

Earlier quoted context omitted.

> One of my complaints about Go is that it seems to be designed with the (mistaken, in my opinion) notion that what we really need is just a better C. It was designed with the notion that all some people really need is just a better C. For other people there's Swift, Rust, etc.

For better or worse, I think Go sucked up a lot of the metaphorical oxygen for similar languages. Swift and Rust are in different niches and don't compete head to head with Go, so this isn't a real problem for them. What I've long wanted is a better C++ , not a better C. It would be hard for any such language to succeed today, because it will have to compete with Go, and Go has a fairly mature toolchain and widesprea…

It seems likely to me that Rust is your better C++. I'm also quite positive that there is (more than) enough oxygen for Rust to succeed.

As for D, I think it has numerous other issues, that don't apply to Rust.

Re: Grumpy: Go running Python

#183

Post by weberc2 »

bb88 wrote: post #123
Floegipoky wrote: post #102

Earlier quoted context omitted.

I'm confused because Jython runs on the JVM, but Go is a compiled language. Can you clarify?

Jython is a python interpreter written in Java. Grumpy is a python transpiler that converts python to navtive go object code. Edited to add: The difference is that Jython doesn't covert python to JVM bytecode.

What's the advantage of writing an interpreter? Go already has an excellent runtime (scheduler, GC, etc)--why should this project reimplement it?

Re: Grumpy: Go running Python

#184

Post by Sphax »

Antwan wrote: post #153

"So we asked ourselves a crazy question: What if we were to implement an alternative ?" As always with Google... What if you humbly contribute to open source projects rather than creating new stuffs, labelled with your own brand, controlled by your own engineers, and with your own design choices, however good they may be ?

How entitled can you be ? Why are they required to do anything of the sort ?

Re: Grumpy: Go running Python

#185

Post by trotterdylan »

agentgt wrote: post #136

* I would love to know how big the codebase is. * It seems like writing a translator to deal with all the use cases is so much more work and risky than iteratively rewriting portions (in whatever faster more concurrent language) and using some form microservice/process message passing to communicate with legacy pieces. * Love to know how they compose async operations currently? Is it some sort of object (e.g. Futures…

> I would love to know how big the codebase is.

Sorry, can't be very specific, but rewriting all the frontend code would take a lot more effort than writing a new Python runtime :)

> It seems like writing a translator to deal with all the use cases is so much more work and risky than iteratively rewriting portions (in whatever faster more concurrent language) and using some form microservice/process message passing to communicate with legacy pieces.

We do iteratively rewrite components as well. We are pursuing multiple strategies.

> Love to know how they compose async operations currently? Is it some sort of object (e.g. Futures, promises, observables, etc)?

Most async operations are performed out-of-process by other servers.

> Is Grumpy going to have some sort of language difference (to Python) to compose async stuff (e.g. async and await)?

I'd love to support async and await at some point.

> Of course being biased towards the JVM (since I know it so well) they could get really fast concurrency if they want with Jython today. Most of the Python tools already work with Jython (assuming 2.7).

We did also do an evaluation of Jython but there were a number of technical issues that made it unsuitable for our codebase and workload. One such example is this longstanding issue: http://bugs.jython.org/issue527524. I just noticed the very recent update on that thread that implemented the workaround outlined in 2010 by Jim Baker. We tried that workaround and found we got a huge performance hit on affected code. There were a few other general performance problems as well but I can't recall all the details.

Please note I'm not at all bashing Jython, I think it's a great project with a sound design, it just wasn't right for us.

> With Jython you could always drop down into Java (or any other JVM lang) if you need more speed as well C for cpython (or even C from Java). It is unclear what you do with Grumpy with performance critical code. Can you interface with Go code or is the plan C?

You can interface with Go code directly, e.g. from the blog post:

  from __go__.net.http import ListenAndServe, RedirectHandler
  handler = RedirectHandler('http://github.com/google/grumpy', 303)
  ListenAndServe('127.0.0.1:8080', handler)

Re: Grumpy: Go running Python

#186

Post by justinsaccount »

jonasvp wrote: post #176
brianwawok wrote: post #163

Earlier quoted context omitted.

Are namedtuples that popular? They always felt awkward to me. If some temp variable with multiple values inside a loop, I either use normal tuple or a dict. If passing data around a dict or a real class. I never got the huge win from namedtuple?

Attrs ( https://attrs.readthedocs.io/ ) replaced namedtuple for us (and many others). It's slightly more verbose but allows all class goodness such as methods, attribute validation, etc.

Doesn't work for everything, but you can subclass a namedtuple:

  from collections import namedtuple
  
  class Foo(namedtuple("Foo", "a b c")):
      @property
      def sum(self):
          return self.a + self.b + self.c
  
  
  f = Foo(1,2,3)
  print f.sum

Re: Grumpy: Go running Python

#187

Post by Sphax »

thedjinn wrote: post #47

I'm curious to why they started this project when there already is low hanging fruit that can speed up Python (e.g. pypy). What makes this better other than to satisfy the inner go-fanboy?

It's right there in the post:

> To solve this problem, we investigated a number of other Python runtimes. Each had trade-offs and none solved the concurrency problem without introducing other issues.

Re: Grumpy: Go running Python

#188

Post by vorg »

the_duke wrote: post #31
oblio wrote: post #6

It seems interesting but my concern is that it's just another Unladen Swallow.

Well, if Google is pushing it and actually using it internally... They also have the benefit of being able to push features into the Go core that they might need for this.

Perhaps Golang will get a VM or something similar to enable stuff like exec/eval to be implemented with rapid response times in Grumpy. The only way I've ever seen a REPL implemented for Go is by recompiling the source code via a call to "os/exec".Command for each statement or declaration entered, which gives a 1-second delay on many computers.

Re: Grumpy: Go running Python

#189

Post by moosingin3space »

vram22 wrote: post #111
moosingin3space wrote: post #92

Earlier quoted context omitted.

It's because Go uses a different stack structure, called "segmented stacks", in order to enable cheap goroutines. Basically, Go stacks start tiny (8 KiB, as opposed to much larger C stacks), then it grows them in small segments. Additionally, Go code runs inside an event loop, which enables excellent I/O performance without kernel context-switches, and ordinary C function calls conflict with this event loop.

>Additionally, Go code runs inside an event loop, which enables excellent I/O performance without kernel context-switches Interesting, didn't know this (that Go code runs in an event loop). Is the reason something to do with goroutines and channels? something like, a routine gets info that data is available for it to read (on a channel, sent by another goroutine), via an event it receives? Also, can you explain this…

One of the reasons an asynchronous I/O event loop can be faster than a threaded model is that the CPU spends more of its time in a single userspace thread per core, switching between clients that are ready. A threaded server will incur a kernel-space context switch each time, while an asynchronous loop will keep the processing time in userspace.

Re: Grumpy: Go running Python

#190

Post by trotterdylan »

driusan wrote: post #138
trotterdylan wrote: post #84

Earlier quoted context omitted.

To be clear, Grumpy cannot yet run YouTube's Python codebase. There's still a lot of work to do on the standard library. There are a handful of C extensions for JSON, protobufs, etc that YouTube uses, but mostly they're small utility functions written by us to optimize particularly hot code paths.

Is the plan to use grumpc to transpile the code to Go and then work in Go in the future, or is the plan to keep coding in Python and add a grumpy step before deploying? (If the former, you could just update the code to use the standard Go JSON/etc packages..)

The idea is to continue to write code in Python. The transpiled code is not suitable for working with directly. That said, there is the possibility of rewriting bits and pieces in Go (e.g. performance critical stuff) and then call into it from Python. Sort of a hybrid approach.
Post reply on HN