Live data from Hacker News

Grumpy: Go running Python

opensource.googleblog.com

381–390 of 463 posts

Re: Grumpy: Go running Python

#381

Earlier quoted context omitted.

It's because Go uses a different stack structure, called "segmented stacks", in order to enable cheap goroutines. Basically, Go stacks start tiny (8 KiB, as opposed to much larger C stacks), then it grows them in small segments. Additionally, Go code runs inside an event loop, which enables excellent I/O performance without kernel context-switches, and ordinary C function calls conflict with this event loop.

Segmented stacks in Go went away in Go 1.3 ( https://golang.org/doc/go1.3#stacks ; June 2014). The alternate stack structure is indeed one issue. The bigger one is the GC, though; the Go runtime needs to know which pointers it is responsible for freeing, and which are the responsibility of the C code.

> The bigger one is the GC; the Go runtime needs to know which pointers it is responsible for freeing

That is not the bigger issue, and AFAIK already handled for C types.

The stack/calling conventions is the reason why cgo is "not go", cgo calls have significantly more overhead than just about every other FFI (the overhead of a cgo call is ~2 orders of magnitude more than a "native" go call, or was around the same time last year, that is you could perform ~100 no-op non-inlined native calls to a do-nothing function by the time you need for a single cgo call to the same).

Re: Grumpy: Go running Python

#382
post #306
post #229

Earlier quoted context omitted.

To me it looks like Google essentially is moving everything to Go, and this project is to help with it. They port their python libraries so they can reference them from their Go code, and then module by module will rewrite it in pure Go.

>Google essentially is moving everything to Go ...[snip]... and then module by module will rewrite it in pure Go. I can see this happening for some core modules sure, but I think you've underestimated the work required to convert the sheer amount of Python code at Google. There is a tool inside Google which graphs the number of lines of each language in Piper; I don't think I can quote numbers from my time there but…

It just needs to happen one piece at a time, even if it takes a few years.

Re: Grumpy: Go running Python

#383

Earlier quoted context omitted.

Is there not a single namedtuple in the entire Google codebase? That's strange :o

Are namedtuples that popular? They always felt awkward to me. If some temp variable with multiple values inside a loop, I either use normal tuple or a dict. If passing data around a dict or a real class. I never got the huge win from namedtuple?

In every introductory python course tuples are presented as just immutable lists. However a "more accurate" way of describing tuples is if you think of them as records with no field names. When you see tuples as records then the fact that are inmutable make sense, since the order and quantity of the items matters (it remains constant). Records usually have field names and here is where namedtuples comes in handy. Also helps to clarify what the tuples wear (see https://youtu.be/wf-BqAjZb8M?t=44m45s), just 2 minutes clip. If you are thinking why don't define a class, I will tell you a couple of reasons:

1) You know before hand that the number of items won't be modified and the order matters since you are handling records. So it is a simple way of accomplishing that constraint.

2) Because they extend tuple they are inmutable too and therefore they don't store attributes per instance __dict__, field names are stored in the class so if you have tons of instances you save a lot of space.

Why creating a class if you just probably need a read-only interaction? But what about if you need some method? Then you can extend your namedtuple class and add the functionality you want. If for example you want to control the values of the fields when you are creating the namedtuple you can create your own namedtuple by overriding __new__. At that point it is worth it to take a look at https://pypi.python.org/pypi/recordclass.

Re: Grumpy: Go running Python

#384
post #251

I can't help but see this balkanization of Python as a sign that the core language is falling apart. How many interpreters are there now? And how many of them have even close to 100% compatibility with Python 2.7 or 3.N? Guido has lost control of the language, but has he's still officially the BDFL there's no real standardization body. His stubborn view on functional mechanisms have held the language back syntactical…

> His stubborn view on functional mechanisms

Reference, for a non-Python dev who hasn't kept up with it?

Re: Grumpy: Go running Python

#385

Earlier quoted context omitted.

Out of curiosity, what C extensions does YouTube use? If this is good enough to run YouTube's python code already it's honestly super impressive. Well done.

To be clear, Grumpy cannot yet run YouTube's Python codebase. There's still a lot of work to do on the standard library. There are a handful of C extensions for JSON, protobufs, etc that YouTube uses, but mostly they're small utility functions written by us to optimize particularly hot code paths.

Will these be rewritten in go then?

Re: Grumpy: Go running Python

#386
post #311

Earlier quoted context omitted.

> A single developer can download 50 deps across his codebase, and update them to later versions 2-3 times a year. As if individual developers are the reason behind the bulk of the downloads. I wonder how many downloads Travis alone counts for? Your hate of Python 3 in every discussion about it is frankly baffling.

> As if individual developers are the reason behind the bulk of the downloads. I wonder how many downloads Travis alone counts for? Travis runs/tests user projects, so there's nothing about it that's especially partial to Python 2 over Python 3. > Your hate of Python 3 in every discussion about it is frankly baffling. Or, you know, my pragmatic assessment of its popularity. That you'd even use the word "hate" (when i…

> even if its mostly tame updates over what 2.7 offers

> The situation is not unlike the perennial "next year is when Linux dominates the desktop", which has been every year since 1999.

Your bias is showing, as it does in every comment section on this site regarding Python 3, as you make comment after comment about how inferior Python 3 is and how nobody is using it at all because your sample of 2 companies shows this and how it personally hurt your family or whatever. You don't stop. Either you hate it or you hate something else and use Python 3 as a vent.

Re: Grumpy: Go running Python

#387

Earlier quoted context omitted.

>- Amusingly, it runs Python 2.7, even though this project started long after Python 3.x came out. Python 2.7 is what's running at Google. Not really surprising they're looking at this considering the fast approaching end of (core dev) support for Python 2.7. Write an interpreter in another language and programatically port modules to Go. Seems pretty sensible to me.

> Python 2.7 is what's running at Google. Not really surprising they're looking at this considering the fast approaching end of (core dev) support for Python 2.7. I'd prefer that all new Python tools that need to support 2.x also support 3.x. It's an additional development cost, but IMHO, a worthwhile investment in the future.

From Google's POV, Go is the future, not Python 3, they created it for this reason. As mainly a sysadmin these days, I tend to agree with them for their use-case. For deployment, performance and overhead, Go is great for system tools, which is reflected in the new toys in the "sysops" toolbox. Pretty much every-one of them is written in Go these days, where that used to be Python or for a brief period Ruby or C/C++ for the more performance sensitive stuff.

Re: Grumpy: Go running Python

#388
post #375
post #358

Earlier quoted context omitted.

Scala, Clojure, F# also allow it, with the added benefit of industrial strength JIT/AOT compilers.

Except without the great Python ecosystem and without great projects like Numpy/Scipy

Numpy/Scipy are only relevant to a minor set of computer users, and even then, there are alternatives like LANPACK and BLAS.

Also Java and .NET ecosystems are just a little bigger than Python.

Re: Grumpy: Go running Python

#389
post #366
post #196

Earlier quoted context omitted.

Highly depends on the use case. Python 3.5 with uvloop+sanic can be faster than node.js without any JIT: https://github.com/channelcat/sanic#benchmarks

Still slower than OCaml, Haskell, Java or .NET.

Mmm, "faster than node.js" isn't a great benchmark out in the wide world. Although node.js being as fast as it is remains an astonishing thing.

Re: Grumpy: Go running Python

#390
post #349

Earlier quoted context omitted.

IIUC, it's not about accessing python libs from go. It's for accessing go libs from your python program and transpiling that python code to go source and compile it with go tool chains. Eg: python code (from blog post) from __go__.net.http import ListenAndServe, RedirectHandler handler = RedirectHandler('http://github.com/google/grumpy', 303) ListenAndServe('127.0.0.1:8080', handler)

sure but the reverse should be equally feasible. It's transpiling Python to Go, so theoretically we should be able to (eventually) "convert" Python libs to Go and call them from Go. A lot of utility libs are available in Python... the Go library ecosystem is relatively sparse

I'm not sure if it is already possible? Is it.
Post reply on HN