Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Codon: A high-performance Python-like compiler using LLVM

github.com

Post reply on HN
Print view View original

141–150 of 184 posts

Re: Codon: A high-performance Python-like compiler using LLVM

#141

Post by dingdingdang »

slt2021 wrote: post #42

power of Python is in ecosystem of libraries, not only Python syntax. Without the ecosystem of libraries, I am afraid use cases for Codon will be very very limited. Because Python developers (just like Node) got used to thinking: Need to do X? Lets see if I can pip install library that does it. Ultimately, python is like super flexible glue between ecosystem of libraries that lets anyone build and prototype high qual…

Perhaps the way forward for Codon in terms of wider adoption would be maintaining a list of libraries that are fully Codon compatible, thereby encouraging devs to aim for cross-compatibility (which would likely naturally exclude usage of a lot of the slower Python features in turn making the libraries faster for both Codon and regular Python users)

Re: Codon: A high-performance Python-like compiler using LLVM

#142

Post by bb88 »

mytherin wrote: post #117
jnxx wrote: post #91

Just out of curiosity: Why is it possible to compile Common Lisp Code (or Scheme, or Clojure) to high-performance native or jit-compiled code, but not Python? It is said that "Python is too dynamic", but is not everything in Lisp dynamic, too? And none of these languages is less powerful than Lisp, lack Unicode support, or whatever, so this can't be the reason.

It is possible to JIT compile Python just fine. There are projects like PyPy that have been doing this for a long time [1]. The reason these alternative projects never take off is because many of Python's most used libraries are written against CPython's C API. This API is giant, and exposes all of the nitty gritty implementation details of the CPython interpreter. As a result, changing anything significant about the…

To be fair the 2 -> 3 upgrade path was terrible. And there wasn't a killer feature in 3 which was terrible. And the tooling around the upgrade was terrible. Basically the python devs completely botched it -- which was terrible.

So one thing of golang that is nice is that go 1.19 compiler will compile go 1.1 just fine, and people can iterate from 1.1 to 1.19 in their own time -- or not if they choose not to. It would not be that hard for golang v2 to continue to allow similar compilation of old code.

Re: Codon: A high-performance Python-like compiler using LLVM

#143

Post by memco »

arshajii wrote: post #73
memco wrote: post #50

Earlier quoted context omitted.

Don't have anything significant, but giving this a quick test with some of my advent of code solutions I found it to be quite a bit slower: time python day_2.py ________________________________________________________ Executed in 57.25 millis fish external usr time 25.02 millis 52.00 micros 24.97 millis sys time 25.01 millis 601.00 micros 24.41 millis time codon run -release day_2.py _________________________________…

I would guess the bulk of the time is being spent in compilation. You might try "codon build -release day_2.py" then "time ./day_2" to measure just runtime.

Good catch! Here's updated runs:

   time python day_2.py

   ________________________________________________________
   Executed in   51.26 millis    fish           external
      usr time   23.38 millis   48.00 micros   23.33 millis
      sys time   21.88 millis  617.00 micros   21.26 millis

   time day_2

   ________________________________________________________
   Executed in  227.06 millis    fish           external
      usr time    8.17 millis   70.00 micros    8.10 millis
      sys time    6.69 millis  708.00 micros    5.98 millis

   time python day_8.py

   ________________________________________________________
   Executed in   53.63 millis    fish           external
      usr time   22.11 millis   51.00 micros   22.06 millis
      sys time   24.63 millis  714.00 micros   23.91 millis

   time day_8

   ________________________________________________________
   Executed in  115.89 millis    fish           external
      usr time    5.83 millis   92.00 micros    5.74 millis
      sys time    4.59 millis  856.00 micros    3.73 millis
Now codon is much faster than Python.

Re: Codon: A high-performance Python-like compiler using LLVM

#144

Post by mk_stjames »

qwefsdf wrote: post #123

Anybody claiming it's almost Python is kidding themselves. This compiler needs to do static type checking. This is inherently impossible in Python. Not just because of some obscure corner cases that nobody uses. It's baked into the language itself. Reality-check: Why do you think type hints and type checkers like mypy and pyright take such a long time to get going and even they are not there yet? If this was all so e…

Just tried: it hangs on a numpy import. Hell it hangs on an "import time" module. If I cannot reinterpret code that is already written, then I might as well just rewrite code in a language that is better suited. Saying I can 'compile' Python code like this does indeed look like an oversell and a half.

I mean... if you had a 'compiler; for python that looked at my code at runtime- including imports- and all my current input and.... given the data types it sees and nothing more, do type-inference and recompilation down to LLVM and then to my machine code, while taking things that were already calling compiled modules (like numpy) and keeping them separate subroutines and thus only operating on the 'slow' parts of my code... with the speedups therein.. I'd be sold.

Of course, I think I basically just described Julia.

Re: Codon: A high-performance Python-like compiler using LLVM

#145

Post by yablak »

UncleEntity wrote: post #139
qwefsdf wrote: post #123

Anybody claiming it's almost Python is kidding themselves. This compiler needs to do static type checking. This is inherently impossible in Python. Not just because of some obscure corner cases that nobody uses. It's baked into the language itself. Reality-check: Why do you think type hints and type checkers like mypy and pyright take such a long time to get going and even they are not there yet? If this was all so e…

If I can write (mostly) python code and get it to run on my GPU they can oversell this all they want. I have a couple of projects I’ve been wanting to tackle but put off because I like python but it wouldn’t be a very good fit due to performance reasons. Now I get a whole new herd of yaks to shave. Plus, extensible compiler? Who doesn’t want linq in python?

Look at numba, which is not quite but sometimes good enough :)

Re: Codon: A high-performance Python-like compiler using LLVM

#146

Post by kmod »

jnxx wrote: post #91

Just out of curiosity: Why is it possible to compile Common Lisp Code (or Scheme, or Clojure) to high-performance native or jit-compiled code, but not Python? It is said that "Python is too dynamic", but is not everything in Lisp dynamic, too? And none of these languages is less powerful than Lisp, lack Unicode support, or whatever, so this can't be the reason.

Having worked on this for a while, one way that might be helpful to understand this is that Python jits (such as the one I work on, Pyston) do in fact make Python code much faster, but the fraction of the time that is spent in "Python code" is only about 20% to begin with, with the other 80% being spent in the language runtime.

For example if you write `l.sort()` where l is a list, we can make it very fast to figure out that you are calling the list_sort() C function. Unfortunately that function is quite slow because every comparison uses a dynamic multiple-dispatch resolution mechanism in order to implement Python semantics.

Re: Codon: A high-performance Python-like compiler using LLVM

#147

Post by froggit »

mangecoeur wrote: post #97
joshmaker wrote: post #95

Earlier quoted context omitted.

In 25 years you’ve never once created a list with more than one type of object in it?

I have also been using python a long time and i honestly can’t remember a time i used mixed type list.

I use mixed lists all the time to store shit, but I'm also a total python newb that really doesn't know better.

Re: Codon: A high-performance Python-like compiler using LLVM

#148

Post by dr_zoidberg »

LtWorf wrote: post #135
dr_zoidberg wrote: post #29

Earlier quoted context omitted.

I got a massive jump in performance when moving from Python 3.8 to 3.10 (over some function call optimizations I think, based on the project). And 3.11 got even better (up to 50% faster on special cases, and 10~15% on average) with respect to 3.10. Python 3.12 is already getting even more speedups and a there's a lot more down the road[0]. But Python core developers value keeping "not breaking anyones code" (Python 3…

I got performance regression going from 3.10 to 3.11 https://ltworf.github.io/typedload/performance.html

While it was still on RC builds I tested it on a few projects I usually run and I got between 0 and 15% faster wrt 3.10, without any regression. But sure, every project is different and it looks like you've bumped into some with your project.

If it's not too big, you might as well just leave things as they are and wait for 3.12 to see what changes then. Apparently the changes will be bigger, so it could be be much better than 3.11 (without being much worse, hopefully!).

Re: Codon: A high-performance Python-like compiler using LLVM

#149

Post by dr_zoidberg »

amelius wrote: post #120
dr_zoidberg wrote: post #29

Earlier quoted context omitted.

I got a massive jump in performance when moving from Python 3.8 to 3.10 (over some function call optimizations I think, based on the project). And 3.11 got even better (up to 50% faster on special cases, and 10~15% on average) with respect to 3.10. Python 3.12 is already getting even more speedups and a there's a lot more down the road[0]. But Python core developers value keeping "not breaking anyones code" (Python 3…

Interesting. What is the status of the GIL these days?

I think 3.12 will get subinterpreters, which will allow you to have multiple interpreters, each on its own thread, sharing the process memory space. So kind of like a stopgap in between having real multithreading and pythons current situation. I'm not sure what to think of it so far, so until there's some beta or RC build to test, I don't think I'll be able to form an opinion.

Re: Codon: A high-performance Python-like compiler using LLVM

#150

Post by JonChesterfield »

jnxx wrote: post #91

Just out of curiosity: Why is it possible to compile Common Lisp Code (or Scheme, or Clojure) to high-performance native or jit-compiled code, but not Python? It is said that "Python is too dynamic", but is not everything in Lisp dynamic, too? And none of these languages is less powerful than Lisp, lack Unicode support, or whatever, so this can't be the reason.

If JavaScript can be compiled effectively, and V8 strongly suggests it can, it's hard to see why python couldn't be.
Post reply on HN