Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

A Python interpreter rewritten in Rust, that can run pip

rustpython.github.io

Post reply on HN
Print view View original

41–50 of 53 posts

Re: A Python interpreter rewritten in Rust, that can run pip

#41

Post by ridiculous_fish »

How are interpreters designed in Rust? What parts can be safe and what can't be?

What is the internal representation of a Python object? In particular how do you mutate a Python object while there may be other references to it - is it UnsafeCell, is there some checking?

How does the GC work? The GC needs to access and perhaps mutate all Python objects. In C++ a common technique is to make a linked list of scopes, stored in native stack frames; I'm not sure how that can be done in safe Rust.

Re: A Python interpreter rewritten in Rust, that can run pip

#42

Post by cat199 »

corytheboyd wrote: post #24
amelius wrote: post #16

Does it include a concurrent garbage collector? And does it solve the infamous GIL (global interpreter lock) problem?

I’m genuinely curious why people want to work around the GIL in languages like Python and Ruby. I very much do not want my simple interpreted language to inherit the complexity of true concurrency. Why not just use a different language?

this is a false dichotomy - witness the numerous parallel lisps (and schemes, etc)

Re: A Python interpreter rewritten in Rust, that can run pip

#43

Post by steveklabnik »

ridiculous_fish wrote: post #41

How are interpreters designed in Rust? What parts can be safe and what can't be? What is the internal representation of a Python object? In particular how do you mutate a Python object while there may be other references to it - is it UnsafeCell, is there some checking? How does the GC work? The GC needs to access and perhaps mutate all Python objects. In C++ a common technique is to make a linked list of scopes, sto…

I was curious too, so I poked around! I had never looked at this codebase before this comment, so, I may have some things wrong.

> How are interpreters designed in Rust? What parts can be safe and what can't be?

Depends on the design. Different interpreters are different.

A quick glance at the source code seems to show unsafe in:

* A data structure called "boxvec"

* Some calls to unreachable_unchecked, get_unchecked, and the like. Wonder what the overheads were here.

* Some lock/mutex implementations

* A few unsafe functions in the jit. They use cranelift. Bet there's a bunch in there too. Usually this is the classic thing that needs unsafe in interpreters.

* The python object data structures; they do some stuff to have a smaller representation

* some FFI code in the standard library, calls directly to libc

> What is the internal representation of a Python object?

https://github.com/RustPython/RustPython/blob/master/vm/src/...

> How does the GC work?

Traditionally, Python uses refcounting + cycle detection. A quick glance at their issue tracker implies that as of last month, they had recounting but no cycle checks yet.

> In C++ a common technique is to make a linked list of scopes, stored in native stack frames; I'm not sure how that can be done in safe Rust.

Looks like they have a RefCell>, instead.

Re: A Python interpreter rewritten in Rust, that can run pip

#44

Post by guerrilla »

pansa2 wrote: post #31
ransom_rs wrote: post #4

This uses [lalrpop]( https://github.com/lalrpop/lalrpop ) under the hood I believe, which is a pretty awesome parser generator library for Rust.

> LALRPOP in fact uses LR(1) by default (though you can opt for LALR(1)), and really I hope to eventually move to something general that can handle all CFGs (like GLL, GLR, LL(*), etc). Seems overkill for a language whose grammar is LL(1)?

> whose grammar is LL(1)?

Just curious, how did you know this?

Re: A Python interpreter rewritten in Rust, that can run pip

#45

Post by huac »

guerrilla wrote: post #44
pansa2 wrote: post #31

Earlier quoted context omitted.

> LALRPOP in fact uses LR(1) by default (though you can opt for LALR(1)), and really I hope to eventually move to something general that can handle all CFGs (like GLL, GLR, LL(*), etc). Seems overkill for a language whose grammar is LL(1)?

> whose grammar is LL(1)? Just curious, how did you know this?

The Python 3.9 (most recent) release notes:

> Python 3.9 uses a new parser, based on PEG instead of LL(1). The new parser’s performance is roughly comparable to that of the old parser, but the PEG formalism is more flexible than LL(1) when it comes to designing new language features. We’ll start using this flexibility in Python 3.10 and later.

https://docs.python.org/3/whatsnew/3.9.html#new-parser

Re: A Python interpreter rewritten in Rust, that can run pip

#46

Post by pansa2 »

guerrilla wrote: post #44
pansa2 wrote: post #31

Earlier quoted context omitted.

> LALRPOP in fact uses LR(1) by default (though you can opt for LALR(1)), and really I hope to eventually move to something general that can handle all CFGs (like GLL, GLR, LL(*), etc). Seems overkill for a language whose grammar is LL(1)?

> whose grammar is LL(1)? Just curious, how did you know this?

That Python can be described by an LL(1) grammar is frequently mentioned in discussions, but I don't know if it's formally documented anywhere.

More specifically, the grammar [0] is in EBNF form and is ELL(1). That means that if each EBNF production is converted to a set of BNF productions in an LL(1)-compliant way, the BNF grammar as a whole is LL(1). It seems that the Python tools themselves do not check this [1], but I have verified it myself.

However, as another commenter mentioned, the grammar doesn't exactly describe Python, but a superset of it. The compiler needs to perform further checks after parsing that "should" be part of the parser itself. A more advanced parser would allow these checks to be performed in the correct place - this is probably what RustPython does when using LR(1), and was one reason why CPython recently replaced its LL(1) parser with one based on PEG.

[0] https://docs.python.org/3.8/reference/grammar.html

[1] https://discuss.python.org/t/should-there-be-a-check-whether...

Re: A Python interpreter rewritten in Rust, that can run pip

#47

Post by seanmcdirmid »

pansa2 wrote: post #46
guerrilla wrote: post #44

Earlier quoted context omitted.

> whose grammar is LL(1)? Just curious, how did you know this?

That Python can be described by an LL(1) grammar is frequently mentioned in discussions, but I don't know if it's formally documented anywhere. More specifically, the grammar [0] is in EBNF form and is ELL(1). That means that if each EBNF production is converted to a set of BNF productions in an LL(1)-compliant way, the BNF grammar as a whole is LL(1). It seems that the Python tools themselves do not check this [1],…

Parsing a superset of the language and then using extra checks to detect bad syntax accepted by the superset is a well known and very effective way of being error tolerant and implementing decent error recovery.

Re: A Python interpreter rewritten in Rust, that can run pip

#49

Post by coldtea »

pansa2 wrote: post #29
corytheboyd wrote: post #24

Earlier quoted context omitted.

I’m genuinely curious why people want to work around the GIL in languages like Python and Ruby. I very much do not want my simple interpreted language to inherit the complexity of true concurrency. Why not just use a different language?

> Why not just use a different language? I’ve asked myself this question (in a different context) and have concluded that, in general, developers are incredibly resistant to learning and using new languages. For example: if they weren’t, then server-side JavaScript would never have become popular.

>I’ve asked myself this question (in a different context) and have concluded that, in general, developers are incredibly resistant to learning and using new languages.

You make it sound like a personal psychological failing.

In real life, using the same language has business benefits: re-use of the same libraries, no duplicated business logic that needs to be used on both client/server, no need for training to the new language, no need for mental context switch due to the language change when working on the client or server part (aside from the essential domain knowledge of client vs server), less/common tooling infrastructure, and so on...

Re: A Python interpreter rewritten in Rust, that can run pip

#50

Post by coldtea »

yudlejoza wrote: post #3

Wow. Looks like Rust is winning the catfight against Go, D, Julia, C++20. Need to grab more popcorn.

There's no "catfight".

Julia targets a different audience.

Go, for the most part, too.

C++20 is not going anywhere, and C++ is used more than ever.

D hasn't been a contender for 10+ years. It could technology wise, but it never gained traction.

All of Rust, Go, Julia, and C++ do well.

(Heck, C even continues to do well. Rust is not really eating into it).

It remains to be seen how the Mozilla situation (abhorent leadership, firings of Rust team people) will affect Rust.

Post reply on HN