Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Making ast.walk 220x Faster

reflex.dev

Post reply on HN
Print view View original

11–20 of 22 posts

Re: Making ast.walk 220x Faster

#11

Post by mananaysiempre »

adhami wrote: post #3
123rust123 wrote: post #2

[flagged]

only because I'm too lazy to learn how to write C with Python, if anything Rust wasn't helpful with all of those unsafes

You’d have to spend a bit of time picking apart your arguments and releasing stuff on early error returns, but otherwise the Python/C API is completely banal and doesn’t really have much to learn. It might be worth the time to poke at it at some point to assure yourself this is the case.

(Or you could switch to C++ and use pybind11, but now you’re just switching from one quite complex and somewhat off-putting language to another really complicated and very ugly one, so the win is less clear.)

Re: Making ast.walk 220x Faster

#13

Post by eska »

flockonus wrote: post #10
eska wrote: post #7

I appreciate that you first tried to optimize the original Python code. Idiomatic Python is unfortunately disappointingly slow and not so interesting to compare to.

> not so interesting to compare to Absolutely disagree here, something that is considered good practice is very interesting to compare to!

I mean that mostly in the sense that there is huge variance in idiomatic code. So your optimized C/Rust code might be 100-1000x faster than two idiomatic versions of writing that code

Re: Making ast.walk 220x Faster

#14

Post by elevation »

jerf wrote: post #8
eska wrote: post #7

I appreciate that you first tried to optimize the original Python code. Idiomatic Python is unfortunately disappointingly slow and not so interesting to compare to.

I often use the rough approximation that Python is 40-50x slower than C. This is what you'll see in the benchmarks. The truly rough thing about Python though is that that is the speed when the code is being written to a benchmark. It is really, really easy to write Python that is multiples slower than that when not writing to a benchmark and just trying to get work done without hyperoptimizing. I did some testing of…

> 40-50x number is more lower bound [...] easily score another order of magnitude slower

This is about what I observe. I had a utility based on `scapy`; there were no obviously bad ideas in the python source, but porting the work loop into a cpython extension module yielded a 500x speedup.

Re: Making ast.walk 220x Faster

#17

Post by anitil »

elevation wrote: post #14
jerf wrote: post #8

Earlier quoted context omitted.

I often use the rough approximation that Python is 40-50x slower than C. This is what you'll see in the benchmarks. The truly rough thing about Python though is that that is the speed when the code is being written to a benchmark. It is really, really easy to write Python that is multiples slower than that when not writing to a benchmark and just trying to get work done without hyperoptimizing. I did some testing of…

> 40-50x number is more lower bound [...] easily score another order of magnitude slower This is about what I observe. I had a utility based on `scapy`; there were no obviously bad ideas in the python source, but porting the work loop into a cpython extension module yielded a 500x speedup.

I feel like scapy is a bit of a special case though. It's unusually dynamic because it allows kind of arbitrarily crazy things. I feel like it's maily best for prototyping. I've had 1000x speedups (100s of milliseconds to dozens of microseconds) just by removing dynamism. In our case we used scapy to work out the packet shapes we cared about and then just implemented just that subset (still in Python!). But that dynamism really helped with the early stages along with wireshark

Re: Making ast.walk 220x Faster

#18

Post by anitil »

I really enjoy making Python faster. I feel like the sweet spot for me is proving a concept with the dumbest possible implementation to show that something would work, and then using that as a comparison implementation to prove that later improvements match the results of the dumb, obviously correct implementation.

Re: Making ast.walk 220x Faster

#19

Post by teo_zero »

The second part, where it says that Rust is faster than Python, is so obvious than doesn't deserve any further comments. What I found interesting, is that idiomatic Python, with generators and all, can be so much worse than "C-style" Python, if you will pardon that term. It's a pity many kids are taught Python before C...

Re: Making ast.walk 220x Faster

#20

Post by westurner »

adhami wrote: post #6
westurner wrote: post #4

Could this ast.sprint ast.walk optimization make libCST or bandit faster? https://news.ycombinator.com/item?id=39111747 libCST: https://github.com/Instagram/LibCST bandit: https://github.com/PyCQA/bandit Links to codemod tools; "Baby Steps into Genetic Programming" https://news.ycombinator.com/item?id=43617655

it's possible! although many of the constraints in this blog were because we wanted to work with ast module in Python. If we were allowed to create our own types, we can do so much better. I think ruff has an even faster walk by those standards. It seems bandit is using some decent optimizations already, looking at the `@test.checks("Call")` seems like they already captured some easy wins. The largest win honestly wo…

ruff probably does.

Docs for ruff_python_ast: https://docs.rs/littrs-ruff-python-ast/latest/ruff_python_as...

src: ruff/crates/ruff_python_ast: https://github.com/astral-sh/ruff/tree/main/crates/ruff_pyth...

ast.toml, generate.py, Cargo.toml, src/ https://github.com/astral-sh/ruff/blob/main/crates/ruff_pyth...

Just rewrote docutils and myst-md-parser in rust with byte-for-byte equivalent output the other day; as westurner/dsport/src/docutilsrs and sphinxdocrs and so on. Pygmentsrs required fancy-regex because rust regex is linear. rstest and cargo-insta helped.

PyCQA/docformatter https://github.com/PyCQA/docformatter and https://github.com/PyCQA/doc8 would be useful in rust, too. This from the other day: "A benchmark for catching when code doesn't do what its documentation claims"; docs fidelity evals https://news.ycombinator.com/item?id=48530786

Post reply on HN