120ms to 30ms: Python to Rust
old.reddit.com
120ms to 30ms: Python to Rust
1–10 of 26 posts
Re: 120ms to 30ms: Python to Rust
#2What I get from this is that the python interpreter is better than I would have guessed.
Re: 120ms to 30ms: Python to Rust
#3What I get from this is that the python interpreter is better than I would have guessed.
Same, until I read "C library accessed through Python"
Re: 120ms to 30ms: Python to Rust
#4What I get from this is that the python interpreter is better than I would have guessed.
Same, until I read "C library accessed through Python"
Well, depending on how much python is written around the C library, you'll get a lot of overhead.
There's also the fact that having everything be compiled simultaneously results in the optimizer being able to get a lot more work done.
Re: 120ms to 30ms: Python to Rust
#5Almost always when I start prototyping something in python, I wish that I stopped half-way where I am now and switched to something else.
Most recent example - converting huge amount of xml files to parquet. I started very fast with python + pyarrow, but when I realized that parallelizing execution would help enormously, I hit GIL or picking/unpickling/multiprocessing costs.
It did work in python, in the end, but I feel that writing that in Rust/C# (even if I don't know Rust besides tutorials) in the end would be much more performant.
Re: 120ms to 30ms: Python to Rust
#6Almost always when I start prototyping something in python, I wish that I stopped half-way where I am now and switched to something else. Most recent example - converting huge amount of xml files to parquet. I started very fast with python + pyarrow, but when I realized that parallelizing execution would help enormously, I hit GIL or picking/unpickling/multiprocessing costs. It did work in python, in the end, but I f…
[deleted]
Re: 120ms to 30ms: Python to Rust
#7Almost always when I start prototyping something in python, I wish that I stopped half-way where I am now and switched to something else. Most recent example - converting huge amount of xml files to parquet. I started very fast with python + pyarrow, but when I realized that parallelizing execution would help enormously, I hit GIL or picking/unpickling/multiprocessing costs. It did work in python, in the end, but I f…
> converting huge amount of xml files
> pickling
Sounds like if this is the tooling and the task at hand, about the most complex things that should be passing through the pickler are partitioned lists of filenames rather than raw data. E.g. you can have each partition generate a parquet for combining in a final step (pyarrow.concat_tables() looks useful), or if it were some other format you were working with, potentially sending flat arrays back to the parent process as giant bytestrings or similar
This is not to say the limitations don't suck, just that very often there are simple approaches to avoid most of the pain
Re: 120ms to 30ms: Python to Rust
#8use the right tool for the right job. your software requirements and design document should drive the language and the tools you use.
Re: 120ms to 30ms: Python to Rust
#9Almost always when I start prototyping something in python, I wish that I stopped half-way where I am now and switched to something else. Most recent example - converting huge amount of xml files to parquet. I started very fast with python + pyarrow, but when I realized that parallelizing execution would help enormously, I hit GIL or picking/unpickling/multiprocessing costs. It did work in python, in the end, but I f…
You can quickly build prototypes with LuaJIT and it’s also quite performant.
It’s worth giving it a try if you haven’t before.
Re: 120ms to 30ms: Python to Rust
#10Almost always when I start prototyping something in python, I wish that I stopped half-way where I am now and switched to something else. Most recent example - converting huge amount of xml files to parquet. I started very fast with python + pyarrow, but when I realized that parallelizing execution would help enormously, I hit GIL or picking/unpickling/multiprocessing costs. It did work in python, in the end, but I f…
> converting huge amount of xml files > pickling Sounds like if this is the tooling and the task at hand, about the most complex things that should be passing through the pickler are partitioned lists of filenames rather than raw data. E.g. you can have each partition generate a parquet for combining in a final step (pyarrow.concat_tables() looks useful), or if it were some other format you were working with, potenti…
[deleted]