Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

120ms to 30ms: Python to Rust

old.reddit.com

Post reply on HN
Print view View original

11–20 of 26 posts

Re: 120ms to 30ms: Python to Rust

#11

Post by nomel »

galkk wrote: post #5

Almost always when I start prototyping something in python, I wish that I stopped half-way where I am now and switched to something else. Most recent example - converting huge amount of xml files to parquet. I started very fast with python + pyarrow, but when I realized that parallelizing execution would help enormously, I hit GIL or picking/unpickling/multiprocessing costs. It did work in python, in the end, but I f…

GIL has been relieved in Python 3.12: https://realpython.com/python312-subinterpreters/

But, I have the same sentiment. Although, I find writing quick C++ extensions (swig is incredible) is a good balance.

Re: 120ms to 30ms: Python to Rust

#12

Post by fullstackchris »

jti107 wrote: post #8

use the right tool for the right job. your software requirements and design document should drive the language and the tools you use.

what i was thinking as well... i love these performance improvement posts but at the same time had to think what kind of choice was it to originally reach for python in the first place if the task was to do a lot of heavy concurrent task management???

Re: 120ms to 30ms: Python to Rust

#15

Post by jmakov »

galkk wrote: post #5

Almost always when I start prototyping something in python, I wish that I stopped half-way where I am now and switched to something else. Most recent example - converting huge amount of xml files to parquet. I started very fast with python + pyarrow, but when I realized that parallelizing execution would help enormously, I hit GIL or picking/unpickling/multiprocessing costs. It did work in python, in the end, but I f…

ray.io

Re: 120ms to 30ms: Python to Rust

#16

Post by flohofwoe »

nolroz wrote: post #14
bin_bash wrote: post #13

I enjoy both languages quite a bit, but a 4x improvement is more of a testament to python than it is to rust imo!

How do you mean? Would you expect it to be much slower?

Yeah, IME Python can be about 100x slower than a native solution. The original solution was a combination of a C library and Python wrapper code, so 4x makes sense for eliminating the Python part.

Re: 120ms to 30ms: Python to Rust

#17

Post by constantcrying »

gumby wrote: post #2

What I get from this is that the python interpreter is better than I would have guessed.

If you are using python as a way to more easily address a C library, it works very well and can be reasonably fast.

The moment you are using "pure" python on large datasets the whole thing starts to crumble.

Re: 120ms to 30ms: Python to Rust

#20

Post by _niki_s_ »

galkk wrote: post #5

Almost always when I start prototyping something in python, I wish that I stopped half-way where I am now and switched to something else. Most recent example - converting huge amount of xml files to parquet. I started very fast with python + pyarrow, but when I realized that parallelizing execution would help enormously, I hit GIL or picking/unpickling/multiprocessing costs. It did work in python, in the end, but I f…

Looks like Mojo will fix that
Post reply on HN