Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

120ms to 30ms: Python to Rust

old.reddit.com

Post reply on HN
Print view View original

21–26 of 26 posts

Re: 120ms to 30ms: Python to Rust

#21

Post by mharig »

galkk wrote: post #5

Almost always when I start prototyping something in python, I wish that I stopped half-way where I am now and switched to something else. Most recent example - converting huge amount of xml files to parquet. I started very fast with python + pyarrow, but when I realized that parallelizing execution would help enormously, I hit GIL or picking/unpickling/multiprocessing costs. It did work in python, in the end, but I f…

1. Pickling is, as already mentioned by a cocommenter, slow. Do not use it.

2. Do not use loops, use itertools. (Python 3.12 got a nice 'batched' function, btw.)

3. Preallocate memory, where possible.

4. multiprocessing.shared_memory may help.

5. Something like cython may help.

6. Combination of multiprocessing with asyncio (or multithreading) may help.

7. memmap file access might help.

You see, you have a lot of options before you need to learn another language.

Re: 120ms to 30ms: Python to Rust

#22

Post by itishappy »

_niki_s_ wrote: post #20
galkk wrote: post #5

Almost always when I start prototyping something in python, I wish that I stopped half-way where I am now and switched to something else. Most recent example - converting huge amount of xml files to parquet. I started very fast with python + pyarrow, but when I realized that parallelizing execution would help enormously, I hit GIL or picking/unpickling/multiprocessing costs. It did work in python, in the end, but I f…

Looks like Mojo will fix that

Right, by switching languages.

Re: 120ms to 30ms: Python to Rust

#23

Post by Talinx »

That is a lot of text for not determining why the new solution is faster. The only relevant part:

> Before our migration, the old pipeline utilized a C library accessed through a Python service, which buffered and bundled data. This was really the critical aspect that was causing our latency.

How much speed up would there have been if they moved to a Rust wrapper around the same C library?

Using something other than Python is almost always going to be faster. This Reddit post does not give any insights into which aspects of Python lead to small/large performance hits. They show that it was the right solution for them with ample documentation which is great, but they don't provide any generalizable information.

Re: 120ms to 30ms: Python to Rust

#25

Post by bdjsiqoocwk »

galkk wrote: post #5

Almost always when I start prototyping something in python, I wish that I stopped half-way where I am now and switched to something else. Most recent example - converting huge amount of xml files to parquet. I started very fast with python + pyarrow, but when I realized that parallelizing execution would help enormously, I hit GIL or picking/unpickling/multiprocessing costs. It did work in python, in the end, but I f…

Julia

Re: 120ms to 30ms: Python to Rust

#26

Post by cedws »

galkk wrote: post #5

Almost always when I start prototyping something in python, I wish that I stopped half-way where I am now and switched to something else. Most recent example - converting huge amount of xml files to parquet. I started very fast with python + pyarrow, but when I realized that parallelizing execution would help enormously, I hit GIL or picking/unpickling/multiprocessing costs. It did work in python, in the end, but I f…

I recently needed to do something similar and used Apache Arrow’s Go library for Parquet. It’s horrifically slow and somehow manages to leak memory. It’s also undocumented. If anybody knows a good Parquet library for Go please let me know.
Post reply on HN