Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Making Python faster with Rust

ohadravid.github.io

Post reply on HN
Print view View original

181–190 of 223 posts

Re: Making Python faster with Rust

#181

Post by tayo42 »

brahbrah wrote: post #87
b0b10101 wrote: post #17

This is a great article but there's still a core problem there - why should developers have to choose between accessibility and performance? So much scientific computing code suffers between core packages being split away from their core language - at what point do we stop and abandon python for languages which actually make sense? Obviously julia is the big example here, but its interest, development and ecosystem d…

They would have gotten the same performance in python with numpy if they did it like this instead of calling norm for every polygon centers = np.array([p.center for p in ps]) norm(centers - point, axis=1) They were just using numpy wrong. You can be slow in any language if you use the tools wrong

what is the difference?

though I do feel like i see this a lot with these kinds of "we re-wrote it in rust and everything is fast". comparing to a language with gc options often the scenario

on one hand, i feel like you should just learn how to use your stuff properly. on the other hand it is interesting to see that people who can't write fast code or use libraries properly are actually writing fast code. like fast code for the masses almost hah. though maybe theyll just run into the same issue when they misuse a library in rust

Re: Making Python faster with Rust

#182

Post by Terretta »

onetonfoot wrote: post #68
fbdab103 wrote: post #32

Earlier quoted context omitted.

Today, there is a Python package for everything . The ecosystem is possibly best in class for having a library available that will do X. You cannot separate the language from the ecosystem. Being better, faster, and stronger means little if I have to write all of my own supporting libraries. Also, few scientific programmers have any notion of what C or Fortran is under the hood. Most are happy to stand on the shoulde…

You can have your cake and eat it with the likes of * PythonCall.jl - https://github.com/cjdoris/PythonCall.jl * NodeCall.jl - https://github.com/sunoru/NodeCall.j * RCall.jl - https://github.com/JuliaInterop/RCall.jl I tend to use Julia for most things and then just dip into another language’s ecosystem if I can’t find something to do the job and it’s too complex to build myself

* NodeCall.jl - https://github.com/sunoru/NodeCall.jl

// just fixed missing 'l' in link

Re: Making Python faster with Rust

#183

Post by ActorNightly »

crabbone wrote: post #160

> Python is a superb API for researchers, Where does this nonsense come from? No. It isn't. It's just a stupid fashion. Something that should be discouraged, not encouraged by trying to make it work when it's obviously broken. As someone who does work with researchers who do use Python a lot, I see the everyday painful experiences of people who use it. And this pain doesn't need to be there. It's just masochism. And…

>Python is just a bad language. Popular, but awful.

You couldn't be more wrong. Python is the language going forward. There is a reason why bleeding edge ML stuff is done through Python, as well as it being the backend to several very popular web platforms and is second most use language on github behind JS solely because JS is hard tied to web.

I have a feeling the hate for Python is just comes from paradigms that are taught in extremely poor CS curriculum in schools. If you think that Python is bad because of dynamic typing, you haven't been paying attention to the direction compute is going.

Re: Making Python faster with Rust

#184

Post by korijn »

pbowyer wrote: post #97
korijn wrote: post #60

A vectorized implementation of find_close_polygons wouldn't be very complex or hard to maintain at all, but the authors would also have to ditch their OOP class based design, and that's the real issue here. The object model doesn't lend itself to performant, vectorized numpy code.

What's a good guide to learn how to make (and see) vectorized code? It's a mindshift and not one I find easy.

The gist of it is that you give numpy two arrays, and what operation to apply. Then numpy will figure out what the for loop(s) should look like depending on the shape of the arrays.

You can look at various tutorials to see how it works. For example: https://jakevdp.github.io/PythonDataScienceHandbook/02.05-co...

Re: Making Python faster with Rust

#185

Post by ActorNightly »

b0b10101 wrote: post #17

This is a great article but there's still a core problem there - why should developers have to choose between accessibility and performance? So much scientific computing code suffers between core packages being split away from their core language - at what point do we stop and abandon python for languages which actually make sense? Obviously julia is the big example here, but its interest, development and ecosystem d…

I don't know whether this sentiment is just a byproduct of CS education, but for some reason people equate a programming language with the compute that goes on under the hood. Like if you write in Python, you are locked into the specific non optimized way of computing that Python does.

Its all machine code under the hood. Everything else on top is essentially description of more and more complex patterns of that code. So its a no brainer that a language that lets you describe those complex but repeating patterns in the most direct way is the most popular. When you use python, you are effectively using a framework on top of C to describe what you need, and then if you want to do something specialized for performance, you go back to the core fundamentals and write it in C.

Re: Making Python faster with Rust

#186

Post by IshKebab »

alex_sf wrote: post #30
INTPenis wrote: post #11

Earlier quoted context omitted.

Yeah what's wrong with that? I think this sounds amazing. It gives you all the fast prototyping and simplicity of Python, but once you hit that bottleneck all you have to do is bring in a ringer to replace key components with a faster language. No need to use Golang or Rust from the start, no need for those resources until you absolutely need the speed improvement. Sounds like a dream to a lot of people who find it m…

Python is a rough language to be productive in. It's a great scratchpad, but dynamic typing, exceptions/poor error handling, and a horrifying deployment and dependency system make me reach for something like Go in any case where I need something to be even vaguely reliable. The more ML I do, the more disappointed I get.

I agree. Add in the fact that you will have to do awkward FFI to some faster language... It's better just to use that language in the first place.

Re: Making Python faster with Rust

#187

Post by INTPenis »

alex_sf wrote: post #30
INTPenis wrote: post #11

Earlier quoted context omitted.

Yeah what's wrong with that? I think this sounds amazing. It gives you all the fast prototyping and simplicity of Python, but once you hit that bottleneck all you have to do is bring in a ringer to replace key components with a faster language. No need to use Golang or Rust from the start, no need for those resources until you absolutely need the speed improvement. Sounds like a dream to a lot of people who find it m…

Python is a rough language to be productive in. It's a great scratchpad, but dynamic typing, exceptions/poor error handling, and a horrifying deployment and dependency system make me reach for something like Go in any case where I need something to be even vaguely reliable. The more ML I do, the more disappointed I get.

That's just habitual. Anyone coming from Python could say the same things about another language. The typing never bothered me because you learn to work with it. But there is of course type hinting now, which I barely use.

Re: Making Python faster with Rust

#188

Post by dralley »

akasakahakada wrote: post #150
oblio wrote: post #143

Earlier quoted context omitted.

This is nice, how would you go about as a performance noob? I can't imagine there's a line in the docs saying "this is slow!".

LineProfiler is the best tool to learn how to write performant Python and do code optimization. https://github.com/pyutils/line_profiler You can literally see the hot spot of your code, then you can grind different algorithms or change the whole architecture to make it faster. For example replace short for loops to list comprehensions, vectorize all numpy operations (only vectorize partially do not help the issue), u…

You should take a look at Scalene - it's even better.

https://github.com/plasma-umass/scalene

Re: Making Python faster with Rust

#189

Post by 0cf8612b2e1e »

crabbone wrote: post #165
pahbloo wrote: post #162

Earlier quoted context omitted.

What are the best alternatives to Python then?

For research? -- Julia seems to be definitely better. It's purpose-built for doing just that. R would also be there. If you want general statistics, then add J to the fold. But specific fields often have their own, bespoke solutions. I've only ever dealt with math, but it has plenty of its own niche languages that are much better than, say, Sage. My personal choice was Maxyma, but that's because I like Common Lisp. F…

>R would also be there.

Surely you jest. Whatever problems Python may have with proper development or deployment practices, R is ten-fold worse. R is and will forever be an interactive, make-it-work-now language and production backbone second. The language is far too accommodating and will go silent casting all manner of sins in the manner of keeping the program running. Package management is still a huge headache as proper isolation is still not well addressed. Too many R packages assume they have root during installation and can do whatever they wish. Volumes can be written about R namespacing.

Re: Making Python faster with Rust

#190

Post by pnt12 »

insanitybit wrote: post #147
junipertea wrote: post #59

Earlier quoted context omitted.

That’s a lot of opinions for so little exposure. There are a lot uses that don’t involve docker or a dozen virtual envs.

I've used Python quite a lot and their experience sounds about right.

To counter these anecdotes, I used python for building web apis and only needed poetry to manage 1 virtual env, and containerizing with docker was straight forward.
Post reply on HN