Live data from Hacker News

Making Python faster with Rust

ohadravid.github.io

211–220 of 223 posts

Re: Making Python faster with Rust

#211
post #151

Earlier quoted context omitted.

Because professional software developers with a background in CS are a minority of people who program today. The learning curve of pointers, memory-allocation, binary operations, programming paradigms, O-Notation and other things you need to understand to efficiently code in something like C is a lot to ask of someone who is for example primarily a sociologist or biologist. The use case btw. is often also very differ…

C is definitely not a good choice for this, I would hate to come back 2 days later to my computation and see “segfault” as the only output.

I'd be happy getting just a "segfault" from C. It would be so much better than subtly wrong results from reading uninitialized memory or out-of-bounds access, results that change depending on debug vs optimized build, or when changing some adjacent code.

Re: Making Python faster with Rust

#212

Earlier quoted context omitted.

I guess you also need to take the time into account to create the quad tree from an unsorted 'polygon soup' first, and in terms of coding effort, a brute force conversion from python to a compiled tight loop over unsorted arrays provides a lot of bang for the buck (and a speedup of 100x for relatively little effort might be 'good enough' for quite a while until the input data grows big enough to require the next opti…

Im assuming the polygons dont change to often so you can amortize the construction of the quadtree. Depending on your world view the implementation is trivial since Shapely, a dependency they already likely have, has an implementation of it.

Author here: actually, in this analogy (as this is just a demo library), the polygons change each time so we couldn't use this type of optimization (at least not in a straightforward way).

Re: Making Python faster with Rust

#213
post #207

Earlier quoted context omitted.

Better clean up the Linux kernel from Oracle contributions then, in case you are using it. https://lwn.net/Articles/915435/

So the Linux kernel is an Oracle licensed product? Better uninstall Ubuntu then.

I am not the one having issues with Oracle.

In fact, I am thankful that they at least avoided Java being stuck in Java 6, and MaximeVM turned into GraalVM, when no one else cared to save Sun from insolvency. Only IBM made a candidate offer that was quickly withdrawn.

So people should stop acting as if there was any magic way to rescue Sun assets.

Also the fact they were one of the first GNU/Linux supporters in enterprise context, which allowed us to actually have a couple of GNU/Linux computers among our Aix, HP-UX and Solaris servers, happily running Oracle instances.

Re: Making Python faster with Rust

#214
post #205
post #45

Earlier quoted context omitted.

It sounds amazing, but bear in mind there are a lot of code which can’t be sped up like this because: - Some code doesn’t have obvious optimization hotspots, and is instead just generally slow everywhere. - Most FFI boundaries incur their own performance cost. I’m not sure about Python, but I wouldn’t be surprised if FFI to rust in a hot loop is often slower than just writing the same code in Python directly. And it’…

> Most FFI boundaries incur their own performance cost. I’m not sure about Python, but I wouldn’t be surprised if FFI to rust in a hot loop is often slower than just writing the same code in Python directly. And it’s not always easy to refactor to avoid this. They definitely do, but I’d usually suggest that if you find this an issue then perhaps the function you’re exposing from the compiled language should be higher…

Maybe. But that can also be a self tightening knot. Sometimes there’s no elegant place to cleave a program or library in two, and you really just want to pick a single language for the whole project.

Mixing languages can also be a bit of a disaster for maintainability. Refactoring codebases which meaningfully span multiple languages is miserable work.

Re: Making Python faster with Rust

#215
post #151

Earlier quoted context omitted.

C is definitely not a good choice for this, I would hate to come back 2 days later to my computation and see “segfault” as the only output.

I'd be happy getting just a "segfault" from C. It would be so much better than subtly wrong results from reading uninitialized memory or out-of-bounds access, results that change depending on debug vs optimized build, or when changing some adjacent code.

Sure, a segfault was just more visual, that’s why I went with that.

Re: Making Python faster with Rust

#216
post #36
post #34

Earlier quoted context omitted.

To some degree, it's about familiarity with your tools. And different tools are optimized for different tasks. Besides, aren't you deploying Docker containers, anyway?

I do, absolutely. But it seems like an exceptionally rare practice for most Python code, at least in the ML space, which puts me back at the start.

Many ML practitioners aren't software engineers. I don't expect that cohort (non-engineers) would manage a deployment well in any language.

Re: Making Python faster with Rust

#217

> Python is a superb API for researchers, Where does this nonsense come from? No. It isn't. It's just a stupid fashion. Something that should be discouraged, not encouraged by trying to make it work when it's obviously broken. As someone who does work with researchers who do use Python a lot, I see the everyday painful experiences of people who use it. And this pain doesn't need to be there. It's just masochism. And…

>Python is just a bad language. Popular, but awful. You couldn't be more wrong. Python is the language going forward. There is a reason why bleeding edge ML stuff is done through Python, as well as it being the backend to several very popular web platforms and is second most use language on github behind JS solely because JS is hard tied to web. I have a feeling the hate for Python is just comes from paradigms that a…

No, you couldn't be more wrong. Nice argument you dug up there!

Language going forward... I hadn't had such a laugh in a while... It was dead in the 70s. Long before it was born. Are you under influence or something?

> There is a reason why bleeding edge ML stuff is done through Python

No, there isn't, and bleeding edge ML isn't done in Python. It's done mostly in C++ and more seldom in C. Which are another garbage languages, but that's just the reality we live in. Python is just a tiny fraction of what's being done, the tip of the tip of the iceberg.

And, no Python is not the backend of popular platforms, it is, again, a tiny little bit of what's going on in those "popular platforms", and, unlike you, I actually worked for those "popular platforms", so, would know how that is.

But, even, imagine that in your fairy-tale world Python is somehow so super-important and successful? Didn't I already explain how this is possible while still being a garbage language? -- I bet I did. You just rushed to spit your despair and frustration at me, because I offended something that you like... well, you think that you like, but really, you just don't know any better, just like other people using Python :( And that's really the sad part. You lack critical thinking to be really able to tell the quality of your tools. You substituted the ability to judge the quality of something by looking at what the majority is doing and following the herd.

And, I didn't study when Python became taught in school. In my days the language of choice for intro to CS was Visual Basic :| It's also garbage... and the whole idea that the intro to CS has to be done by learning a random fashionable language of the day is just dumb. It's a misunderstanding of SICP, which to someone who didn't understand what the book is about looks as if the author tried to teach the students Scheme, and later intro to CS classes were modeled on this book, but replaced Scheme with another language, while completely forgetting that Scheme was used in the first place as a language with "little syntax", to avoid spending time on learning the language.

You again are confused when you use the expression "dynamic typing", but you don't even know what that is. But you will never accept it because, again, you don't have the ability to critically think, you follow the herd. You red a "definition" somewhere that says that Python is "dynamically typed language", and you believed this nonsense, even though you never really thought about what that might mean...

One thing we agree on though: the direction the compute is going is to utilize more low-skill programmers, and that means, beside other things, following fads more than doing any sensible work. Python is the current fad, and so, for a while, we are stuck with this garbage. And, the way things are going, at best, I can hope for another equally idiotic language to come to replace it. But, probably not very soon.

Re: Making Python faster with Rust

#218

Earlier quoted context omitted.

For research? -- Julia seems to be definitely better. It's purpose-built for doing just that. R would also be there. If you want general statistics, then add J to the fold. But specific fields often have their own, bespoke solutions. I've only ever dealt with math, but it has plenty of its own niche languages that are much better than, say, Sage. My personal choice was Maxyma, but that's because I like Common Lisp. F…

> For research? -- Julia seems to be definitely better. It's purpose-built for doing just that. What does this even mean lol, “research” is incredibly broad

I'll elaborate. When I use the word "research" I mean activity performed by accredited researchers. Researcher is a rank, or a title if you want. It's similar to "software engineer" -- if you are in an academic institution the word "researcher" could be part of your title / job description.

The activity has as its goal to publish the findings. Findings are expected to be meaningful and to be found in the domain of sciences. It is also possible to extend the scope to arts, but I'm hesitant about it, and the way I used the word doesn't truly relate to what happens in arts department (for this purpose, math is an art).

Accredited, here means association with academic institution by means of employment or similar.

On conceptual level, all discoveries in sciences must have a hypothesis whose validity and truthfulness is to be established through empirical evidence, i.e. experiment. To interpret experiments one needs to use statistics. This is the vehicle driving experiments. This is where the kind of programming I'm talking about comes in.

Note that researchers might use programming for other activities s.a. eg. automation of keeping their diaries, or automating their correspondence and so on. But I meant specifically the statistical aspect of doing research.

Re: Making Python faster with Rust

#219

Earlier quoted context omitted.

For research? -- Julia seems to be definitely better. It's purpose-built for doing just that. R would also be there. If you want general statistics, then add J to the fold. But specific fields often have their own, bespoke solutions. I've only ever dealt with math, but it has plenty of its own niche languages that are much better than, say, Sage. My personal choice was Maxyma, but that's because I like Common Lisp. F…

>R would also be there. Surely you jest. Whatever problems Python may have with proper development or deployment practices, R is ten-fold worse. R is and will forever be an interactive, make-it-work-now language and production backbone second. The language is far too accommodating and will go silent casting all manner of sins in the manner of keeping the program running. Package management is still a huge headache as…

> R is and will forever be an interactive

I don't see this as a problem for researchers. Quite the contrary. To compare this to Python: Python's interactivity sucks. Bad syntax prevents it from using it interactively efficiently. Abysmal debugger. And if you consider that a typical environment in which Python is used in research setting is a notebook, then add to it an even worse wrapper for debugger available in notebooks.

Python is garbage for production systems too. If you want to use the results of your research for practical stuff, you will not use Python code you wrote for research. I'm more familiar with the world of medical research, and can confidently say that I've never seen a practical medical device or software product that used Python. Medical equipment typically wants to be realtime, which is a world completely closed to Python, for example.

However, Python (or R) being bad for production systems is perceived to be an acceptable price... I wish it wasn't, but that's how things are.

You cannot unironically claim that Python has superior package management... It's the shittiest ever. I have not seen a language which has done it worse than Python, and I've worked with at least 2-3 dozens of them. In my day job, I'm in the ops / infra department. I maintain a lot of Python packages both for commercial entities and for open-source independent developers. Part of me is very upset that Python is such a shit-show when it comes to packaging, but another part of me is happy because it means that I will have a job dealing with the fallout of Python packaging for a while.

Bottom line, if I had to support a bunch of scientists doing their research, and I had to deal with packaging their stuff, I'll take R over Python in a heartbeat.

Re: Making Python faster with Rust

#220
post #206

Earlier quoted context omitted.

For python, you can use pandas. It even has arrow support now.

Yeah, I need to try it. But the parent was suggesting that metaprogramming could make readability issues go away and I can't wrap my head around how it would do that.

In Julia you can use https://juliaarrays.github.io/StructArrays.jl/stable/ which lets your code use the array of structs interface but internally it’s a structure of arrays
Post reply on HN