what's up we the all the shouting in the code ?
Why I Use Nim instead of Python for Data Processing
11–20 of 183 posts
Re: Why I Use Nim instead of Python for Data Processing
#12How does it compare to Julia? Anyone with experience in both Nim and Julia?
I would have thought Cython would be the closest analogue for comparison.
Re: Why I Use Nim instead of Python for Data Processing
#13does nim have anything like scikit yet?
Closest I can find is https://nimble.directory/pkg/science
Re: Why I Use Nim instead of Python for Data Processing
#14It's primarily a testament to how simply mind bogglingly slow Python is outside of its optimised numerical science ecosystem. Which also why I don't use it that much, because while numerical analysis is a big part of what I do, so is what I would call "symbolic manipulation" and unless you go to quite some effort to transform every problem into a numerical one, Python is just awful at that. But Nim is only one of a w…
> It's primarily a testament to how simply mind bogglingly slow Python is outside of its optimised numerical science ecosystem.
From my experience in using Python at my last job, I'll also add that Python is decent at tasks that aren't CPU-bound.
I wrote a lot of scripts that polled large amounts of network devices for information and then did something with it (typically upsert the data into a database, either via direct SQL or a REST API to whatever service owns the database). All these tasks were heavily network-bound. The amount of time the CPU was doing any work was minuscule compared to the amount of time it was waiting to get data back from the network. I doubt Nim or any other language would have been a significant performance improvement in this case.
For what it's worth, that made these scripts excellent candidates for multithreading. I'd run them with 20+ threads, and it was glorious. At first I did multiprocessing, because of all the GIL horror stories, but multiprocessing made it very difficult to cache data, so eventually I said "well, all this is network-bound so the GIL doesn't even apply" and switched over to multiprocessing.dummy (which implements pools using the same API as multiprocessing but with threads instead of processes), and I never looked back.
Edit: For what it's worth, Nim sounds like a really cool language, and it's right up my alley in several ways, I just don't think Python is particularly slow at network-bound tasks that use very little CPU.
Re: Why I Use Nim instead of Python for Data Processing
#15TLDR: Because Python is slow Yes, that is the achilles heel of Python. I am always torn between Python and PHP for new projects because of this. The Python Syntax plus its import system are huge advantages over PHP. On the other hand, you suffer a 6x slowdown if you go with Python. Decisions decisions. I so dearly wish I could have the good parts of both worlds.
Python is indeed slow, but why would anyone ever use php as a comparison point?
And for data processing of all things ...
Php is also very slow, on top of being many other kinds of unpleasant and broken.
Re: Why I Use Nim instead of Python for Data Processing
#16It's primarily a testament to how simply mind bogglingly slow Python is outside of its optimised numerical science ecosystem. Which also why I don't use it that much, because while numerical analysis is a big part of what I do, so is what I would call "symbolic manipulation" and unless you go to quite some effort to transform every problem into a numerical one, Python is just awful at that. But Nim is only one of a w…
Yeah should really be "Why I don't use Python for Data Processing". I would consider Typescript as an alternative too which I'm sure would get a similar speedup.
Also I don't know how anyone could design a language in the 21st century and make basic mistakes like this:
> Nim treats identifiers as equal if they are the same after removing capitalization (except for the first letter) and underscore, which means that you can use whichever style you want.
If that's any indication of the sanity of the rest of Nim then I'd say steer well clear!
Re: Why I Use Nim instead of Python for Data Processing
#17i have seldom seen data engineers write raw python loops the way you did with your examples. they usually use numpy, scipy, etc.
Re: Why I Use Nim instead of Python for Data Processing
#18NIM also has a very good JavaScript backend. You can generate both C and JavaScript code from a nim program.
Last time I used it, I liked it but didn't use it long enough to have a strong opinion.
Re: Why I Use Nim instead of Python for Data Processing
#19It's primarily a testament to how simply mind bogglingly slow Python is outside of its optimised numerical science ecosystem. Which also why I don't use it that much, because while numerical analysis is a big part of what I do, so is what I would call "symbolic manipulation" and unless you go to quite some effort to transform every problem into a numerical one, Python is just awful at that. But Nim is only one of a w…
Yeah should really be "Why I don't use Python for Data Processing". I would consider Typescript as an alternative too which I'm sure would get a similar speedup. Also I don't know how anyone could design a language in the 21st century and make basic mistakes like this: > Nim treats identifiers as equal if they are the same after removing capitalization (except for the first letter) and underscore, which means that yo…
How is that a basic mistake? I think having two distinct identifiers that differ only by case sounds like a mistake!
Re: Why I Use Nim instead of Python for Data Processing
#20It's primarily a testament to how simply mind bogglingly slow Python is outside of its optimised numerical science ecosystem. Which also why I don't use it that much, because while numerical analysis is a big part of what I do, so is what I would call "symbolic manipulation" and unless you go to quite some effort to transform every problem into a numerical one, Python is just awful at that. But Nim is only one of a w…
> It's primarily a testament to how simply mind bogglingly slow Python is outside of its optimised numerical science ecosystem. From my experience in using Python at my last job, I'll also add that Python is decent at tasks that aren't CPU-bound. I wrote a lot of scripts that polled large amounts of network devices for information and then did something with it (typically upsert the data into a database, either via d…
You have to be reaaaaaally slow to be beaten by a network. Also, touched by the OP, if it takes 3 hours to write some code that in Python takes only 1, or if the compile times are huge, Python can beat other languages in speed (that edge fades when the same program is used over and over again).
But as demonstrated, Nim is fast to write and fast to compile, so Python has little edge. Just it's huge ecosystem.