Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Why I Use Nim instead of Python for Data Processing

benjamindlee.com

Post reply on HN
Print view View original

21–30 of 183 posts

Re: Why I Use Nim instead of Python for Data Processing

#21

Post by m_mueller »

amyjess wrote: post #14
zmmmmm wrote: post #8

It's primarily a testament to how simply mind bogglingly slow Python is outside of its optimised numerical science ecosystem. Which also why I don't use it that much, because while numerical analysis is a big part of what I do, so is what I would call "symbolic manipulation" and unless you go to quite some effort to transform every problem into a numerical one, Python is just awful at that. But Nim is only one of a w…

> It's primarily a testament to how simply mind bogglingly slow Python is outside of its optimised numerical science ecosystem. From my experience in using Python at my last job, I'll also add that Python is decent at tasks that aren't CPU-bound. I wrote a lot of scripts that polled large amounts of network devices for information and then did something with it (typically upsert the data into a database, either via d…

I agree with your entire post but, and I‘m saying this as a fulltime python dev, there‘s often a point where it starts being bothersome, and that usually comes only later in the lifecycle of an application after it had some organic growth. Some day e.g. a sales manager comes down to your lair and asks you if you couldn‘t just also parse this little 200MB Excel spreadsheet after it came over the network such that your ETL process could save it into a new table. And boom you‘re into CPU bound lands now. Often it‘s fine, you can wait those 1-2min for a daily occuring process. But what if for example you put this whole component behind a REST API that is behind a load balancer that is set with a certain timeout? There are even strict upper limits if for example you chose AWS Lambda for your stuff.

And suddenly you need to introduce quite a bit more technical complexity into this story that‘s gonna be hard to explain to management - all they see is that you now can insert a couple of millions of DB rows and their Big Data consultants[TM] told them that this is nowadays not even worth thinking about.

Point being: If your performance ceiling is low, you‘re gonna hit it sooner.

Re: Why I Use Nim instead of Python for Data Processing

#22

Post by csmpltn »

amyjess wrote: post #14
zmmmmm wrote: post #8

It's primarily a testament to how simply mind bogglingly slow Python is outside of its optimised numerical science ecosystem. Which also why I don't use it that much, because while numerical analysis is a big part of what I do, so is what I would call "symbolic manipulation" and unless you go to quite some effort to transform every problem into a numerical one, Python is just awful at that. But Nim is only one of a w…

> It's primarily a testament to how simply mind bogglingly slow Python is outside of its optimised numerical science ecosystem. From my experience in using Python at my last job, I'll also add that Python is decent at tasks that aren't CPU-bound. I wrote a lot of scripts that polled large amounts of network devices for information and then did something with it (typically upsert the data into a database, either via d…

> "I'll also add that Python is decent at tasks that aren't CPU-bound"

IO-bound tasks are almost by definition outside of your Python application's control. You yield control to the system to execute the actual task, and from that point on - you're no longer in control of how long the task will take to complete.

In other words, Python "being fast" by waiting on a Socket to complete receiving data isn't a particularily impressive feat.

Re: Why I Use Nim instead of Python for Data Processing

#23

Post by pjmlp »

ur-whale wrote: post #15
TekMol wrote: post #9

TLDR: Because Python is slow Yes, that is the achilles heel of Python. I am always torn between Python and PHP for new projects because of this. The Python Syntax plus its import system are huge advantages over PHP. On the other hand, you suffer a 6x slowdown if you go with Python. Decisions decisions. I so dearly wish I could have the good parts of both worlds.

Python is indeed slow, but why would anyone ever use php as a comparison point? And for data processing of all things ... Php is also very slow, on top of being many other kinds of unpleasant and broken.

PHP has a proper JIT compiler on its reference implementation.

Re: Why I Use Nim instead of Python for Data Processing

#24

Post by otherme123 »

thewakalix wrote: post #19
IshKebab wrote: post #16

Earlier quoted context omitted.

Yeah should really be "Why I don't use Python for Data Processing". I would consider Typescript as an alternative too which I'm sure would get a similar speedup. Also I don't know how anyone could design a language in the 21st century and make basic mistakes like this: > Nim treats identifiers as equal if they are the same after removing capitalization (except for the first letter) and underscore, which means that yo…

How is that a basic mistake? I think having two distinct identifiers that differ only by case sounds like a mistake!

What I found useful is that even if some author uses snake in his library, it doesn't spread to your code, so you can still adhere to the standard.

Re: Why I Use Nim instead of Python for Data Processing

#25

Post by shoo »

blondin wrote: post #17

i have seldom seen data engineers write raw python loops the way you did with your examples. they usually use numpy, scipy, etc.

that can be a good approach where the computation maps cleanly onto some C or fortran code prepared earlier and wrapped as a numpy or scipy primitive. but sometimes expressing what you want to do as a bunch of numpy operations becomes a lot harder to read and perhaps also slower than if you just directly wrote the raw loops over some arrays in C.

in cases where what you want to do doesn't exactly fit standard operations, cython can be pretty nice. e.g. 200x -- 1000x speedups for translating C-oriented number crunching code from python to cython. but if you do want performance, you have to think about it while writing the code (avoid needlessly allocating memory in tight loops, data-oriented programming with simple arrays, statically type all of your variables, ...).

Re: Why I Use Nim instead of Python for Data Processing

#26

Post by brabel »

> Nim compilation process took an additional 702 ms

That's horrifyingly slow for a compiler. The author mentioned "modern languages look like Python but run as fast as C", which is a common promise those languages make that never really materialize except for a few very happy path cases they heavily optmised the language for. Julia, for example, makes this promise too, but compiles even slower than that and takes ridiculous amounts of RAM even for hello world.

Did the author post the data set they used for the examples? Would be nice to try it out on a few languages to see how fast that can compile and run on a mature language like Common Lisp (which is just as easy to write) or even node.js.

Re: Why I Use Nim instead of Python for Data Processing

#27

Post by yellow_lead »

amyjess wrote: post #14
zmmmmm wrote: post #8

It's primarily a testament to how simply mind bogglingly slow Python is outside of its optimised numerical science ecosystem. Which also why I don't use it that much, because while numerical analysis is a big part of what I do, so is what I would call "symbolic manipulation" and unless you go to quite some effort to transform every problem into a numerical one, Python is just awful at that. But Nim is only one of a w…

> It's primarily a testament to how simply mind bogglingly slow Python is outside of its optimised numerical science ecosystem. From my experience in using Python at my last job, I'll also add that Python is decent at tasks that aren't CPU-bound. I wrote a lot of scripts that polled large amounts of network devices for information and then did something with it (typically upsert the data into a database, either via d…

As other commenters point out, how can a language be fast in a way besides something CPU bound? You are saying it is fast when it's not doing anything. Not sure I understand.

Re: Why I Use Nim instead of Python for Data Processing

#28

Post by Sanguinaire »

xiaodai wrote: post #10

How does it compare to Julia? Anyone with experience in both Nim and Julia?

I have tinkered with both. Basically if you need to write a FORTRAN application but don't want to use FORTRAN, Julia is the correct choice. If ease of use is more important than having a diverse collection of pre-existing numerical libraries to play with, I'd go with Nim.

If Nim had cloud SDKs I would use it as my default language for pretty much everything.

Re: Why I Use Nim instead of Python for Data Processing

#29

Post by benjamin-lee »

IshKebab wrote: post #16
zmmmmm wrote: post #8

It's primarily a testament to how simply mind bogglingly slow Python is outside of its optimised numerical science ecosystem. Which also why I don't use it that much, because while numerical analysis is a big part of what I do, so is what I would call "symbolic manipulation" and unless you go to quite some effort to transform every problem into a numerical one, Python is just awful at that. But Nim is only one of a w…

Yeah should really be "Why I don't use Python for Data Processing". I would consider Typescript as an alternative too which I'm sure would get a similar speedup. Also I don't know how anyone could design a language in the 21st century and make basic mistakes like this: > Nim treats identifiers as equal if they are the same after removing capitalization (except for the first letter) and underscore, which means that yo…

I actually use TypeScript/JavaScript a lot for this reason, especially for biological algorithms that I want to run in the browser. The developer tooling is also as good as you can hope for, especially when using VS Code. I actually wrote a circular RNA sequence deduplication algorithm in it just recently [1].

With respect to the identifier resolution in Nim, it strikes me as more of a matter of preference. Especially given the universal function call syntax in Nim, at least it's consistent. For example, Nim treats "ATGCA".lowerCase() the same as lowercase("ATGCA"). I do appreciate the fact that you can use a chaining syntax instead of a nesting one when doing multiple function calls but this is also a matter of style more than substance.

[1] https://github.com/Benjamin-Lee/viroiddb/blob/main/scripts/c...

Re: Why I Use Nim instead of Python for Data Processing

#30

Post by IshKebab »

thewakalix wrote: post #19
IshKebab wrote: post #16

Earlier quoted context omitted.

Yeah should really be "Why I don't use Python for Data Processing". I would consider Typescript as an alternative too which I'm sure would get a similar speedup. Also I don't know how anyone could design a language in the 21st century and make basic mistakes like this: > Nim treats identifiers as equal if they are the same after removing capitalization (except for the first letter) and underscore, which means that yo…

How is that a basic mistake? I think having two distinct identifiers that differ only by case sounds like a mistake!

Various programming languages (and file systems!) have had some form of case insensitivity and it always turns out to be an absolutely terrible idea:

* It makes searching for identifiers harder. For Nim you can't even use case insensitive search because of the underscore thing! Better practice your regexes.

* The case insensitivity rules are usually super complicated and don't apply to everything, so now it's an extra thing you have to mentally compute when coding. This is probably the biggest problem and I'm sure it has led to bugs, e.g. in SQL.

* Do you enjoy the tabs vs spaces debate? How about single quote Vs double quotes? Ugly inconsistently styled code? Well you'll love this!

* Unicode case insensitivity is actually really really complicated (this mostly applies to filesystems).

You're basically opening yourself up to an array of annoyances and gotchas for essentially no benefits.

I've literally never seen anyone use two identifiers that only differ by case, but if that were actually a big problem it could be solved just by making that illegal. You don't have to resort to the insanity of case insensitivity.

Post reply on HN