Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Why I Use Nim instead of Python for Data Processing

benjamindlee.com

Post reply on HN
Print view View original

141–150 of 183 posts

Re: Why I Use Nim instead of Python for Data Processing

#141

Post by hexo »

maleldil wrote: post #123
hexo wrote: post #104

Earlier quoted context omitted.

It actually makes a lot of sense. I personally don't want to touch .Net langs because of their PascalCase and camelCase coding style which I find very unfortunate arbitrary decision. And well, didn't really want to get into Nim because of camelCase but since I've learned about this feature I'm reconsidering again.

Do you really dislike camelCase that much to avoid an _entire_ ecosystem because of it? I prefer snake_case too, but avoiding a language because of a minor convention like that seems weird.

Only language where I can stand it is Haskell. camelCase is relatively ok, but PascalCase for non-types (i.e. function names) is a big no-no :)

Re: Why I Use Nim instead of Python for Data Processing

#142

Post by elcritch »

teleforce wrote: post #134
nine_k wrote: post #78

Why do we use Python for data processing? Because we use it as a nice syntactic frontend to numpy, a large and highly optimized library written in C++ and Fortran (sic). That is, we actually don't use "Python-native" code much, and numpy is essentially APL-like array-oriented thing where e.g. you don't normally need loops. For native-language data processing, Python is slow; Nim or Julia would easily outperform it, w…

Please add D language to the mix as well. Interestingly, you can simply replace Nim with D in the blog article and most of the contents will still make sense! The funny thing is that Nim and Julia libraries are still wrapping Fortran numerical library while D has beaten the old and trusted Fortran library in its home turf five years back: http://blog.mir.dlang.io/glas/benchmark/openblas/2016/09/23/...

There’s been a tremendous amount of work optimizing blas _and_ ensuring it’s numerically stable. Julia made a good choice to use blas first. Though it’s good to see new native implementations.

For Nim, there’s also NimTorch which is interesting in that it builds on Nim’s C++ target to generate native PyTorch code. Even Python is technically a second class citizen for the C++ code. Most ML libraries are C++ all the way down.

https://github.com/sinkingsugar/nimtorch

Re: Why I Use Nim instead of Python for Data Processing

#143

Post by jimbob45 »

elcritch wrote: post #40
piqufoh wrote: post #31

Python sometimes runs slowly, because it's not designed to run fast. It's designed to be readable and easy to write, which in turn makes developing python faster. It's a compromise, but I always prioritise _my_ time over my computers time, so if I can write something quickly and just go and get a coffee while it runs - I will do that. I won't spend twice as long writing a single-run script just because it'll finish b…

That’s where Nim can shine. For simple scripts both Python and Nim are about as easy to write. But the Nim version usually runs a lot faster. Static types help for basic data munging when you haven’t used a script for months to get up to speed and make tweaks.

Sadly, I think you’re spot-on about Nim’s future as the realization of the alternative timeline where Python didn’t make several stupid design choices (e.g. the GIL, Python 3).

It’s a shame because I think Nim has some neat features that allow it to present as a serious competitor to Rust but it will ultimately have to compete against Python instead to secure its niche.

Re: Why I Use Nim instead of Python for Data Processing

#144

Post by someguy101010 »

m_mueller wrote: post #21
amyjess wrote: post #14

Earlier quoted context omitted.

> It's primarily a testament to how simply mind bogglingly slow Python is outside of its optimised numerical science ecosystem. From my experience in using Python at my last job, I'll also add that Python is decent at tasks that aren't CPU-bound. I wrote a lot of scripts that polled large amounts of network devices for information and then did something with it (typically upsert the data into a database, either via d…

I agree with your entire post but , and I‘m saying this as a fulltime python dev, there‘s often a point where it starts being bothersome, and that usually comes only later in the lifecycle of an application after it had some organic growth. Some day e.g. a sales manager comes down to your lair and asks you if you couldn‘t just also parse this little 200MB Excel spreadsheet after it came over the network such that you…

Is there a problem with binding c or c++ to your python project in these situations?

Re: Why I Use Nim instead of Python for Data Processing

#146

Post by soundmasterj »

This is reasonably idiomatic Python and 10x faster than the implementation in the original post:

  with open("orthocoronavirinae.fasta") as f:
      text = ''.join((line.rstrip() for line in f.readlines() if not line.startswith('>')))
      gc = text.count('G') + text.count('C')
      total = len(text)
Or if you want to be explicit, this is just as fast (and might scale better for particularly long genomes):

  gc = 0
  total = 0
  
  with open("orthocoronavirinae.fasta") as f:
      for line in f.readlines():
          if not line.startswith('>'):
              line = line.rstrip()
              gc += line.count('C') + line.count('G')
              total += len(line)
I didn't test Nim but the author reports Nim is 30x faster than his Python implementation, so mine would be about 3x slower than his Nim.

Re: Why I Use Nim instead of Python for Data Processing

#147

Post by PartiallyTyped »

CraigJPerry wrote: post #81
zmmmmm wrote: post #8

It's primarily a testament to how simply mind bogglingly slow Python is outside of its optimised numerical science ecosystem. Which also why I don't use it that much, because while numerical analysis is a big part of what I do, so is what I would call "symbolic manipulation" and unless you go to quite some effort to transform every problem into a numerical one, Python is just awful at that. But Nim is only one of a w…

The thing with Python is it's usually pretty easy to optimise quite impressively. E.g. random example: Sprinkle some cdef's in your python and suddenly you're faster than c++ https://github.com/luizsol/PrimesResult https://github.com/PlummersSoftwareLLC/Primes/blob/drag-race... 25.8 seconds down to 1.5

There is also numba, which is very impressive in its own right, and also pypy, which supports features up to Python 3.7.

Some may consider Jax, and its XLA compiler, but unless you require gradients, numba will be significantly faster, an instance of this is available here [1].

XLA runs on a higher level than LLVM and therefore can't achieve the same optimizations as numba does using the latter. IIRC numba also has a Python to Cuda compiler, which is also very impressive.

[1] https://github.com/scikit-hep/iminuit/blob/develop/doc/tutor...

Re: Why I Use Nim instead of Python for Data Processing

#148

Post by bmicraft »

csmpltn wrote: post #22
amyjess wrote: post #14

Earlier quoted context omitted.

> It's primarily a testament to how simply mind bogglingly slow Python is outside of its optimised numerical science ecosystem. From my experience in using Python at my last job, I'll also add that Python is decent at tasks that aren't CPU-bound. I wrote a lot of scripts that polled large amounts of network devices for information and then did something with it (typically upsert the data into a database, either via d…

> "I'll also add that Python is decent at tasks that aren't CPU-bound" IO-bound tasks are almost by definition outside of your Python application's control. You yield control to the system to execute the actual task, and from that point on - you're no longer in control of how long the task will take to complete. In other words, Python "being fast" by waiting on a Socket to complete receiving data isn't a particularil…

The main point (I think) was that python is a viable language for many use cases that are not processing intensive, while also being very easy and quick to write which is often the most important thing.

Re: Why I Use Nim instead of Python for Data Processing

#149

Post by epidemian »

soundmasterj wrote: post #146

This is reasonably idiomatic Python and 10x faster than the implementation in the original post: with open("orthocoronavirinae.fasta") as f: text = ''.join((line.rstrip() for line in f.readlines() if not line.startswith('>'))) gc = text.count('G') + text.count('C') total = len(text) Or if you want to be explicit, this is just as fast (and might scale better for particularly long genomes): gc = 0 total = 0 with open("…

I think this is missing the point of the article.

Yes, you can implement a faster Python version, but notice also:

* This faster version is reading all the file into memory (except comment lines). The article mentions the data being 150MB, which should fit in memory, but for larger datasets, this approach would be unfeasible

* The faster version is actually delegating a lot of work to Python's C internals by using text.count('G'). All the internal looping and comparisons is done in C, while on the original version, goes through Python

So yes, you can definitely write faster Python by delegating most of the work to C.

The point of the article is not about how to optimize Python, but about how given almost identical implementations in Python and Nim, Nim can outperform Python by 1 or 2 orders of magnitude without resorting to use C internals for basic things like looping or comparing characters.

Re: Why I Use Nim instead of Python for Data Processing

#150

Post by Mikeb85 »

DeathArrow wrote: post #41

While Nim is for certain interesting and even pleasant to write code in, its small user base and environment discourage people to use it. I don't write code only for myself. How would I convince my employer to let me use Nim instead of a better known language? And even I would convince my employer, if we want to start a new project how could we find programmers well-versed in Nim? And even id we can find those people…

> And even I would convince my employer, if we want to start a new project how could we find programmers well-versed in Nim?

Nim's easy to learn if you have any experience with any compiled language and can understand anything along the line of C#, Kotlin or Python syntax. Also because it compiles to C and JS it makes it easy to add it to a project incrementally in many cases.

Post reply on HN