Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Why I Use Nim instead of Python for Data Processing

benjamindlee.com

Post reply on HN
Print view View original

81–90 of 183 posts

Re: Why I Use Nim instead of Python for Data Processing

#81

Post by CraigJPerry »

zmmmmm wrote: post #8

It's primarily a testament to how simply mind bogglingly slow Python is outside of its optimised numerical science ecosystem. Which also why I don't use it that much, because while numerical analysis is a big part of what I do, so is what I would call "symbolic manipulation" and unless you go to quite some effort to transform every problem into a numerical one, Python is just awful at that. But Nim is only one of a w…

The thing with Python is it's usually pretty easy to optimise quite impressively.

E.g. random example:

Sprinkle some cdef's in your python and suddenly you're faster than c++

https://github.com/luizsol/PrimesResult

https://github.com/PlummersSoftwareLLC/Primes/blob/drag-race...

25.8 seconds down to 1.5

Re: Why I Use Nim instead of Python for Data Processing

#82

Post by baq »

otherme123 wrote: post #20
amyjess wrote: post #14

Earlier quoted context omitted.

> It's primarily a testament to how simply mind bogglingly slow Python is outside of its optimised numerical science ecosystem. From my experience in using Python at my last job, I'll also add that Python is decent at tasks that aren't CPU-bound. I wrote a lot of scripts that polled large amounts of network devices for information and then did something with it (typically upsert the data into a database, either via d…

You have to be reaaaaaally slow to be beaten by a network. Also, touched by the OP, if it takes 3 hours to write some code that in Python takes only 1, or if the compile times are huge, Python can beat other languages in speed (that edge fades when the same program is used over and over again). But as demonstrated, Nim is fast to write and fast to compile, so Python has little edge. Just it's huge ecosystem.

> Just it's huge ecosystem.

self-contradiction at its best. kindly be reminded the same advantage was the only thing that kept Java alive for so long until it finally started to enter the XXI century a couple years ago.

you just can't discount an ecosystem, especially if its huge.

Re: Why I Use Nim instead of Python for Data Processing

#83

Post by baq »

yellow_lead wrote: post #27
amyjess wrote: post #14

Earlier quoted context omitted.

> It's primarily a testament to how simply mind bogglingly slow Python is outside of its optimised numerical science ecosystem. From my experience in using Python at my last job, I'll also add that Python is decent at tasks that aren't CPU-bound. I wrote a lot of scripts that polled large amounts of network devices for information and then did something with it (typically upsert the data into a database, either via d…

As other commenters point out, how can a language be fast in a way besides something CPU bound? You are saying it is fast when it's not doing anything. Not sure I understand.

it's a shorthand for 'fast enough'. is it slow if nobody would notice if it was 10x faster?

Re: Why I Use Nim instead of Python for Data Processing

#84

Post by _dain_ »

santiagobasulto wrote: post #79
_dain_ wrote: post #74

Earlier quoted context omitted.

>total = len(list(gc_lines)) That won't work properly; you've already exhausted the gc_lines generator in the previous line.

True, you'd need to re create the generator expression. Still, the other implementation seemed too naive.

There's probably some itertools trick to do it with only one iteration.

EDIT: you can do it with functools.reduce and a generator of tuples:

    from functools import reduce

    with open("orthocoronavirinae.fasta") as f:
        lines = (line.strip() for line in f if not line.startswith(">"))
        sums = ((len(line), sum(1 for ch in line if ch in "CG")) for line in lines)
        total, gc = reduce(lambda x, y: (x[0] + y[0], x[1] + y[1]), sums)
Besides, I really don't think that any of our solutions will be that much faster than the one in the OP. All of them are using lazy iteration on the file object, just written differently. The differences amount to micro-optimizations. The real way to speed it up would be to use something like pandas, where the loading and summing calls into fast C implementations.

Re: Why I Use Nim instead of Python for Data Processing

#85

Post by nerdponx »

amyjess wrote: post #14
zmmmmm wrote: post #8

It's primarily a testament to how simply mind bogglingly slow Python is outside of its optimised numerical science ecosystem. Which also why I don't use it that much, because while numerical analysis is a big part of what I do, so is what I would call "symbolic manipulation" and unless you go to quite some effort to transform every problem into a numerical one, Python is just awful at that. But Nim is only one of a w…

> It's primarily a testament to how simply mind bogglingly slow Python is outside of its optimised numerical science ecosystem. From my experience in using Python at my last job, I'll also add that Python is decent at tasks that aren't CPU-bound. I wrote a lot of scripts that polled large amounts of network devices for information and then did something with it (typically upsert the data into a database, either via d…

You might want to look into async (asyncio or anyio) instead of or in addition to threads for network-heavy code. Async coroutines I find can be much easier to debug and develop than OS-threaded code.

Re: Why I Use Nim instead of Python for Data Processing

#86

Post by nerdponx »

zmmmmm wrote: post #8

It's primarily a testament to how simply mind bogglingly slow Python is outside of its optimised numerical science ecosystem. Which also why I don't use it that much, because while numerical analysis is a big part of what I do, so is what I would call "symbolic manipulation" and unless you go to quite some effort to transform every problem into a numerical one, Python is just awful at that. But Nim is only one of a w…

I often recommend PyPy for "non-numerical" data processing when performance matters.

Also, "awful" is too harsh. Probably 90% of Python code just doesn't need to be faster than it is.

Re: Why I Use Nim instead of Python for Data Processing

#87

Post by brabel »

benjamin-lee wrote: post #35
brabel wrote: post #26

> Nim compilation process took an additional 702 ms That's horrifyingly slow for a compiler. The author mentioned "modern languages look like Python but run as fast as C", which is a common promise those languages make that never really materialize except for a few very happy path cases they heavily optmised the language for. Julia, for example, makes this promise too, but compiles even slower than that and takes rid…

I didn't post it because it's quite big (150M) but readily available from the NCBI Virus portal [1]. I would love to see how well other languages compete both for speed and simplicity. [1] https://www.ncbi.nlm.nih.gov/labs/virus/vssi/#/virus?SeqType...

I couldn't get your 150M file, so I used one of the smaller files I could get by clicking on the first set shown in the table (the FASTA file was only 30KB) and duplicated it until it was around 150MB.

Here's a comparison with Common Lisp:

~/fasta-dna $ time python3 run.py

0.3797277865097147

21.828 secs

~/fasta-dna $ time sbcl --script run.lisp

0.37972778

2.415 secs

~/fasta-dna $ ls -al nc_045512.2.fasta

-rw-r--r-- 1 156095639 2021-09-25 11:15 nc_045512.2.fasta

So, almost as fast as Nim (the time includes compilation time)?

Here's the Common Lisp code:

    (with-open-file (in "nc_045512.2.fasta")
      (loop for line = (read-line in nil)
            while line
            with gc = 0 with total = 0 do
              (unless (eql (aref line 0) #\>)
                (loop for i from 0 below (length line)
                      for ch = (char line i) do
                        (setf total (1+ total))
                        (when (or (eql ch #\C) (eql ch #\G))
                          (setf gc (1+ gc)))))
            finally (format t "~f~%" (/ gc total))))
With a top-level function and some type declarations it could run even faster, I think.

EDIT: compiling the Lisp code to FASL and annotating the types brings the total runtime to 2.0 seconds. Running it from source increases the time very slightly, to 2.08 seconds, showing how the SBCL compiler is incredibly fast. Taking 0.7 seconds to compile a few lines of code is crazy, imagine when your project grows to many thousands of lines.

The Lisp code still can't really match Nim, which is really C at runtime, in speed when excluding compile-time, but if you need a scripting language, CL is great (specially when used with the REPL and SLIME).

Re: Why I Use Nim instead of Python for Data Processing

#88

Post by IshKebab »

elcritch wrote: post #32
IshKebab wrote: post #16

Earlier quoted context omitted.

Yeah should really be "Why I don't use Python for Data Processing". I would consider Typescript as an alternative too which I'm sure would get a similar speedup. Also I don't know how anyone could design a language in the 21st century and make basic mistakes like this: > Nim treats identifiers as equal if they are the same after removing capitalization (except for the first letter) and underscore, which means that yo…

> Yeah should really be "Why I don't use Python for Data Processing". Not entirely. Nim‘s benefit here is that it’s superficially similar enough to Python that’s it’s easy for people from that world to pickup and start using Nim. > Also I don't know how anyone could design a language in the 21st century and make basic mistakes like this: > If that's any indication of the sanity of the rest of Nim then I'd say steer w…

> It’s intent is to allow a given codebase to maintain a consistent style (eg camel vs snake) even when making use of upstream libraries that use different styles.

This doesn't make sense. For an entirely new language you can just have the entire ecosystem use the same style, e.g. like Rust does. Or even Python!

Re: Why I Use Nim instead of Python for Data Processing

#89

Post by baggiponte »

santiagobasulto wrote: post #37

I don't doubt Nim, looks like a great language. But that is just an awful Python implementation. I'd do it in this way: lines = (line for line in lines("orthocoronavirinae.fasta") if not line.startswith(">")) gc_lines = (1 if ('G' in line or 'C' in line) else 0 for line in lines) gc = sum(gc_lines) total = len(list(gc_lines)) # Alternatively, a more "memory efficient" total would be: total = sum(1 for _ in lines) Edi…

The (…) return an iterator, right?

Re: Why I Use Nim instead of Python for Data Processing

#90

Post by nerdponx »

piqufoh wrote: post #31

Python sometimes runs slowly, because it's not designed to run fast. It's designed to be readable and easy to write, which in turn makes developing python faster. It's a compromise, but I always prioritise _my_ time over my computers time, so if I can write something quickly and just go and get a coffee while it runs - I will do that. I won't spend twice as long writing a single-run script just because it'll finish b…

You can also drop in PyPy and get a significant speedup on "loopy" string processing tasks with no changes in your code.
Post reply on HN