Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Why I Use Nim instead of Python for Data Processing

benjamindlee.com

Post reply on HN
Print view View original

151–160 of 183 posts

Re: Why I Use Nim instead of Python for Data Processing

#151

Post by CraigJPerry »

Jensson wrote: post #117
CraigJPerry wrote: post #81

Earlier quoted context omitted.

The thing with Python is it's usually pretty easy to optimise quite impressively. E.g. random example: Sprinkle some cdef's in your python and suddenly you're faster than c++ https://github.com/luizsol/PrimesResult https://github.com/PlummersSoftwareLLC/Primes/blob/drag-race... 25.8 seconds down to 1.5

I've tried cython and it isn't competitive with C in real world cases. Cherry picking a case where it is competitive doesn't help your case. Its performance tend to be around the level of Java, another language people say is competitive or faster than C++ but in practice C++ is still twice faster than Java in most cases. Still, getting Java level performance out of python is a huge improvement and should be enough fo…

Ahh my point was more that you can score a 17x performance boost with minimal effort rather than any absolute performance advantage over c++.

For a c++ comparison python would be much better pointing out the productivity advantage it has over the notoriously low productivity of c++ development - rather than competing on execution performance.

Re: Why I Use Nim instead of Python for Data Processing

#152

Post by adenozine »

ignorem3 wrote: post #145

So many python speed apologists. Yes, you can pour over your python code and given enough time and effort you can eek out another 25% improvement, but it's still much slower than the alternatives.

Python gets WAY too much work done in WAY too many fields to just handwave away "waaah, it's faster to use blah-lang"

Re: Why I Use Nim instead of Python for Data Processing

#153

Post by PartiallyTyped »

ignorem3 wrote: post #145

So many python speed apologists. Yes, you can pour over your python code and given enough time and effort you can eek out another 25% improvement, but it's still much slower than the alternatives.

Counter argument, how many times does one need to do data processing and what's the most expensive process in the equation?

The answer for the latter is programmer time, and some things can be scaled easily using `joblib`, or `dask`. Now, it isn't as trivial as importing parallel iterators with rust and changing `.into_iter` to `.into_par_iter`, but still needs less time, and once it is done, I don't need to think about it again.

Re: Why I Use Nim instead of Python for Data Processing

#154

Post by cb321 »

brabel wrote: post #87
benjamin-lee wrote: post #35

Earlier quoted context omitted.

I didn't post it because it's quite big (150M) but readily available from the NCBI Virus portal [1]. I would love to see how well other languages compete both for speed and simplicity. [1] https://www.ncbi.nlm.nih.gov/labs/virus/vssi/#/virus?SeqType...

I couldn't get your 150M file, so I used one of the smaller files I could get by clicking on the first set shown in the table (the FASTA file was only 30KB) and duplicated it until it was around 150MB. Here's a comparison with Common Lisp: ~/fasta-dna $ time python3 run.py 0.3797277865097147 21.828 secs ~/fasta-dna $ time sbcl --script run.lisp 0.37972778 2.415 secs ~/fasta-dna $ ls -al nc_045512.2.fasta -rw-r--r-- 1…

@brabel - The Nim compiler actually builds a relatively large `system` package every time. (They are also working on speeding up compiles.) So, compile time does not scale as badly as you think. E.g., you might have to 50..100x the "user level" source code to double the time.

Also, @benjamin-lee this version of the Nim program is a bit lower level, but probably much faster:

    import memfiles as mf
    var gc = 0
    var total = 0

    var f = mf.open("orthocoronavirinae.fasta")
    for line in memSlices(f):
        let n = line.size
        let cs = cast[cstring](line.data)
        if n > 0 and cs[0] == '>': # ignore comment lines
            continue
        for i in 0 ..
Compile with -d:danger and so on, of course. { On a small 30kB test file I got about a 1.7x speed-up over that of the blog post. I also could not find the 150 MB file. Multiplying up the tiny 30 KB file like @brabel, I got only a 1.25x speed-up down to 0.5 seconds. So, might not be worth the low levelness, but a real file might tilt more towards the 1.7x end. }

Re: Why I Use Nim instead of Python for Data Processing

#155

Post by akho »

rsyring wrote: post #122
m_mueller wrote: post #21

Earlier quoted context omitted.

I agree with your entire post but , and I‘m saying this as a fulltime python dev, there‘s often a point where it starts being bothersome, and that usually comes only later in the lifecycle of an application after it had some organic growth. Some day e.g. a sales manager comes down to your lair and asks you if you couldn‘t just also parse this little 200MB Excel spreadsheet after it came over the network such that you…

> there‘s often a point where it starts being bothersome My team and I have been using Python for web, scripting, and ETL development since 2007. I don't recall the last time Python wasn't "fast enough" for anything I needed to do. I'm sure it's legitimately too slow for plenty of use cases and classes of programming domains. But for a general purpose language that makes our developers incredibly productive (which is…

Does Python give you an actual productivity edge? I do not see how.

Python developers are easier to hire (certainly than Nim), have a monolanguage experience (meaning less onboarding), but are not magically more productive. Developer availability is still a great reason to choose Python.

Re: Why I Use Nim instead of Python for Data Processing

#156

Post by mark_l_watson »

Nice writeup, glad the author has a language and environment that they like.

Python has never been one of my favorite languages, but easy support in Google Colab, AWS SageMaker, etc. as well as most of my professional deep learning work using TensorFLow + Keras, it makes Python a go-to language for me. If you want a Lisp syntax on top of Python, you can try Hy (and get a free copy of my Hy book at https://leanpub.com/hy-lisp-python by setting the price to $0.00).

That said, for unpaid experiments I like Julia + Flux, which also solves the author's preference to avoid slow programming languages. Julia is really a nice language but no one has ever paid me to use it.

Re: Why I Use Nim instead of Python for Data Processing

#157

Post by elcritch »

jimbob45 wrote: post #143
elcritch wrote: post #40

Earlier quoted context omitted.

That’s where Nim can shine. For simple scripts both Python and Nim are about as easy to write. But the Nim version usually runs a lot faster. Static types help for basic data munging when you haven’t used a script for months to get up to speed and make tweaks.

Sadly, I think you’re spot-on about Nim’s future as the realization of the alternative timeline where Python didn’t make several stupid design choices (e.g. the GIL, Python 3). It’s a shame because I think Nim has some neat features that allow it to present as a serious competitor to Rust but it will ultimately have to compete against Python instead to secure its niche.

Oh yes, Nim definitely feels like an alternate reality where Python 2 became static and dumped some poor design choices.

Well I believe there's room between Rust and Python where Nim can grow. It made the TIOBE top 50 lately even. Likely it can eat enough market share from the edges of both Rust & Python to become more well known (more libs, tools, etc).

Rust is fantastic but tedious to program (to me at least) and it's community focuses on more formal type traits, etc making "scripting" trickier. Python is great for a mix of quick scripts, web dev, and data science but it's slow enough (and getting complex enough!) for many to want something faster and more stable yet still easy to write. Nim lives in between them and is more enjoyable to write than either for many. Also, Nim _could_ add Rust as a backend target and be relevant even if Rust displaces C/C++. ;)

Nim is also great for embedded systems too! I've been using it a fair bit and it's really nice [1]. There's a lot of room to grow in that field.

1: https://github.com/elcritch/nesper

Re: Why I Use Nim instead of Python for Data Processing

#158

Post by user5994461 »

soundmasterj wrote: post #146

This is reasonably idiomatic Python and 10x faster than the implementation in the original post: with open("orthocoronavirinae.fasta") as f: text = ''.join((line.rstrip() for line in f.readlines() if not line.startswith('>'))) gc = text.count('G') + text.count('C') total = len(text) Or if you want to be explicit, this is just as fast (and might scale better for particularly long genomes): gc = 0 total = 0 with open("…

One liner to count gc, without buffering.

    import io
    f = io.StringIO(
    """
    AB
    CD
    EF
    GH
    """
    )

    total = sum(map(lambda s: 0 if s[0]==">" else s.count('G') + s.count('C'), f.readlines()))

    print(total)

Re: Why I Use Nim instead of Python for Data Processing

#159

Post by mattbillenstein »

nerdponx wrote: post #86
zmmmmm wrote: post #8

It's primarily a testament to how simply mind bogglingly slow Python is outside of its optimised numerical science ecosystem. Which also why I don't use it that much, because while numerical analysis is a big part of what I do, so is what I would call "symbolic manipulation" and unless you go to quite some effort to transform every problem into a numerical one, Python is just awful at that. But Nim is only one of a w…

I often recommend PyPy for "non-numerical" data processing when performance matters. Also, "awful" is too harsh. Probably 90% of Python code just doesn't need to be faster than it is.

I'd guess closer to 99%...

Re: Why I Use Nim instead of Python for Data Processing

#160

Post by ghostly_s »

choneone wrote: post #64
ur-whale wrote: post #11

what's up we the all the shouting in the code ?

It's because the `font-feature-settings` of the main font leak to the code font. The feature 'case' turns the code all uppercase.

Ha, I thought the author was intentionally writing the python in all-caps to highlight the similarity in syntax to the alternative language (not being familiar with Nim, I figured it must be conventional there). I was going to post a comment expressing my surprise that was even legal.
Post reply on HN