Live data from Hacker News

Why I Use Nim instead of Python for Data Processing

benjamindlee.com

161–170 of 183 posts

Re: Why I Use Nim instead of Python for Data Processing

#161

This is reasonably idiomatic Python and 10x faster than the implementation in the original post: with open("orthocoronavirinae.fasta") as f: text = ''.join((line.rstrip() for line in f.readlines() if not line.startswith('>'))) gc = text.count('G') + text.count('C') total = len(text) Or if you want to be explicit, this is just as fast (and might scale better for particularly long genomes): gc = 0 total = 0 with open("…

One liner to count gc, without buffering. import io f = io.StringIO( """ AB CD EF GH """ ) total = sum(map(lambda s: 0 if s[0]==">" else s.count('G') + s.count('C'), f.readlines())) print(total)

And reading the file as binary. There's a lesson about the overhead of unicode strings here ;)

Your first example takes 3.1 seconds, my previous comment takes 2.3 seconds, this one takes 1.4 seconds.

    start = time.perf_counter()

    with open("orthocoronavirinae.fasta", "rb") as f:
    total = sum(map(lambda s: 0 if s[0]==65 else s.count(b"G") + s.count(b"C"), f.readlines()))
    
    end = time.perf_counter()
    
    print(total, " total")
    print(end-start, " seconds")

Re: Why I Use Nim instead of Python for Data Processing

#162

The author makes a fair point, however, that is a rather non optimal implementation in Python. You likely could use chunked Pandas to speed up or the code, or at least replace some of the for loops with a list comprehension syntax. However, in any case I would never replace Python with Nim as it is too niche of a language and you would struggle with recruiting. I could consider Julia if it's popularity keeps growing.…

Agreed. I am certainly inclined to believe that Nim is a better language than Python, but it's not so much better thlo justify moving off of the ecosystem.

Re: Why I Use Nim instead of Python for Data Processing

#163

Earlier quoted context omitted.

I agree with your entire post but , and I‘m saying this as a fulltime python dev, there‘s often a point where it starts being bothersome, and that usually comes only later in the lifecycle of an application after it had some organic growth. Some day e.g. a sales manager comes down to your lair and asks you if you couldn‘t just also parse this little 200MB Excel spreadsheet after it came over the network such that you…

Is there a problem with binding c or c++ to your python project in these situations?

The whole build gets quite complex. Numba is interesting but also still a bit limited in what it supports. There are no silver bullets and sometimes you can find a good fit, but IMO the JVM or C#/.net overall have a better story when it comes to write performance critical applications (but don’t need to get the last 20% such as HFT or HPC sims), as you can also find people on the market who understand how to do and maintain that more easily. There are tons of python devs but in my experience at least on European market their understanding of underlying systems tends to be limited.

Re: Why I Use Nim instead of Python for Data Processing

#164

Earlier quoted context omitted.

I agree with your entire post but , and I‘m saying this as a fulltime python dev, there‘s often a point where it starts being bothersome, and that usually comes only later in the lifecycle of an application after it had some organic growth. Some day e.g. a sales manager comes down to your lair and asks you if you couldn‘t just also parse this little 200MB Excel spreadsheet after it came over the network such that you…

> there‘s often a point where it starts being bothersome My team and I have been using Python for web, scripting, and ETL development since 2007. I don't recall the last time Python wasn't "fast enough" for anything I needed to do. I'm sure it's legitimately too slow for plenty of use cases and classes of programming domains. But for a general purpose language that makes our developers incredibly productive (which is…

YMMV. My background is in financial applications and there it’s a quite common problem popping up, especially (but not exclusively) in analytics.

Re: Why I Use Nim instead of Python for Data Processing

#165
post #78

Why do we use Python for data processing? Because we use it as a nice syntactic frontend to numpy, a large and highly optimized library written in C++ and Fortran (sic). That is, we actually don't use "Python-native" code much, and numpy is essentially APL-like array-oriented thing where e.g. you don't normally need loops. For native-language data processing, Python is slow; Nim or Julia would easily outperform it, w…

Please add D language to the mix as well. Interestingly, you can simply replace Nim with D in the blog article and most of the contents will still make sense! The funny thing is that Nim and Julia libraries are still wrapping Fortran numerical library while D has beaten the old and trusted Fortran library in its home turf five years back: http://blog.mir.dlang.io/glas/benchmark/openblas/2016/09/23/...

Not necessarily true with Julia. Many libraries like DifferentialEquations.jl are Julia all of the way down because the pure Julia BLAS tools outperform OpenBLAS and MKL in certain areas. For example see:

https://github.com/YingboMa/RecursiveFactorization.jl/pull/2...

So a stiff ODE solve is pure Julia, LU-factorizations and all. This is what allows it to outperform the common C and Fortran libraries very consistently. See https://benchmarks.sciml.ai/html/MultiLanguage/wrapper_packa... and https://benchmarks.sciml.ai/html/Bio/BCR.html

Re: Why I Use Nim instead of Python for Data Processing

#167
I'll probably always dislike Python. I do like Nim, but for strictly data processing I'd take R over either in a heartbeat. R has too many libraries, its semantics are perfect for data processing, RStudio is too nice and while pure R is slow as shit, in practice it's fast because it's basically a scripting language for a bunch of C and Fortran bits that are doing the real work.

If I were writing something from scratch that dealt with data, I would probably use Nim though. It's super easy to write something fast in and is more pleasant than pretty much any other compiled language.

Re: Why I Use Nim instead of Python for Data Processing

#168

Earlier quoted context omitted.

The compiler might not be sensitive to style, but *I* am.

I guess the best thing would be having a formatter that can switch the style for you locally. I don’t think Nim has this currently, would be neat if they did.

I don't think most people care about a specific style, as long as it's consistent. That's why I love Black, we all disagree with it equally, but each for different reasons.

Re: Why I Use Nim instead of Python for Data Processing

#169
post #8

It's primarily a testament to how simply mind bogglingly slow Python is outside of its optimised numerical science ecosystem. Which also why I don't use it that much, because while numerical analysis is a big part of what I do, so is what I would call "symbolic manipulation" and unless you go to quite some effort to transform every problem into a numerical one, Python is just awful at that. But Nim is only one of a w…

> mind bogglingly slow Python is outside of its optimised numerical science ecosystem

Granted, but inside its optimised numerical science ecosystem, Python is, in fact, fast enough. If most of your program is calls into numpy, Python will get you where you need to go. In my experience, one scalar Python math operation takes about the same amount of time as the equivalent numpy operation on a million-element array. Linked against a recent libblas, numpy will even distribute work across multiple cores. So much for the GIL.

Re: Why I Use Nim instead of Python for Data Processing

#170

Earlier quoted context omitted.

Yeah should really be "Why I don't use Python for Data Processing". I would consider Typescript as an alternative too which I'm sure would get a similar speedup. Also I don't know how anyone could design a language in the 21st century and make basic mistakes like this: > Nim treats identifiers as equal if they are the same after removing capitalization (except for the first letter) and underscore, which means that yo…

I actually use TypeScript/JavaScript a lot for this reason, especially for biological algorithms that I want to run in the browser. The developer tooling is also as good as you can hope for, especially when using VS Code. I actually wrote a circular RNA sequence deduplication algorithm in it just recently [1]. With respect to the identifier resolution in Nim, it strikes me as more of a matter of preference. Especiall…

The main beef I have with the javascript ecosystem for data analysis is lack of multicore. Yes lots of things can be solved by converting multi-core to multi-process or using other workarounds but there are a whole class of problems where shared memory access makes a huge amount of sense.
Post reply on HN