Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Why I Use Nim instead of Python for Data Processing

benjamindlee.com

Post reply on HN
Print view View original

61–70 of 183 posts

Re: Why I Use Nim instead of Python for Data Processing

#61

Post by ellimilial »

santiagobasulto wrote: post #37

I don't doubt Nim, looks like a great language. But that is just an awful Python implementation. I'd do it in this way: lines = (line for line in lines("orthocoronavirinae.fasta") if not line.startswith(">")) gc_lines = (1 if ('G' in line or 'C' in line) else 0 for line in lines) gc = sum(gc_lines) total = len(list(gc_lines)) # Alternatively, a more "memory efficient" total would be: total = sum(1 for _ in lines) Edi…

Isn’t this counting the lines with any G/C in them vs the total number of G/C literals?

Re: Why I Use Nim instead of Python for Data Processing

#62

Post by haxscramper »

Another nim & python thread that has not been mentioned yet here

https://news.ycombinator.com/item?id=28506531 - project allows creating pythonic bindings for your nim libraries pretty easily, which can be useful if you still want to write most of your toplevel code in python, but leverage nim's speed when it matters.

If you want to make your nim code even more "pythonic" there is a https://github.com/Yardanico/nimpylib, and for calling some python code from nim there is a https://github.com/yglukhov/nimpy

Re: Why I Use Nim instead of Python for Data Processing

#63

Post by cyber_kinetist »

brabel wrote: post #26

> Nim compilation process took an additional 702 ms That's horrifyingly slow for a compiler. The author mentioned "modern languages look like Python but run as fast as C", which is a common promise those languages make that never really materialize except for a few very happy path cases they heavily optmised the language for. Julia, for example, makes this promise too, but compiles even slower than that and takes rid…

Nim is actually one of the fastest to compile out of the compiled languages out there, on par with Go. Although this is a bit subjective, I think a second of compilation is good enough for light scripting tasks. (And being a statically-typed languages it catches a good chunk of errors before compilation is finished.)

Nim's advantage is that it uses a good old C compiler for the backend (which has been hyperoptimized for decades), but the frontend (transpiler) is also pretty fast. Nim's compilation speed should improve a bit when incremental compilation support is added (which would probably solve a lot of other current issues for Nim, for example better IDE tooling)

Re: Why I Use Nim instead of Python for Data Processing

#65

Post by benjamin-lee »

brabel wrote: post #54
benjamin-lee wrote: post #35

Earlier quoted context omitted.

I didn't post it because it's quite big (150M) but readily available from the NCBI Virus portal [1]. I would love to see how well other languages compete both for speed and simplicity. [1] https://www.ncbi.nlm.nih.gov/labs/virus/vssi/#/virus?SeqType...

I clicked on the big Download button and selected "all records", it downloaded over 3.5GB before I gave up... which file exactly should I use??

I'm sorry, I completely forgot that the file I used was from six months ago when I wrote the blog post (and then promptly forgot to publish it). In the last half year, the number of coronavirus sequences has increased dramatically. One thing that you could do to drop the file size down is to filter for only complete and unambiguous sequences, which drops the number down from 1.6 million to ~100k [1].

Alternatively, the exact file I used for the post is available for one week here with MD5 sum 3c33c3c4c2610f650c779291668450c9 [2]. Anyone who wants the file is free to reach out to me directly (email is on site).

[1] https://www.ncbi.nlm.nih.gov/labs/virus/vssi/#/virus?SeqType...

[2] https://file.io/nUNc7cG5i8gj

Re: Why I Use Nim instead of Python for Data Processing

#66

Post by nxpnsv »

Missing : on the first line of code. When speed mattes, people use libraries that are considerably faster than plain python. It’s these libraries that turned python so popular in data science. Giving them up maybe makes sense, but that mans a whole lot of learning and development to replace already pretty well established tools.

Re: Why I Use Nim instead of Python for Data Processing

#67

Post by qwerty1793 »

benjamin-lee wrote: post #29
IshKebab wrote: post #16

Earlier quoted context omitted.

Yeah should really be "Why I don't use Python for Data Processing". I would consider Typescript as an alternative too which I'm sure would get a similar speedup. Also I don't know how anyone could design a language in the 21st century and make basic mistakes like this: > Nim treats identifiers as equal if they are the same after removing capitalization (except for the first letter) and underscore, which means that yo…

I actually use TypeScript/JavaScript a lot for this reason, especially for biological algorithms that I want to run in the browser. The developer tooling is also as good as you can hope for, especially when using VS Code. I actually wrote a circular RNA sequence deduplication algorithm in it just recently [1]. With respect to the identifier resolution in Nim, it strikes me as more of a matter of preference. Especiall…

That's great. However the method that you use to find the canonical representative [1] is quadratic (when the string has length N, there are N rotations and for each rotation you need to check N characters to determine whether this is earlier than the best on that you have found so far). For large strings you would probably want to switch to one of the linear minimal string rotation algorithms [2], for example Booth's Algorithm.

[1] https://github.com/Benjamin-Lee/viroiddb/blob/main/scripts/c...

[2] https://en.wikipedia.org/wiki/Lexicographically_minimal_stri...

Re: Why I Use Nim instead of Python for Data Processing

#69

Post by scoopertrooper »

samuel wrote: post #68

For me the corollary of this post should be, try PyPy. You may get a 10x speedup for free.

For me the corollary of this comment should be, try reading the post. You may learn that the author did get a 10x speedup for free, but it was still 3.3x slower than Nim.

Re: Why I Use Nim instead of Python for Data Processing

#70

Post by benjamin-lee »

ellimilial wrote: post #51

A context might be useful. From what I gather, the author is a researcher in bioinformatics related field. This may indicate that they tend to work either alone or in a relatively small group. The domain is small scope data processing/manipulation, research/exploratory code, ,likely short-lived or even one-off. The progress in this context will possibly be governed by sheer processing speed (e.g. it’s unlikely anyone…

Author here. This is spot on. The majority of the code I write is either piping data around to existing tools using shell scripting and Snakemake or writing the data processing code myself when there isn't a tool that does what I need. Usually, I'm working alone or with a few other computational biologists. Many of my scripts are one-off but they have the distinct tendency of growing in complexity and scope if they are useful. That's one of the big advantages with Nim in my mind: you can write a quick and dirty script and have it be pretty fast and then go back later and optimize it to a few percent of C without having to rewrite your code in another language. In this sense, it's quite like Julia (another really good language).
Post reply on HN