Why I Use Nim instead of Python for Data Processing
benjamindlee.com
Why I Use Nim instead of Python for Data Processing
1–10 of 183 posts
Re: Why I Use Nim instead of Python for Data Processing
#2While I trust the author on this, I don’t think DNA datasets and string analysis was a great example.
One of the big, big things for improving performance on DNA analysis of ANY kind is converting these large text files into binary (4 letters easily converts to 2 bit encoding) and massively improves basically any analysis you’re trying to do.
Not only does it compress your dataset (2 bits vs 16 bits), it allows absurdly faster numerical libraries to be used in lieu of string methods.
There’s no real point in showing off that a compiled language is faster at doing something the slow way…
Re: Why I Use Nim instead of Python for Data Processing
#3While I trust the author on this, I don’t think DNA datasets and string analysis was a great example. One of the big, big things for improving performance on DNA analysis of ANY kind is converting these large text files into binary (4 letters easily converts to 2 bit encoding) and massively improves basically any analysis you’re trying to do. Not only does it compress your dataset (2 bits vs 16 bits), it allows absur…
You make a fair point that using optimized numerical libraries instead of string methods will be ridiculously fast because they're compiled anyway. For example, scikit-bio does just this for their reverse complement operation [1]. However, they use an 8 bit representation since they need to be able to represent the extended IUPAC notation for ambiguous bases, which includes things like the character N for "aNy" nucleotide [2]. One could get creative with a 4 bit encoding and still end up saving space (assuming you don't care about the distinction between upper versus lowercase characters in your sequence [3]). Or, if you know in advance your sequence is unambiguous (unlikely in DNA sequencing-derived data) you could use the 2 bit encoding. When dealing with short nucleotide sequences, another approach is to encode the sequence as an integer. I would love to see a library—Python, Nim, or otherwise—that made using the most efficient encoding for a sequence transparent to the developer.
[1] https://github.com/biocore/scikit-bio/blob/b470a55a8dfd054ae...
[2] https://en.wikipedia.org/wiki/Nucleic_acid_notation
[3] https://bioinformatics.stackexchange.com/questions/225/upper...
Re: Why I Use Nim instead of Python for Data Processing
#4While I trust the author on this, I don’t think DNA datasets and string analysis was a great example. One of the big, big things for improving performance on DNA analysis of ANY kind is converting these large text files into binary (4 letters easily converts to 2 bit encoding) and massively improves basically any analysis you’re trying to do. Not only does it compress your dataset (2 bits vs 16 bits), it allows absur…
You make a fair point that using optimized numerical libraries instead of string methods will be ridiculously fast because they're compiled anyway. For example, scikit-bio does just this for their reverse complement operation [1]. However, they use an 8 bit representation since they need to be able to represent the extended IUPAC notation for ambiguous bases, which includes things like the character N for "aNy" nucle…
Yeah, this is why my comment led with “I trust the author”…
I’m surprised you need the full 4 bits to deal with ambiguous bases, but it probably makes sense at some lower level I don’t understand.
Re: Why I Use Nim instead of Python for Data Processing
#5Earlier quoted context omitted.
You make a fair point that using optimized numerical libraries instead of string methods will be ridiculously fast because they're compiled anyway. For example, scikit-bio does just this for their reverse complement operation [1]. However, they use an 8 bit representation since they need to be able to represent the extended IUPAC notation for ambiguous bases, which includes things like the character N for "aNy" nucle…
Yeah, this is why my comment led with “I trust the author”… I’m surprised you need the full 4 bits to deal with ambiguous bases, but it probably makes sense at some lower level I don’t understand.
This is because there's four bases and each can either be included or excluded from a given combination. So there are 4*2 = 16 combinations each of which with their own letter. In all honesty, these are pretty rarely used in practice these days except for N (any base) although they do sometimes show up when representing consensus sequences.
Re: Why I Use Nim instead of Python for Data Processing
#6While I trust the author on this, I don’t think DNA datasets and string analysis was a great example. One of the big, big things for improving performance on DNA analysis of ANY kind is converting these large text files into binary (4 letters easily converts to 2 bit encoding) and massively improves basically any analysis you’re trying to do. Not only does it compress your dataset (2 bits vs 16 bits), it allows absur…
Aren't the reads emitting a set of size greater than 4 bases per position, with a wildcard or "?" perhaps one option?
(As in GATTACA might be read as is, but might be read as GAT?ACA.)
Still that's a minimal of 3 bits versus much longer.
[Edit : i see another commenter with the same observation, more thoroughly explained! ]
Re: Why I Use Nim instead of Python for Data Processing
#7does nim have anything like scikit yet?
Re: Why I Use Nim instead of Python for Data Processing
#8It's primarily a testament to how simply mind bogglingly slow Python is outside of its optimised numerical science ecosystem. Which also why I don't use it that much, because while numerical analysis is a big part of what I do, so is what I would call "symbolic manipulation" and unless you go to quite some effort to transform every problem into a numerical one, Python is just awful at that.
But Nim is only one of a whole suite of languages that easily cruise to a 10x performance win over Python. And that isn't counting multicore - if you count that you quickly get to a 100x improvement.
Personally I use Groovy for much of what I do for similar reasons (which is somewhat unusual) but its just a placeholder for "use anything except python".
Re: Why I Use Nim instead of Python for Data Processing
#9TLDR: Because Python is slow
Yes, that is the achilles heel of Python.
I am always torn between Python and PHP for new projects because of this.
The Python Syntax plus its import system are huge advantages over PHP. On the other hand, you suffer a 6x slowdown if you go with Python. Decisions decisions. I so dearly wish I could have the good parts of both worlds.
Re: Why I Use Nim instead of Python for Data Processing
#10How does it compare to Julia? Anyone with experience in both Nim and Julia?