Live data from Hacker News

Why I Use Nim instead of Python for Data Processing

benjamindlee.com

131–140 of 183 posts

Re: Why I Use Nim instead of Python for Data Processing

#131
post #78

Why do we use Python for data processing? Because we use it as a nice syntactic frontend to numpy, a large and highly optimized library written in C++ and Fortran (sic). That is, we actually don't use "Python-native" code much, and numpy is essentially APL-like array-oriented thing where e.g. you don't normally need loops. For native-language data processing, Python is slow; Nim or Julia would easily outperform it, w…

Apparently there's also a data processing library for Nim called Arraymancer[0] that's inspired by Numpy and PyTorch. It claims to be faster than both.

[0] https://mratsim.github.io/Arraymancer/

Re: Why I Use Nim instead of Python for Data Processing

#132

I notice that python has `rstrip` while `nim` doesn't. Python's `rstrip()` is allocating a whole new line there. Does the nim iterator skip over whitespace or something? Do those bits of code output the same thing? The presence or absence of the whitespace will affect the `total` count.

This is explained a bit further into the article:

> A nice feature of the lines function is that it automatically strips newline characters such as LF and CRLF so we no longer need to doline.rstrip().

Re: Why I Use Nim instead of Python for Data Processing

#133
post #132

I notice that python has `rstrip` while `nim` doesn't. Python's `rstrip()` is allocating a whole new line there. Does the nim iterator skip over whitespace or something? Do those bits of code output the same thing? The presence or absence of the whitespace will affect the `total` count.

This is explained a bit further into the article: > A nice feature of the lines function is that it automatically strips newline characters such as LF and CRLF so we no longer need to doline.rstrip().

Oh, nice! Missed that. Thanks.

Re: Why I Use Nim instead of Python for Data Processing

#134
post #78

Why do we use Python for data processing? Because we use it as a nice syntactic frontend to numpy, a large and highly optimized library written in C++ and Fortran (sic). That is, we actually don't use "Python-native" code much, and numpy is essentially APL-like array-oriented thing where e.g. you don't normally need loops. For native-language data processing, Python is slow; Nim or Julia would easily outperform it, w…

Please add D language to the mix as well. Interestingly, you can simply replace Nim with D in the blog article and most of the contents will still make sense!

The funny thing is that Nim and Julia libraries are still wrapping Fortran numerical library while D has beaten the old and trusted Fortran library in its home turf five years back:

http://blog.mir.dlang.io/glas/benchmark/openblas/2016/09/23/...

Re: Why I Use Nim instead of Python for Data Processing

#135
post #78

Why do we use Python for data processing? Because we use it as a nice syntactic frontend to numpy, a large and highly optimized library written in C++ and Fortran (sic). That is, we actually don't use "Python-native" code much, and numpy is essentially APL-like array-oriented thing where e.g. you don't normally need loops. For native-language data processing, Python is slow; Nim or Julia would easily outperform it, w…

Please add D language to the mix as well. Interestingly, you can simply replace Nim with D in the blog article and most of the contents will still make sense! The funny thing is that Nim and Julia libraries are still wrapping Fortran numerical library while D has beaten the old and trusted Fortran library in its home turf five years back: http://blog.mir.dlang.io/glas/benchmark/openblas/2016/09/23/...

Julia is in the process of replacing many C and Fortran numerical libraries with pure Julia implementations, because they have similar performance.

Re: Why I Use Nim instead of Python for Data Processing

#136
post #78

Why do we use Python for data processing? Because we use it as a nice syntactic frontend to numpy, a large and highly optimized library written in C++ and Fortran (sic). That is, we actually don't use "Python-native" code much, and numpy is essentially APL-like array-oriented thing where e.g. you don't normally need loops. For native-language data processing, Python is slow; Nim or Julia would easily outperform it, w…

Please add D language to the mix as well. Interestingly, you can simply replace Nim with D in the blog article and most of the contents will still make sense! The funny thing is that Nim and Julia libraries are still wrapping Fortran numerical library while D has beaten the old and trusted Fortran library in its home turf five years back: http://blog.mir.dlang.io/glas/benchmark/openblas/2016/09/23/...

>Julia libraries are still wrapping Fortran numerical library

You say that, but Julia is rapidly acquiring native numerical libraries that outperform OpenBLAS:

https://discourse.julialang.org/t/realistically-how-close-is...

Re: Why I Use Nim instead of Python for Data Processing

#137

Earlier quoted context omitted.

This is because there's four bases and each can either be included or excluded from a given combination. So there are 4*2 = 16 combinations each of which with their own letter. In all honesty, these are pretty rarely used in practice these days except for N (any base) although they do sometimes show up when representing consensus sequences.

What do you mean that each base can be included or excluded? Isn't only one extra value needed? Sort of like nil?

Because there are notations for any combination of bases. There's a way to indicate "C or G", "A or T", "C, G, or A", etc.

Re: Why I Use Nim instead of Python for Data Processing

#138
post #31

Python sometimes runs slowly, because it's not designed to run fast. It's designed to be readable and easy to write, which in turn makes developing python faster. It's a compromise, but I always prioritise _my_ time over my computers time, so if I can write something quickly and just go and get a coffee while it runs - I will do that. I won't spend twice as long writing a single-run script just because it'll finish b…

You can have Python syntax without Python slow.

And functional and static with F#.

Re: Why I Use Nim instead of Python for Data Processing

#139
post #137

Earlier quoted context omitted.

What do you mean that each base can be included or excluded? Isn't only one extra value needed? Sort of like nil?

Because there are notations for any combination of bases. There's a way to indicate "C or G", "A or T", "C, G, or A", etc.

Oh. So what's the grand total of all possible permutations of single and multiple (connected with an "or") values?

I'll also read through your links, thanks for posting them.

Re: Why I Use Nim instead of Python for Data Processing

#140
post #57

While Nim is for certain interesting and even pleasant to write code in, its small user base and environment discourage people to use it. I don't write code only for myself. How would I convince my employer to let me use Nim instead of a better known language? And even I would convince my employer, if we want to start a new project how could we find programmers well-versed in Nim? And even id we can find those people…

>if we want to start a new project how could we find programmers well-versed in Nim? If a programmer can't pick up a language like Nim in a few weekends (from what I gather, it's similar to Python and not much different from most common languages, i.e. not something relatively exotic like Haskell) then I don't know. Our mainly PHP shop transitioned to Go quite effortlessly. Today we hire PHP juniors without any Go ex…

I think it is true for people who have computer science/software engineering or any programming background, but for my statistician/data scientist friends - they skip a bit when they think they might need to learn a different language.
Post reply on HN