Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Why I Use Nim instead of Python for Data Processing

benjamindlee.com

Post reply on HN
Print view View original

71–80 of 183 posts

Re: Why I Use Nim instead of Python for Data Processing

#71

Post by benjamin-lee »

qwerty1793 wrote: post #67
benjamin-lee wrote: post #29

Earlier quoted context omitted.

I actually use TypeScript/JavaScript a lot for this reason, especially for biological algorithms that I want to run in the browser. The developer tooling is also as good as you can hope for, especially when using VS Code. I actually wrote a circular RNA sequence deduplication algorithm in it just recently [1]. With respect to the identifier resolution in Nim, it strikes me as more of a matter of preference. Especiall…

That's great. However the method that you use to find the canonical representative [1] is quadratic (when the string has length N, there are N rotations and for each rotation you need to check N characters to determine whether this is earlier than the best on that you have found so far). For large strings you would probably want to switch to one of the linear minimal string rotation algorithms [2], for example Booth'…

You hit the nail on the head. This is just the lexicographically minimal string rotation with a canonicalization step. I have actually had this on my to-do list for a while. The truth is that the data set I'm applying this to is small (though I'm doing my best to change that by discovering new viroid-like agents) so the optimization has yet to become pressing. At some point, I'd like to spin this script out into a standalone tool but I'm not sure the demand is there yet.

Re: Why I Use Nim instead of Python for Data Processing

#72

Post by stjohnswarts »

DeathArrow wrote: post #45
piqufoh wrote: post #31

Python sometimes runs slowly, because it's not designed to run fast. It's designed to be readable and easy to write, which in turn makes developing python faster. It's a compromise, but I always prioritise _my_ time over my computers time, so if I can write something quickly and just go and get a coffee while it runs - I will do that. I won't spend twice as long writing a single-run script just because it'll finish b…

>It's designed to be readable and easy to write So is Golang.

Go is precisely what I switched to for my test/automation efforts.

Re: Why I Use Nim instead of Python for Data Processing

#73

Post by rualca »

elcritch wrote: post #32
IshKebab wrote: post #16

Earlier quoted context omitted.

Yeah should really be "Why I don't use Python for Data Processing". I would consider Typescript as an alternative too which I'm sure would get a similar speedup. Also I don't know how anyone could design a language in the 21st century and make basic mistakes like this: > Nim treats identifiers as equal if they are the same after removing capitalization (except for the first letter) and underscore, which means that yo…

> Yeah should really be "Why I don't use Python for Data Processing". Not entirely. Nim‘s benefit here is that it’s superficially similar enough to Python that’s it’s easy for people from that world to pickup and start using Nim. > Also I don't know how anyone could design a language in the 21st century and make basic mistakes like this: > If that's any indication of the sanity of the rest of Nim then I'd say steer w…

> It may seem like a design mistake at first glance but it’s surprisingly useful. > It’s intent is to allow a given codebase to maintain a consistent style (eg camel vs snake) even when making use of upstream libraries that use different styles.

That doesn't sound right at all. It sounds like a design choice aimed at achieving the exact opposite: inconsistency without any positive tradeoff in return.

> Not including the first letter avoids most of the annoyance of wantonly mixing all cap constants or lower case and linters avoid teams mismatching internal styles.

That does not sound right at all. At most, it sounds like the compiler does not throw errors when stumbling on what would otherwise be syntax errors, but you still have all the mismatches in internal styles and linters complaining about code and teams wasting time with internal piss matches, and more importantly a way to foster completely futile nitpicking discussions within a language community.

Re: Why I Use Nim instead of Python for Data Processing

#74

Post by _dain_ »

santiagobasulto wrote: post #37

I don't doubt Nim, looks like a great language. But that is just an awful Python implementation. I'd do it in this way: lines = (line for line in lines("orthocoronavirinae.fasta") if not line.startswith(">")) gc_lines = (1 if ('G' in line or 'C' in line) else 0 for line in lines) gc = sum(gc_lines) total = len(list(gc_lines)) # Alternatively, a more "memory efficient" total would be: total = sum(1 for _ in lines) Edi…

>total = len(list(gc_lines))

That won't work properly; you've already exhausted the gc_lines generator in the previous line.

Re: Why I Use Nim instead of Python for Data Processing

#75

Post by samuel »

scoopertrooper wrote: post #69
samuel wrote: post #68

For me the corollary of this post should be, try PyPy. You may get a 10x speedup for free.

For me the corollary of this comment should be, try reading the post. You may learn that the author did get a 10x speedup for free, but it was still 3.3x slower than Nim.

I did, I don't think the snarkiness was necessary.

My point, which apparently it wasn't evident enough, is that you can get the most of the benefits by doing nothing, just trying a different Python implementation, without the hassle of learning a niche language, as easy as it might be.

BTW, if you take into account compilation times the difference is even meager, and in all fairness the PyPy warmup period should have had to be discounted.

Re: Why I Use Nim instead of Python for Data Processing

#76

Post by mkl95 »

I'm an experienced Python developer who also dabbles with some applications written in C++. In my experience Python is much faster when it comes to development speed but it's way more demanding when it comes to optimization.

When you write C++, you kind of cheat because even code with high computational complexity is pretty fast. Whereas the equivalent code in Python will be awfully slow.

So, while it's true that Python requires less development time, this statement can't be used generally. I have spent hours optimizing Python code when in C++ I would have just moved on to my next task.

Re: Why I Use Nim instead of Python for Data Processing

#77

Post by pjmlp »

piqufoh wrote: post #31

Python sometimes runs slowly, because it's not designed to run fast. It's designed to be readable and easy to write, which in turn makes developing python faster. It's a compromise, but I always prioritise _my_ time over my computers time, so if I can write something quickly and just go and get a coffee while it runs - I will do that. I won't spend twice as long writing a single-run script just because it'll finish b…

There are plenty of compiled languages that are also designed to be readable and easy to write.

Re: Why I Use Nim instead of Python for Data Processing

#78

Post by nine_k »

Why do we use Python for data processing?

Because we use it as a nice syntactic frontend to numpy, a large and highly optimized library written in C++ and Fortran (sic). That is, we actually don't use "Python-native" code much, and numpy is essentially APL-like array-oriented thing where e.g. you don't normally need loops.

For native-language data processing, Python is slow; Nim or Julia would easily outperform it, while being comparably ergonomic.

Re: Why I Use Nim instead of Python for Data Processing

#79

Post by santiagobasulto »

_dain_ wrote: post #74
santiagobasulto wrote: post #37

I don't doubt Nim, looks like a great language. But that is just an awful Python implementation. I'd do it in this way: lines = (line for line in lines("orthocoronavirinae.fasta") if not line.startswith(">")) gc_lines = (1 if ('G' in line or 'C' in line) else 0 for line in lines) gc = sum(gc_lines) total = len(list(gc_lines)) # Alternatively, a more "memory efficient" total would be: total = sum(1 for _ in lines) Edi…

>total = len(list(gc_lines)) That won't work properly; you've already exhausted the gc_lines generator in the previous line.

True, you'd need to re create the generator expression. Still, the other implementation seemed too naive.

Re: Why I Use Nim instead of Python for Data Processing

#80

Post by rualca »

nxpnsv wrote: post #66

Missing : on the first line of code. When speed mattes, people use libraries that are considerably faster than plain python. It’s these libraries that turned python so popular in data science. Giving them up maybe makes sense, but that mans a whole lot of learning and development to replace already pretty well established tools.

> When speed mattes, people use libraries that are considerably faster than plain python.

This.

The general guideline has always been that Python is ideal for glue code and non-performance-critical code, and when performance became an issue then Python code would simply be used as glue code to invoke specialized libraries. Perhaps the most popular example of this approach is bumpy, which uses BLAS and LAPACK internally to handle linear algebra stuff.

This Nim advertisement sounds awfully desperate with the way it resorts to what feels like a poorly assembled strawman, while giving absolutely nothing in return.

Post reply on HN