Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Why I Use Nim instead of Python for Data Processing

benjamindlee.com

Post reply on HN
Print view View original

171–180 of 183 posts

Re: Why I Use Nim instead of Python for Data Processing

#171

Post by cycomanic »

leephillips wrote: post #129
leephillips wrote: post #113

Earlier quoted context omitted.

Although Nim using weird order of operations is unfortunate, your example is not well chosen. Replace the exponent 3 with an even integer and your point will be clear.

Maybe not that weird, although not the most common syntax. In Nim 0 - 2^2 evaluates to -4, while -2^2 evaluates to +4. So in -2, the - is treated as a unary minus and binds tightly to the 2. It would be bad if Nim were doing + or - before ^, or before *. I have a feeling some other languages treat the unary minus the same way, but don’t know of any examples off hand.

Yes when I asked a question about this on their gitter channel, people brought up several languages I think bc is one of them for example. I understand the reasoning I still find it extremely unintuitive personally.

Re: Why I Use Nim instead of Python for Data Processing

#172

Post by soundmasterj »

epidemian wrote: post #149
soundmasterj wrote: post #146

This is reasonably idiomatic Python and 10x faster than the implementation in the original post: with open("orthocoronavirinae.fasta") as f: text = ''.join((line.rstrip() for line in f.readlines() if not line.startswith('>'))) gc = text.count('G') + text.count('C') total = len(text) Or if you want to be explicit, this is just as fast (and might scale better for particularly long genomes): gc = 0 total = 0 with open("…

I think this is missing the point of the article. Yes, you can implement a faster Python version, but notice also: * This faster version is reading all the file into memory (except comment lines). The article mentions the data being 150MB, which should fit in memory, but for larger datasets, this approach would be unfeasible * The faster version is actually delegating a lot of work to Python's C internals by using te…

I didn't try to write optimized code, but idiomatic Python. Which also happens to be 10x faster.

To make it streaming, take the second version and remove the readlines (directly iterate over f).

Delegating work to Python's C internals is fine IMO because "batteries included" is a key feature of Python. "Nim outperforms unidiomatic Python that deliberately ignores key language features" is perhaps true, but less flashy of a headline.

And to be honest, I mainly wrote this because the other top level Python implementations for this one were terrible at the time of the post.

Re: Why I Use Nim instead of Python for Data Processing

#173

Post by soundmasterj »

user5994461 wrote: post #161
user5994461 wrote: post #158

Earlier quoted context omitted.

One liner to count gc, without buffering. import io f = io.StringIO( """ AB CD EF GH """ ) total = sum(map(lambda s: 0 if s[0]==">" else s.count('G') + s.count('C'), f.readlines())) print(total)

And reading the file as binary. There's a lesson about the overhead of unicode strings here ;) Your first example takes 3.1 seconds, my previous comment takes 2.3 seconds, this one takes 1.4 seconds. start = time.perf_counter() with open("orthocoronavirinae.fasta", "rb") as f: total = sum(map(lambda s: 0 if s[0]==65 else s.count(b"G") + s.count(b"C"), f.readlines())) end = time.perf_counter() print(total, " total") p…

This is good too. I would use a generator expression instead of the map though probably.

Re: Why I Use Nim instead of Python for Data Processing

#174

Post by cycomanic »

tzs wrote: post #130
cycomanic wrote: post #34

One pitfall that I ran into when trying out Nim for scientific computing is that Nim follows more a computer science convention than mathematics convention for exponentiation and negation operators. That is in Nim -2^3=(-2)^3 unlike more scientific computing oriented languages where -2^3=-(2^3). To someone like myself who mainly does scientific work this was quite unexpected and it causes a surprising amount of menta…

I don't think that is a mathematics convention vs computer science convention thing. Most languages, whether aimed at mathematics or computer science, have exponentiation at higher precedence than unary minus. (JavaScript does not, but it also does not allow unary minus directly in front of the base of an exponentiation so you have to add parenthesis no matter which interpretation you want). The main places you find…

IIRC the argument is that unitary negation is the operator with the highest presedence and one should be consistent across logical and mathematical operations. I think this is a not a completely unreasonable stance to take (and several other languages take the same), but it is unintuitive for someone who does significant scientific computing.

Re: Why I Use Nim instead of Python for Data Processing

#175

Post by nimmer »

IshKebab wrote: post #30
thewakalix wrote: post #19

Earlier quoted context omitted.

How is that a basic mistake? I think having two distinct identifiers that differ only by case sounds like a mistake!

Various programming languages (and file systems!) have had some form of case insensitivity and it always turns out to be an absolutely terrible idea: * It makes searching for identifiers harder. For Nim you can't even use case insensitive search because of the underscore thing! Better practice your regexes. * The case insensitivity rules are usually super complicated and don't apply to everything, so now it's an extr…

> For Nim you can't even use case insensitive search because of the underscore thing! Better practice your regexes.

Not at all, unless you decide to mix different styles in the same codebase.

Re: Why I Use Nim instead of Python for Data Processing

#176

Post by nimmer »

zmmmmm wrote: post #8

It's primarily a testament to how simply mind bogglingly slow Python is outside of its optimised numerical science ecosystem. Which also why I don't use it that much, because while numerical analysis is a big part of what I do, so is what I would call "symbolic manipulation" and unless you go to quite some effort to transform every problem into a numerical one, Python is just awful at that. But Nim is only one of a w…

> It's primarily a testament to how simply mind bogglingly slow Python is

No, Nim is truly among the top fastest languages when writing idiomatic code as shown in many benchmarks.

> But Nim is only one of a whole suite of languages that easily cruise to a 10x performance win over Python

...while also being very friendly to Python programmers, intuitive and expressive. Unlike many other languages.

Re: Why I Use Nim instead of Python for Data Processing

#177

Post by nimmer »

DeathArrow wrote: post #41

While Nim is for certain interesting and even pleasant to write code in, its small user base and environment discourage people to use it. I don't write code only for myself. How would I convince my employer to let me use Nim instead of a better known language? And even I would convince my employer, if we want to start a new project how could we find programmers well-versed in Nim? And even id we can find those people…

> How would I convince my employer to let me use Nim instead of a better known language?

This is a rephrasing of "nobody ever got fired for buying IBM".

Some organization prioritize innovation and technical acumen.

> So having a nice, performant and good language is just a small part of achieving your goals. You also need the people and the ecosystem.

Many applications don't need a large ecosystem. People can learn.

> Go, Rust, Kotlin, Swift and even Julia have the luck of having some industry heavyweights behind them

Python was never corporate-driven, thankfully, and it is successful.

Re: Why I Use Nim instead of Python for Data Processing

#178

Post by patrick451 »

matthiaswh wrote: post #106
IshKebab wrote: post #16

Earlier quoted context omitted.

Yeah should really be "Why I don't use Python for Data Processing". I would consider Typescript as an alternative too which I'm sure would get a similar speedup. Also I don't know how anyone could design a language in the 21st century and make basic mistakes like this: > Nim treats identifiers as equal if they are the same after removing capitalization (except for the first letter) and underscore, which means that yo…

I wish this tired, boring argument didn't derail every conversation that remotely mentions Nim. See: literally every past HN post with "Nim" in the title. Nim's underlying, perhaps understated philosophy is that it lets you write code the way you want to write code. If you like snake case, use it. If you want camel case, sure. Write your code base how you want to write it, keep it internally consistent if you want, o…

> I wish this tired, boring argument didn't derail every conversation that remotely mentions Nim. See: literally every past HN post with "Nim" in the title.

This is a serious design flaw. It absolutely should be front and center when Nim is discussed.

> Nim's underlying, perhaps understated philosophy is that it lets you write code the way you want to write code. If you like snake case, use it. If you want camel case, sure. Write your code base how you want to write it, keep it internally consistent if you want, or don't. Nim doesn't really care.

I want to write code where "myFoo != my_foo". Evidently, nim doesn't allow that, so this argument seems pretty hollow.

> Now you're stuck with screaming linters or random `# noqa` lines stuffed in your code, and that one variable that you're using from a library sticks out like a sore thumb.

This is because we have crappy linters. If a linter can't tell that an identifier is camel case because it is third party, it's a bad linter.

> Could Nim have forced everyone to snake_case naming structures for everything from the start?

This would have been preferable to this madness.

Re: Why I Use Nim instead of Python for Data Processing

#179

Post by [deleted] »

thewakalix wrote: post #19
IshKebab wrote: post #16

Earlier quoted context omitted.

Yeah should really be "Why I don't use Python for Data Processing". I would consider Typescript as an alternative too which I'm sure would get a similar speedup. Also I don't know how anyone could design a language in the 21st century and make basic mistakes like this: > Nim treats identifiers as equal if they are the same after removing capitalization (except for the first letter) and underscore, which means that yo…

How is that a basic mistake? I think having two distinct identifiers that differ only by case sounds like a mistake!

[deleted]

Re: Why I Use Nim instead of Python for Data Processing

#180

Post by IshKebab »

matthiaswh wrote: post #106
IshKebab wrote: post #16

Earlier quoted context omitted.

Yeah should really be "Why I don't use Python for Data Processing". I would consider Typescript as an alternative too which I'm sure would get a similar speedup. Also I don't know how anyone could design a language in the 21st century and make basic mistakes like this: > Nim treats identifiers as equal if they are the same after removing capitalization (except for the first letter) and underscore, which means that yo…

I wish this tired, boring argument didn't derail every conversation that remotely mentions Nim. See: literally every past HN post with "Nim" in the title. Nim's underlying, perhaps understated philosophy is that it lets you write code the way you want to write code. If you like snake case, use it. If you want camel case, sure. Write your code base how you want to write it, keep it internally consistent if you want, o…

> 3rd party libraries where someone chose a different standard because that's their preference

C & C++ don't have a standard style but Python and JavaScript definitely do. And Rust and Go and I would guess most projects. Find me a popular library in one of those languages that doesn't use the standard style.

Post reply on HN