Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Why I Use Nim instead of Python for Data Processing

benjamindlee.com

Post reply on HN
Print view View original

51–60 of 183 posts

Re: Why I Use Nim instead of Python for Data Processing

#51

Post by ellimilial »

A context might be useful.

From what I gather, the author is a researcher in bioinformatics related field. This may indicate that they tend to work either alone or in a relatively small group. The domain is small scope data processing/manipulation, research/exploratory code, ,likely short-lived or even one-off.

The progress in this context will possibly be governed by sheer processing speed (e.g. it’s unlikely anyone will delve deep into the code, a lot of iterations to ‘just get it done’ instead of testing etc.).

If this is more or less correct, the point that Nim might be more useful than Python for the author sounds very sensible to me. It’s a nice spot between command line tools and more functionality-loaded languages.

Re: Why I Use Nim instead of Python for Data Processing

#53

Post by [deleted] »

santiagobasulto wrote: post #37

I don't doubt Nim, looks like a great language. But that is just an awful Python implementation. I'd do it in this way: lines = (line for line in lines("orthocoronavirinae.fasta") if not line.startswith(">")) gc_lines = (1 if ('G' in line or 'C' in line) else 0 for line in lines) gc = sum(gc_lines) total = len(list(gc_lines)) # Alternatively, a more "memory efficient" total would be: total = sum(1 for _ in lines) Edi…

[deleted]

Re: Why I Use Nim instead of Python for Data Processing

#54

Post by brabel »

benjamin-lee wrote: post #35
brabel wrote: post #26

> Nim compilation process took an additional 702 ms That's horrifyingly slow for a compiler. The author mentioned "modern languages look like Python but run as fast as C", which is a common promise those languages make that never really materialize except for a few very happy path cases they heavily optmised the language for. Julia, for example, makes this promise too, but compiles even slower than that and takes rid…

I didn't post it because it's quite big (150M) but readily available from the NCBI Virus portal [1]. I would love to see how well other languages compete both for speed and simplicity. [1] https://www.ncbi.nlm.nih.gov/labs/virus/vssi/#/virus?SeqType...

I clicked on the big Download button and selected "all records", it downloaded over 3.5GB before I gave up... which file exactly should I use??

Re: Why I Use Nim instead of Python for Data Processing

#55

Post by egwor »

DeathArrow wrote: post #41

While Nim is for certain interesting and even pleasant to write code in, its small user base and environment discourage people to use it. I don't write code only for myself. How would I convince my employer to let me use Nim instead of a better known language? And even I would convince my employer, if we want to start a new project how could we find programmers well-versed in Nim? And even id we can find those people…

You could make the same argument about rust right now., but rust is further along the 'programming language life path'.

Re: Why I Use Nim instead of Python for Data Processing

#56

Post by haxscramper »

IshKebab wrote: post #30
thewakalix wrote: post #19

Earlier quoted context omitted.

How is that a basic mistake? I think having two distinct identifiers that differ only by case sounds like a mistake!

Various programming languages (and file systems!) have had some form of case insensitivity and it always turns out to be an absolutely terrible idea: * It makes searching for identifiers harder. For Nim you can't even use case insensitive search because of the underscore thing! Better practice your regexes. * The case insensitivity rules are usually super complicated and don't apply to everything, so now it's an extr…

> Do you enjoy the tabs vs spaces debate? How about single quote Vs double quotes? Ugly inconsistently styled code?

Nim solved tabs/spaces debate by allowing only spaces. Single/double quote are also completely separate, so there is no inconsistency as well. About "inconsistently styled code" - due to style insensitivity, effects are not viral. If you depend on a library that uses `get_name` but your project adopted `getName()` your don't need to suffer.

> Unicode case insensitivity is actually really really complicated

Which is why nim only handles case insensitivity for ASCII identifiers and not Unicode. Which makes sense because 99.9% code is written in ASCII.

> The case insensitivity rules are usually super complicated and don't apply to everything

Quoting from the manual - "only the first letters are compared in a case-sensitive manner. Other letters are compared case-insensitively within the ASCII range, and underscores are ignored." Unicode is not handled in style-insensetive manner, so the rule is pretty simple.

> It makes searching for identifiers harder. For Nim you can't even use case insensitive search because of the underscore thing!

Technically true, but in reality this comes up so rarely, I didn't have any issues with this ever. Nim projects usually adopt camel/pascal case.

For some reason, people often assume that allowing style insensitivity instantly throws the whole language ecosystem in complete disarray and everyone starts writing code mixing every possible style of writing at once, swapping styles on every second identifier just for their amusement.

Re: Why I Use Nim instead of Python for Data Processing

#57

Post by kgeist »

DeathArrow wrote: post #41

While Nim is for certain interesting and even pleasant to write code in, its small user base and environment discourage people to use it. I don't write code only for myself. How would I convince my employer to let me use Nim instead of a better known language? And even I would convince my employer, if we want to start a new project how could we find programmers well-versed in Nim? And even id we can find those people…

>if we want to start a new project how could we find programmers well-versed in Nim?

If a programmer can't pick up a language like Nim in a few weekends (from what I gather, it's similar to Python and not much different from most common languages, i.e. not something relatively exotic like Haskell) then I don't know. Our mainly PHP shop transitioned to Go quite effortlessly. Today we hire PHP juniors without any Go experience (easier to find), we teach them, and then they work on Go codebases already after a month of internship. So lack of "professional Nim programmers" doesn't look like a problem to me.

Lack of libraries is a good point but from what I read, Nim compiles to C, so I understand they can have access to tens (hundreds?) of thousands C libraries without writing everything from scratch.

However, indeed, if you are to choose between, for example, Nim and Go for a new project, then I am not sure why would anyone prefer Nim. I'm really interested to know.

Re: Why I Use Nim instead of Python for Data Processing

#58

Post by fctorial »

TekMol wrote: post #9

TLDR: Because Python is slow Yes, that is the achilles heel of Python. I am always torn between Python and PHP for new projects because of this. The Python Syntax plus its import system are huge advantages over PHP. On the other hand, you suffer a 6x slowdown if you go with Python. Decisions decisions. I so dearly wish I could have the good parts of both worlds.

> that is the achilles heel of Python

I think that is pip.

Re: Why I Use Nim instead of Python for Data Processing

#59

Post by paulluuk »

As a Data Engineer, I mainly use Python in conjunction with PySpark. So essentially I just use Python as an easy-to-read wrapper around Spark, and it works great when working together with Data Scientists, who are mostly used to Pandas, Keras, Tensorflow, etc.

In my use case, I don't really see how Nim would make my life easier right now.

Re: Why I Use Nim instead of Python for Data Processing

#60

Post by tandav »

benjamin-lee wrote: post #35
brabel wrote: post #26

> Nim compilation process took an additional 702 ms That's horrifyingly slow for a compiler. The author mentioned "modern languages look like Python but run as fast as C", which is a common promise those languages make that never really materialize except for a few very happy path cases they heavily optmised the language for. Julia, for example, makes this promise too, but compiles even slower than that and takes rid…

I didn't post it because it's quite big (150M) but readily available from the NCBI Virus portal [1]. I would love to see how well other languages compete both for speed and simplicity. [1] https://www.ncbi.nlm.nih.gov/labs/virus/vssi/#/virus?SeqType...

can you upload somewhere your 150M file. If i follow the link in your comment there are bunch of small files, did you concatenate them?
Post reply on HN