Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Why I Use Nim instead of Python for Data Processing

benjamindlee.com

Post reply on HN
Print view View original

41–50 of 183 posts

Re: Why I Use Nim instead of Python for Data Processing

#41

Post by DeathArrow »

While Nim is for certain interesting and even pleasant to write code in, its small user base and environment discourage people to use it.

I don't write code only for myself.

How would I convince my employer to let me use Nim instead of a better known language?

And even I would convince my employer, if we want to start a new project how could we find programmers well-versed in Nim?

And even id we can find those people, it would mean we would have to write many things ourselves, which in other languages we can take for granted as they have libraries for almost anything.

So having a nice, performant and good language is just a small part of achieving your goals. You also need the people and the ecosystem.

Go, Rust, Kotlin, Swift and even Julia have the luck of having some industry heavyweights behind them, pushing the ecosystem and contributing with money and developers. Nim has only a bunch of passionate people behind it.

Re: Why I Use Nim instead of Python for Data Processing

#42

Post by DeathArrow »

ur-whale wrote: post #15
TekMol wrote: post #9

TLDR: Because Python is slow Yes, that is the achilles heel of Python. I am always torn between Python and PHP for new projects because of this. The Python Syntax plus its import system are huge advantages over PHP. On the other hand, you suffer a 6x slowdown if you go with Python. Decisions decisions. I so dearly wish I could have the good parts of both worlds.

Python is indeed slow, but why would anyone ever use php as a comparison point? And for data processing of all things ... Php is also very slow, on top of being many other kinds of unpleasant and broken.

I had a nightmare where I was writing an operating system in PHP, which ran in hypervisor written in PHP, which ran in a virtual machine written in PHP, which in a browser written in PHP, which ran in an operating system written in PHP...

Re: Why I Use Nim instead of Python for Data Processing

#43

Post by henbruas »

kiidev wrote: post #39
santiagobasulto wrote: post #37

I don't doubt Nim, looks like a great language. But that is just an awful Python implementation. I'd do it in this way: lines = (line for line in lines("orthocoronavirinae.fasta") if not line.startswith(">")) gc_lines = (1 if ('G' in line or 'C' in line) else 0 for line in lines) gc = sum(gc_lines) total = len(list(gc_lines)) # Alternatively, a more "memory efficient" total would be: total = sum(1 for _ in lines) Edi…

Isn't that going to be even slower, since a 156mb file would probably make gc_lines a very big tuple before summing it?

gc_lines isn't a tuple here, it's a generator expression. It will be lazily evaluated.

Re: Why I Use Nim instead of Python for Data Processing

#44

Post by makapuf »

kiidev wrote: post #39
santiagobasulto wrote: post #37

I don't doubt Nim, looks like a great language. But that is just an awful Python implementation. I'd do it in this way: lines = (line for line in lines("orthocoronavirinae.fasta") if not line.startswith(">")) gc_lines = (1 if ('G' in line or 'C' in line) else 0 for line in lines) gc = sum(gc_lines) total = len(list(gc_lines)) # Alternatively, a more "memory efficient" total would be: total = sum(1 for _ in lines) Edi…

Isn't that going to be even slower, since a 156mb file would probably make gc_lines a very big tuple before summing it?

You could do gc=sum(1 for _ on lines if ...) or use Counter

Re: Why I Use Nim instead of Python for Data Processing

#45

Post by DeathArrow »

piqufoh wrote: post #31

Python sometimes runs slowly, because it's not designed to run fast. It's designed to be readable and easy to write, which in turn makes developing python faster. It's a compromise, but I always prioritise _my_ time over my computers time, so if I can write something quickly and just go and get a coffee while it runs - I will do that. I won't spend twice as long writing a single-run script just because it'll finish b…

>It's designed to be readable and easy to write

So is Golang.

Re: Why I Use Nim instead of Python for Data Processing

#46

Post by cnmlp »

DeathArrow wrote: post #41

While Nim is for certain interesting and even pleasant to write code in, its small user base and environment discourage people to use it. I don't write code only for myself. How would I convince my employer to let me use Nim instead of a better known language? And even I would convince my employer, if we want to start a new project how could we find programmers well-versed in Nim? And even id we can find those people…

In the case of Python, the "industry heavyweights" do very little and have a negative influence now.

Why the phrase "only a bunch of passionate people"? This is how software gets written, parasitical corporations and their unproductive developers who are installed in existing OSS projects come later and mainly associate themselves with the result (speaking of Python again).

Re: Why I Use Nim instead of Python for Data Processing

#47

Post by dejj »

DeathArrow wrote: post #41

While Nim is for certain interesting and even pleasant to write code in, its small user base and environment discourage people to use it. I don't write code only for myself. How would I convince my employer to let me use Nim instead of a better known language? And even I would convince my employer, if we want to start a new project how could we find programmers well-versed in Nim? And even id we can find those people…

> How would I convince my employer Turntables: if you were an employer, what would convince you to use Nim?

Hiring for Nim skills can be a signal that a company has people who learn languages beyond the run-of-the-mill ones. A bunch of passionate people you might say. That would make the company promising to work for.

Re: Why I Use Nim instead of Python for Data Processing

#48

Post by DeathArrow »

m_mueller wrote: post #21
amyjess wrote: post #14

Earlier quoted context omitted.

> It's primarily a testament to how simply mind bogglingly slow Python is outside of its optimised numerical science ecosystem. From my experience in using Python at my last job, I'll also add that Python is decent at tasks that aren't CPU-bound. I wrote a lot of scripts that polled large amounts of network devices for information and then did something with it (typically upsert the data into a database, either via d…

I agree with your entire post but , and I‘m saying this as a fulltime python dev, there‘s often a point where it starts being bothersome, and that usually comes only later in the lifecycle of an application after it had some organic growth. Some day e.g. a sales manager comes down to your lair and asks you if you couldn‘t just also parse this little 200MB Excel spreadsheet after it came over the network such that you…

>Point being: If your performance ceiling is low, you‘re gonna hit it sooner.

And that isn't the only culprit. Large code bases will be hard to structure, maintain and organize. And you will spend more time writing tests than writing productive code. Because you don't have many defenses against errors and because once a bug will hit production it will be very difficult to debug.

Re: Why I Use Nim instead of Python for Data Processing

#49

Post by anyfactor »

oxfordmale wrote: post #33

The author makes a fair point, however, that is a rather non optimal implementation in Python. You likely could use chunked Pandas to speed up or the code, or at least replace some of the for loops with a list comprehension syntax. However, in any case I would never replace Python with Nim as it is too niche of a language and you would struggle with recruiting. I could consider Julia if it's popularity keeps growing.…

There are objectively better niche solutions for niche problems out there. But we pickup things that can be applied to solve a number of different problem, that are more versatile and has a community behind them.

Re: Why I Use Nim instead of Python for Data Processing

#50

Post by gpderetta »

amyjess wrote: post #14
zmmmmm wrote: post #8

It's primarily a testament to how simply mind bogglingly slow Python is outside of its optimised numerical science ecosystem. Which also why I don't use it that much, because while numerical analysis is a big part of what I do, so is what I would call "symbolic manipulation" and unless you go to quite some effort to transform every problem into a numerical one, Python is just awful at that. But Nim is only one of a w…

> It's primarily a testament to how simply mind bogglingly slow Python is outside of its optimised numerical science ecosystem. From my experience in using Python at my last job, I'll also add that Python is decent at tasks that aren't CPU-bound. I wrote a lot of scripts that polled large amounts of network devices for information and then did something with it (typically upsert the data into a database, either via d…

In my experience, Python is so slow that it will make CPU bound tasks that have no business being CPU bound.
Post reply on HN