Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Why I Use Nim instead of Python for Data Processing

benjamindlee.com

Post reply on HN
Print view View original

91–100 of 183 posts

Re: Why I Use Nim instead of Python for Data Processing

#91

Post by stavros »

CraigJPerry wrote: post #81
zmmmmm wrote: post #8

It's primarily a testament to how simply mind bogglingly slow Python is outside of its optimised numerical science ecosystem. Which also why I don't use it that much, because while numerical analysis is a big part of what I do, so is what I would call "symbolic manipulation" and unless you go to quite some effort to transform every problem into a numerical one, Python is just awful at that. But Nim is only one of a w…

The thing with Python is it's usually pretty easy to optimise quite impressively. E.g. random example: Sprinkle some cdef's in your python and suddenly you're faster than c++ https://github.com/luizsol/PrimesResult https://github.com/PlummersSoftwareLLC/Primes/blob/drag-race... 25.8 seconds down to 1.5

Apparently mypyc does the same if you have types in your code, though I've never used it.

Re: Why I Use Nim instead of Python for Data Processing

#92

Post by WJW »

dejj wrote: post #47
DeathArrow wrote: post #41

While Nim is for certain interesting and even pleasant to write code in, its small user base and environment discourage people to use it. I don't write code only for myself. How would I convince my employer to let me use Nim instead of a better known language? And even I would convince my employer, if we want to start a new project how could we find programmers well-versed in Nim? And even id we can find those people…

> How would I convince my employer Turntables: if you were an employer, what would convince you to use Nim? Hiring for Nim skills can be a signal that a company has people who learn languages beyond the run-of-the-mill ones. A bunch of passionate people you might say. That would make the company promising to work for.

Interestingly this was exactly the place Python was in back in the day: niche enough that anyone knowing it must've learned it because they thought it was cool rather than because it would get them a job. Those days are long gone of course.

Re: Why I Use Nim instead of Python for Data Processing

#93

Post by sound1 »

kgeist wrote: post #57
DeathArrow wrote: post #41

While Nim is for certain interesting and even pleasant to write code in, its small user base and environment discourage people to use it. I don't write code only for myself. How would I convince my employer to let me use Nim instead of a better known language? And even I would convince my employer, if we want to start a new project how could we find programmers well-versed in Nim? And even id we can find those people…

>if we want to start a new project how could we find programmers well-versed in Nim? If a programmer can't pick up a language like Nim in a few weekends (from what I gather, it's similar to Python and not much different from most common languages, i.e. not something relatively exotic like Haskell) then I don't know. Our mainly PHP shop transitioned to Go quite effortlessly. Today we hire PHP juniors without any Go ex…

> However, indeed, if you are to choose between, for example, Nim and Go for a new project, then I am not sure why would anyone prefer Nim. I'm really interested to know.

Same here, curious to know what HN crowd recommends between Nim vs Go for new projects.

Re: Why I Use Nim instead of Python for Data Processing

#94

Post by stillblue »

scoopertrooper wrote: post #69
samuel wrote: post #68

For me the corollary of this post should be, try PyPy. You may get a 10x speedup for free.

For me the corollary of this comment should be, try reading the post. You may learn that the author did get a 10x speedup for free, but it was still 3.3x slower than Nim.

Do you feel better about yourself by being snarky to people on the internet? How does it work? I'm curious

Re: Why I Use Nim instead of Python for Data Processing

#95

Post by hexo »

benjamin-lee wrote: post #65
brabel wrote: post #54

Earlier quoted context omitted.

I clicked on the big Download button and selected "all records", it downloaded over 3.5GB before I gave up... which file exactly should I use??

I'm sorry, I completely forgot that the file I used was from six months ago when I wrote the blog post (and then promptly forgot to publish it). In the last half year, the number of coronavirus sequences has increased dramatically. One thing that you could do to drop the file size down is to filter for only complete and unambiguous sequences, which drops the number down from 1.6 million to ~100k [1]. Alternatively, t…

The file at [2] is already gone :(

Re: Why I Use Nim instead of Python for Data Processing

#96

Post by stavros »

IshKebab wrote: post #88
elcritch wrote: post #32

Earlier quoted context omitted.

> Yeah should really be "Why I don't use Python for Data Processing". Not entirely. Nim‘s benefit here is that it’s superficially similar enough to Python that’s it’s easy for people from that world to pickup and start using Nim. > Also I don't know how anyone could design a language in the 21st century and make basic mistakes like this: > If that's any indication of the sanity of the rest of Nim then I'd say steer w…

> It’s intent is to allow a given codebase to maintain a consistent style (eg camel vs snake) even when making use of upstream libraries that use different styles. This doesn't make sense. For an entirely new language you can just have the entire ecosystem use the same style, e.g. like Rust does. Or even Python!

I'd agree, if you want a consistent style, ship a formatter with your package manager, like Rust does with `cargo fmt`. If it's there by default, it'll quickly be a standard.

Re: Why I Use Nim instead of Python for Data Processing

#97

Post by m_mueller »

DeathArrow wrote: post #48
m_mueller wrote: post #21

Earlier quoted context omitted.

I agree with your entire post but , and I‘m saying this as a fulltime python dev, there‘s often a point where it starts being bothersome, and that usually comes only later in the lifecycle of an application after it had some organic growth. Some day e.g. a sales manager comes down to your lair and asks you if you couldn‘t just also parse this little 200MB Excel spreadsheet after it came over the network such that you…

>Point being: If your performance ceiling is low, you‘re gonna hit it sooner. And that isn't the only culprit. Large code bases will be hard to structure, maintain and organize. And you will spend more time writing tests than writing productive code. Because you don't have many defenses against errors and because once a bug will hit production it will be very difficult to debug.

I somewhat disagree there. IMO you can structure a large application just as well in python as in Java, but it requires more architecture guidance as your degree of freedom is higher. Regarding testing, mypy and later python versions have been a big help there, I don't think it's much of a difference anymore and I'd take it over Node.js for that reason because the core language IMO is more consistent and strict and requires way way less outside dependencies. In fact the standard lib for me is the main reason to choose python today.

So for me it's really performance being the main thing to think about with python. If one anticipates tighter latency or throughput requirements I'd go for something else.

Re: Why I Use Nim instead of Python for Data Processing

#98

Post by gameswithgo »

zmmmmm wrote: post #8

It's primarily a testament to how simply mind bogglingly slow Python is outside of its optimised numerical science ecosystem. Which also why I don't use it that much, because while numerical analysis is a big part of what I do, so is what I would call "symbolic manipulation" and unless you go to quite some effort to transform every problem into a numerical one, Python is just awful at that. But Nim is only one of a w…

Nim isn’t unique in its performance, but it “feels” a lot like Python making it a nice gateway drug for Python users to start wasting less electricity.

Re: Why I Use Nim instead of Python for Data Processing

#99

Post by flohofwoe »

zmmmmm wrote: post #8

It's primarily a testament to how simply mind bogglingly slow Python is outside of its optimised numerical science ecosystem. Which also why I don't use it that much, because while numerical analysis is a big part of what I do, so is what I would call "symbolic manipulation" and unless you go to quite some effort to transform every problem into a numerical one, Python is just awful at that. But Nim is only one of a w…

IME Python is pretty great as cross-platform scripting-, automation- and glue-language up to a few thousand lines of code. Essentially replacing bash scripts and Windows batch files or for simple command line tools that don't need the performance. It should be just one language in one's language toolbox though.

Re: Why I Use Nim instead of Python for Data Processing

#100

Post by _dain_ »

sound1 wrote: post #93
kgeist wrote: post #57

Earlier quoted context omitted.

>if we want to start a new project how could we find programmers well-versed in Nim? If a programmer can't pick up a language like Nim in a few weekends (from what I gather, it's similar to Python and not much different from most common languages, i.e. not something relatively exotic like Haskell) then I don't know. Our mainly PHP shop transitioned to Go quite effortlessly. Today we hire PHP juniors without any Go ex…

> However, indeed, if you are to choose between, for example, Nim and Go for a new project, then I am not sure why would anyone prefer Nim. I'm really interested to know. Same here, curious to know what HN crowd recommends between Nim vs Go for new projects.

Nim has hygienic AST macros, so it has really good metaprogramming capabilities. An example of the sort of thing you can do with it: use pattern matching to implement a functional programming DSL:

https://nim-lang.org/blog/2021/03/10/fusion-and-pattern-matc...

This makes it really easy to reduce boilerplate and create low- or 0-cost abstractions for your problem domain. Example of this done in a microcontroller project here:

https://www.youtube.com/watch?v=j0fUqdYC71k

Async/await is also implemented by metaprogramming, rather than as a "core" part of the language:

https://www.youtube.com/watch?v=i0RB7UqxERE

Unrelated to the above, Nim also compiles to Javascript. So you can use the same language for both the backend and frontend.

Post reply on HN