Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Test for lists in Cython

github.com

Post reply on HN
Print view View original

131–140 of 147 posts

Re: Test for lists in Cython

#131

Post by neolog »

anotheranonym wrote: post #129
optimalsolver wrote: post #126

Earlier quoted context omitted.

Thanks for this. Will Julia advocates every use honest benchmarks to make their language look good? I doubt it.

1000% this. Multiple times I’ve encountered a Julia benchmark claiming to show its superiority in a task I routinely perform. And every time the pro Julia benchmark turned out to be total BS.

I wonder if it's more common for that language than other languages. Probably not many people have enough expertise in many different languages to compare fairly across them.

Re: Test for lists in Cython

#132

Post by ChrisRackauckas »

optimalsolver wrote: post #126
dr_zoidberg wrote: post #125

The cython code is a bit messy. Changing from: cpdef float iterate_list(a_list): cdef double count = 0 cdef int i, j for i in range(len(a_list)): internal_list = a_list[i] for j in range(len(internal_list)): count += internal_list[j] print(count) return count To: cpdef float iterate_list(list a_list): cdef double count = 0 cdef double val = 0 cdef list ilist for ilist in a_list: for val in ilist: count += val print(c…

Thanks for this. Will Julia advocates every use honest benchmarks to make their language look good? I doubt it.

Note that the OP is a Python user, not a Julia user. The Github profile is a bunch of Python packages and the Julia code wasn't even optimized (https://github.com/00sapo/cython_list_test/pull/5). If this test says anything, it at least would say that a inexperienced Python user could pick up Julia and do pretty well, even if the code they write isn't great. I think the right thing to do is just to help this guy learn to code for performance a bit better: it'll be better for him and would bring some positivity.

Even if it doesn't say that, bashing people who use Julia for a repository made by a Python user is a new level of HN trolling.

Re: Test for lists in Cython

#133

Post by adgjlsfhk1 »

leephillips wrote: post #107
duckerude wrote: post #102

Earlier quoted context omitted.

If join were a method on lists then you wouldn't be able to use it with other iterables. Putting it on str is more useful, if more confusing. Maybe it would be better as a standalone function. But then you'd either have to import it or it would pollute the global namespace. Other OOP models offer different solutions.

The Julia join function works on any iterable whose members are or can be converted into strings.

That requires multiple dispatch.

Re: Test for lists in Cython

#134

Post by edgyquant »

zozbot234 wrote: post #40
azangru wrote: post #36

Earlier quoted context omitted.

> Also a big downside with JavaScript. Can't Node run multiple processes (or multiple workers, not sure of the proper terminology) on multiple cores? As discussed in this thread on StackOverflow: https://stackoverflow.com/questions/61893497/node-js-on-mult...

> Can't Node run multiple processes Of course, but you can also run multiple Python processes.

It’s way different though, node itself is multithreaded (just under the hood)

Re: Test for lists in Cython

#135

Post by eigenspace »

anotheranonym wrote: post #129
optimalsolver wrote: post #126

Earlier quoted context omitted.

Thanks for this. Will Julia advocates every use honest benchmarks to make their language look good? I doubt it.

1000% this. Multiple times I’ve encountered a Julia benchmark claiming to show its superiority in a task I routinely perform. And every time the pro Julia benchmark turned out to be total BS.

It's hard to say anything about your use-case without more information, but I will say that one thing is that the way people talk about julia often seems to give people a mistaken impression about how to attain it's performance claims.

Namely, realizing these claims requires learning the language, and actually taking advantage of it's strengths rather than just writing 'python in julia'.

It's very common for Python users to show up on the Discourse forum and complain that julia is slower than python and then show some code that's basically just Python code written in julia, including a huge proliferation of global variables, allocating huge amounts of temporary arrays, etc.

There's also a huge spectrum of 'benchmark quality' out there. E.g. the benchmarks this HN post features seem pretty shitty and are not measuring anything interesting or useful as far as I can tell.

Re: Test for lists in Cython

#136

Post by pantsforbirds »

ZuLuuuuuu wrote: post #28
mhh__ wrote: post #13

Earlier quoted context omitted.

If you are genuinely hitting a wall like that surely it's time to move on from python? Or at least think about it

The thing is that the current ecosystem (numpy + scipy + pandas + PyCharm) fits our company structure perfectly (very few software engineers, mostly test engineers who are not very proficient coders). And we already have tens of thousands lines of code. So changing the whole ecosystem just because 5% of the problems are slow is too big of a jump for us. For now, it is easier for us to write a bit of C code for places…

You don't have to replace the core python parts. We moved a decent amount of our data pre-processing into Scala, Golang, and Data Base queries and then just store the processed data for the python models to use to train/run (which ends up using the super efficient C/Fortran/Cuda code).

Re: Test for lists in Cython

#137

Post by otabdeveloper4 »

Nimitz14 wrote: post #39

one word: pybind11 I honestly think people are severely underestimating what a massive impact this is currently having in increasing productivity of python devs who know a little C++.

I concur, pybind11 is the smoothest Python extension story I've experienced so far.

Re: Test for lists in Cython

#138

Post by captain_price7 »

ZuLuuuuuu wrote: post #80
pletnes wrote: post #37

Earlier quoted context omitted.

Numpy operations release the GIL (usually at least) so you can use a threadpool and, indeed, share memory. Just try it and you may be pleasantly surprised. Dask is great if you’re processing large amounts of data, and it recommends and supports threads for this reason.

I didn't know that. So let's say I have 4 regular Python threads calling the same function, and in this function, let's say I call numpy.add on the same array (but different parts of the array), then will it actually use different cores for these 4 different threads? I will try it out, if it works, then that's actually great and would be super easy.

The answer is No, isn't it?

Re: Test for lists in Cython

#139

Post by [deleted] »

adwn wrote: post #58
mhh__ wrote: post #23

Earlier quoted context omitted.

Godbolt example?

Here you go: https://play.rust-lang.org/?version=stable&mode=debug&editio... Because the data structure LotsOfData doesn't implement the Clone trait, attempting to create a copy will fail at compile-time.

[deleted]
Post reply on HN