The computers are fast, but you don't know it
1–10 of 819 posts
Re: The computers are fast, but you don't know it
#2Did the author beat pandas group an aggregate by using standard Python lists?
Re: The computers are fast, but you don't know it
#3This afternoon, discussing with my boss, why issuing two x 64 byte loads per cycle is pushing it; to the point where l1 says no..
400GB of l1 bandwidth is all we have..
Is all we have..
I remember when we
could move maybe 50KB/s.. Ans that was more than enough..
Re: The computers are fast, but you don't know it
#4Did the author beat pandas group an aggregate by using standard Python lists?
The main optimization at that stage seems to be preallocating the weights. I don't know pandas but such a thing would have been possible without dropping any of the linalg libraries I do know how to use.
I doubt the author's C++ implementations beat BLAS/LAPACK, but since they're not shown I can only guess.
I've done stuff like this before but the tooling is really no fun, somewhere between 2 and 3 I'd just write it all in C++.
Changing the interface just to get parallelism out seems not great - give it to the user for free if the array is long enough - but maybe it was more reasonable for the non-trivial real problem.
Re: The computers are fast, but you don't know it
#5Did the author beat pandas group an aggregate by using standard Python lists?
Most likely a missued of Pandas. DF are heavy to create, but calculations on them are fast if you stay in the numpy world and stay vectorized.
Re: The computers are fast, but you don't know it
#6I've been lightly banging the drum the last few years that a lot of programmers don't seem to understand how fast computers are, and often ship code that is just miserably slower than it needs to be, like the code in this article, because they simply don't realize that their code ought to be much, much faster. There's still a lot of very early-2000s ideas of how fast computers are floating around. I've wondered how much of it is the still-extensive use of dynamic scripting languages and programmers not understanding just how much performance you can throw away how quickly with those things. It isn't even just the slowdown you get just from using one at all; it's really easy to pile on several layers of indirection without really noticing it. And in the end, the code seems to run "fast enough" and nobody involved really notices that what is running in 750ms really ought to run in something more like 200us.
I have a hard time using (pure) Python anymore for any task that speed is even remotely a consideration for anymore. Not only is it slow even at the best of times, but so many of its features beg you to slow down even more without thinking about it.
Re: The computers are fast, but you don't know it
#7Hmm, interesting that single threaded C++ is 25% of Python exec time. It feels like C++ implementation might have area for improvement.
My usual 1-to-1 translations result in C++ being 1-5% of Python exec time, even on combinatorial stuff.
Re: The computers are fast, but you don't know it
#8Did the author beat pandas group an aggregate by using standard Python lists?
Most likely a missued of Pandas. DF are heavy to create, but calculations on them are fast if you stay in the numpy world and stay vectorized.
"It's fast so long as you don't use any of the many parts that aren't fast!"
This isn't great.
Re: The computers are fast, but you don't know it
#9Did the author beat pandas group an aggregate by using standard Python lists?
Yes. Apparently they have used Python lists to beat highly optimized library which builds upon numpy, in C. Yeah right.
Note that I'm not saying that their second version of the code wasn't faster, just that this has nothing to do with python vs. pandas.
Re: The computers are fast, but you don't know it
#10I've been lightly banging the drum the last few years that a lot of programmers don't seem to understand how fast computers are, and often ship code that is just miserably slower than it needs to be, like the code in this article, because they simply don't realize that their code ought to be much, much faster. There's still a lot of very early-2000s ideas of how fast computers are floating around. I've wondered how m…
Python allows one to save development time in exchange for execution time