Yup. We have gotten into the habit of leaving a lot of potential performance on the floor in the interest of productivity/accessibility. What always amazes me is when I have to work with a person who only speaks Python or only speaks JS and is completely unaware of the actual performance potential of a system. I think a lot of people just accept the performance they get as normal even if they are doing things that take 1000x (or worse) the time and/or space than it could (even without heroic work).
The computers are fast, but you don't know it
11–20 of 819 posts
Re: The computers are fast, but you don't know it
#12Re: The computers are fast, but you don't know it
#13I've been lightly banging the drum the last few years that a lot of programmers don't seem to understand how fast computers are, and often ship code that is just miserably slower than it needs to be, like the code in this article, because they simply don't realize that their code ought to be much, much faster. There's still a lot of very early-2000s ideas of how fast computers are floating around. I've wondered how m…
Python allows one to save development time in exchange for execution time
Python hasn't saved me development time since distutils was the right and only way to build things.
Re: The computers are fast, but you don't know it
#14It's hard to evaluate this article without seeing the detail of the "algorithm_wizardry", there's no detail here just where it would be interesting.
Re: The computers are fast, but you don't know it
#15I wonder how much power (and resulting CO2 emissions) could be saved if all code had to go through such optimization.
And on a slightly ranty note, Apple's A12z and A14 are still apparently "too weak" to run multiple windows simultaneously :)
Re: The computers are fast, but you don't know it
#16Earlier quoted context omitted.
Most likely a missued of Pandas. DF are heavy to create, but calculations on them are fast if you stay in the numpy world and stay vectorized.
"It's fast so long as you don't use any of the many parts that aren't fast!" This isn't great.
That's true for everything in computing.
Don't use a hammer as a screwdriver.
I'm not even implying they shouldn't have used pandas for this, I'm suggesting they probably wrote the wrong pandas code for this.
Pandas is typically 3 times faster than raw Python, not 10 times slower.
Re: The computers are fast, but you don't know it
#17My entire career, we never optimize code as well as we can, we optimize as well as we need to. Obviously the result is that computer performance is only "just okay" despite the hardware being capable of much more. This pattern repeats itself across the industry over decades without changing much.
Re: The computers are fast, but you don't know it
#18Hmm, interesting that single threaded C++ is 25% of Python exec time. It feels like C++ implementation might have area for improvement. My usual 1-to-1 translations result in C++ being 1-5% of Python exec time, even on combinatorial stuff.
I recently ported some very simple combinatorial code from Python to Rust. I was expecting around 100x speed up. I was surprised when the code ended running only 14 times faster.
Re: The computers are fast, but you don't know it
#19I've been lightly banging the drum the last few years that a lot of programmers don't seem to understand how fast computers are, and often ship code that is just miserably slower than it needs to be, like the code in this article, because they simply don't realize that their code ought to be much, much faster. There's still a lot of very early-2000s ideas of how fast computers are floating around. I've wondered how m…
It's not only a matter of 750ms instead of 200ms. I'm astonished every time I open some tool like Visual Studio, SAP Power Designer, or Libre Office that can stay for the most part of a minute on its loading screen.
What do those tools even do for that long? They can read enough data from the disk to overflow my computer's main memory a few times during it.
Re: The computers are fast, but you don't know it
#20It's hard to evaluate this article without seeing the detail of the "algorithm_wizardry", there's no detail here just where it would be interesting.
The author says:
"The function looks something like this:"
Then he says:
"I replaced Pandas with simple python lists and implemented the algorithm manually to do the group-by and sort."