Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Performance of the Python 3.14 tail-call interpreter

blog.nelhage.com

Post reply on HN
Print view View original

71–80 of 180 posts

Re: Performance of the Python 3.14 tail-call interpreter

#71

Post by jraph »

kenjin4096 wrote: post #51

Hello. I'm the author of the PR that landed the tail-calling interpreter in CPython. First, I want to say thank you to Nelson for spending almost a month to get to the root of this issue. Secondly, I want to say I'm extremely embarrassed and sorry that I made such a huge oversight. I, and probably the rest of the CPython team did not expect the compiler we were using for the baseline to have that bug. I posted an apo…

Reading that you are extremely embarrassed and sorry that you made such a huge oversight, I was imagining you had broken something / worsened CPython's performance.

But it's nothing like this. You announced a 10-15% perf improvement but that improvement is more like 1-5% on a non buggy compiler. It's not even like that 10-15% figure is wrong, it's just that it's correct only under very specific conditions, unknowingly to you.

IIUC, you did your homework: you made an improvement, you measured a 10-15% perf improvement, the PR was reviewed by other people, etc. It just so happens that this 10-15% figure is misleading because of an issue with the version of clang you happened to use to measure. Unless I'm missing something, it looks like a fair mistake anyone could have reasonably made. It even looks like it was hard to not fall into this trap. You could have been more suspicious seeing such a high number, but hindsight is 20/20.

Apparently, you still brought significant performance improvements, your work also helped uncover a compiler regression. The wrong number seems quite minor in comparison. I wonder who was actually hurt by this. I only discover the "case" right now but at a first glance it doesn't feel like you owe an apology to anyone. Kudos for all this!

Re: Performance of the Python 3.14 tail-call interpreter

#72

Post by acdha »

kenjin4096 wrote: post #51

Hello. I'm the author of the PR that landed the tail-calling interpreter in CPython. First, I want to say thank you to Nelson for spending almost a month to get to the root of this issue. Secondly, I want to say I'm extremely embarrassed and sorry that I made such a huge oversight. I, and probably the rest of the CPython team did not expect the compiler we were using for the baseline to have that bug. I posted an apo…

I feel bad for you since a change on the order of 5% would still have been seen as a very nice bit of work. I appreciate the class with with you’re dealing with an honest mistake, and all of the hard work you’ve given the Python community.

Re: Performance of the Python 3.14 tail-call interpreter

#73

Post by MaulingMonkey »

jmillikin wrote: post #50
MaulingMonkey wrote: post #42

Earlier quoted context omitted.

error: could not convert 'true' from 'bool' to 'std::string' {aka 'std::__cxx11::basic_string '} I don't think anyone's claiming C nor C++'s dumpster fires have signed integer overflow at the top of the pile of problems, but when the optimizer starts deleting security or bounds checks and other fine things - because of signed integer overflow, or one of the million other causes of undefined behavior - I will pray for…

For context, I did not pick that type signature at random. It was in actual code that was shipping to customers. If I remember correctly there was some sort of bool -> int -> char -> std::string path via `operator()` conversions and constructors that allowed it to compile, though I can't remember what the value was (probably "\x01"). --- My experience with the C/C++ optimizer is that it's fairly timid, and only misbe…

> but aside from the Linux kernel I've never encountered developers who put the blame on the compiler.

I encounter them frequently.

99.99% of the time it's undefined behavior and they're "wrong".

Frequently novices who have been failed by their teachers and documentation (see previous rant using atoi as an example of the poor quality of documentation about UB: https://news.ycombinator.com/item?id=14861917 .)

Less frequently, it's experienced devs half joking out of a need for catharsis.

Rarely, experienced devs finally getting to the end of their rope, and are finally beginning to seriously consider if they've got a codegen bug. They don't, but they're considering it. They know they were wrong the last 10 times they considered it, but they're considering it again damnit!

The linux kernel devs aren't quite unique in "just because you can, doesn't mean you should"ing their way into blaming the compiler for what could be argued to be defects in the standard or fundamental design of the language (the defect being making UB so common), but that's probably among the rarest slice of the pie of people blaming the compiler for UB. Few have the will to tilt at that windmill and voice their opinions when the compiler devs can easily just blame the standard - better to keep such unproductive rants close to heart instead, or switch to another language. Something actually productive.

0.01% of the time, it's a legitimate codegen bug on well-defined behavior code. Last one I tracked down to a bug tracker, was MSVC miscompiling 4x4 matrix multiplications by failing to spill a 17th value to stack when it only had 16 SSE register to work with. Caught by unit tests, but not by CI, since people updated compiler versions at their own random pace, and who runs `math_tests` on their personal machines when they're not touching `math`?

Re: Performance of the Python 3.14 tail-call interpreter

#74

Post by lifthrasiir »

hun3 wrote: post #40
jmillikin wrote: post #34

Earlier quoted context omitted.

From a high-level academic view, yes, the compiler is allowed to perform any legal transformation. But in practice C compilers are pretty conservative about what they emit, especially when code is compiled without -march= . You don't have to take my word for it. Go find a moderately complex open-source library written in C, compile it, then open up the result in Hexrays/Ghidra/radare2/whatever. Compare the compiled f…

-O3 does autovectorization: turning your loops into a bunch of SIMD instructions, sometimes even drastically changing performance profile. If autovectorization is "not that much magic" then idk what else it is.

Nowadays it's -O2. I was also surprised when I first learned this.

Re: Performance of the Python 3.14 tail-call interpreter

#75

Post by lifthrasiir »

johnisgood wrote: post #59
WJW wrote: post #28

Earlier quoted context omitted.

C might be low level from the perspective of other systems languages, but that is like calling Apollo 11 simple from the perspective of modern spacecraft. C as written is not all that close to what actually gets executed. For a small example, there are many compilers who would absolutely skip incrementing 'a' in the following code: uint32_t add_and_subtract_1(uint32_t a) { a += 1; a -= 1; return a; } Even though that…

Why would you want it to increment 1 if we decrement 1 from the same variable? That would be a waste of cycles and a good compiler knows how to optimize it out, or what am I misunderstanding here? What do you expect "it" to do and what does it really do? See: https://news.ycombinator.com/item?id=43320495

It is unlikely as is, but it frequently arises from macro expansions and inlining.

Re: Performance of the Python 3.14 tail-call interpreter

#76

Post by DannyBee »

kenjin4096 wrote: post #51

Hello. I'm the author of the PR that landed the tail-calling interpreter in CPython. First, I want to say thank you to Nelson for spending almost a month to get to the root of this issue. Secondly, I want to say I'm extremely embarrassed and sorry that I made such a huge oversight. I, and probably the rest of the CPython team did not expect the compiler we were using for the baseline to have that bug. I posted an apo…

FWIW - the fix was merged since you wrote that blog post ;)

Beyond that - 3-5% is a lot for something as old as the python interpreter if it holds up. I would still be highly proud of that.

After 30 years, i've learned (like i expect you have) to be suspicious of any significant (IE >1%) performance improvement in a system that has existed a long time.

They happen for sure, but are less common. Often, people are shifting time around, and so it just isn't part of your benchmark anymore[1]. Secondarily, benchmarking is often done in controlled environments, to try to isolate the effect. Which seems like the right thing to do. But then the software is run in non-isolated environments (IE with a million other things on a VM or desktop computer), which isn't what you benchmarked it in. I've watched plenty of verifiably huge isolated gains disappear or go negative when put in production environments.

You have the particularly hard job that you have to target lots of environments - you can't even do something like say "okay look if it doesn't actually speed it up in production, it didn't really speed it up", because you have no single production target. That's a really hard world to live in and try to improve.

In the end, performance tuning and measurement is really hard. You have nothing to be sorry for, except learning that :)

Don't let this cause you to be afraid to get it wrong - you will get it wrong anyway. We all do. Just do what you are doing here - say 'whoops, i think we screwed this up', try to figure out how to deal with it, and try to figure out how to avoid it in the future (if you can).

[1] This is common not just in performance, but in almost anything, including human processes. For example, to make something up, the team working on the code review tool would say "we've reduced code review time by 15% and thus sped up everyone's workflow!". Awesome! But actually, it turns out they made more work for some other part of the system, so the workflow didn't get any faster, they just moved the 15% into a part of the world they weren't measuring :)

Re: Performance of the Python 3.14 tail-call interpreter

#77

Post by acdha »

johnisgood wrote: post #59
WJW wrote: post #28

Earlier quoted context omitted.

C might be low level from the perspective of other systems languages, but that is like calling Apollo 11 simple from the perspective of modern spacecraft. C as written is not all that close to what actually gets executed. For a small example, there are many compilers who would absolutely skip incrementing 'a' in the following code: uint32_t add_and_subtract_1(uint32_t a) { a += 1; a -= 1; return a; } Even though that…

Why would you want it to increment 1 if we decrement 1 from the same variable? That would be a waste of cycles and a good compiler knows how to optimize it out, or what am I misunderstanding here? What do you expect "it" to do and what does it really do? See: https://news.ycombinator.com/item?id=43320495

That’s a contrived example but in a serious program there would often be code in between or some level of indirection (e.g. one of those values is a lookup, a macro express, or the result of another function).

Nothing about that is cheating, it just says that even C programmers cannot expect to look at the compiled code and see a direct mapping from their source code. Your ability to reason about what’s actually executing requires you to internalize how the compiler works in addition to your understanding of the underlying hardware and your application.

Re: Performance of the Python 3.14 tail-call interpreter

#78

Post by vikramkr »

johnisgood wrote: post #59
WJW wrote: post #28

Earlier quoted context omitted.

C might be low level from the perspective of other systems languages, but that is like calling Apollo 11 simple from the perspective of modern spacecraft. C as written is not all that close to what actually gets executed. For a small example, there are many compilers who would absolutely skip incrementing 'a' in the following code: uint32_t add_and_subtract_1(uint32_t a) { a += 1; a -= 1; return a; } Even though that…

Why would you want it to increment 1 if we decrement 1 from the same variable? That would be a waste of cycles and a good compiler knows how to optimize it out, or what am I misunderstanding here? What do you expect "it" to do and what does it really do? See: https://news.ycombinator.com/item?id=43320495

I'm pretty sure that's replying directly to the comment about how c is close to assembly and that if you add that line of code somewhere you know there's a variable getting incremented. Doesn't really matter whether or not it's useful, the point is that the behavior isn't exactly what you wrote

Re: Performance of the Python 3.14 tail-call interpreter

#79

Post by acdha »

jmillikin wrote: post #25
eru wrote: post #20

Earlier quoted context omitted.

I know that for 'int a' the statement 'a += 1' can give rather surprising results. And you made a universal statement that 'a += 1' can be trusted. Not just that it can sometimes be trusted. In C++ the code you gave above can also be trusted as far as I can tell. At least as much as the C version.

I'll expand my point to be clearer. In C there is no operator overloading, so an expression like `a += 1` is easy to understand as incrementing a numeric value by 1, where that value's type is one of a small set of built-in types. You'd need to look further up in the function (and maybe chase down some typedefs) to see what that type is, but the set of possible types generally boils down to "signed int, unsigned int,…

> That means if you see `int a = some_fn(); assert(a I completely agree that C++ is orders of magnitude worse but I’ve seen at least a couple counter-examples with code almost that simple. A researcher I used to support compared each release against a set of reference results, and got a surprise when they didn’t match but his program was working. This turned out to be a new compiler release being smart enough to inline and reorder his code to use a fused multiply-add instruction, which had greater internal precision and so the result was very slightly different from his saved referenced set. GCC has -fexcess-precision=standard for this but you have to understand the problem first.

Re: Performance of the Python 3.14 tail-call interpreter

#80

Post by jmillikin »

gpderetta wrote: post #70
jmillikin wrote: post #53

Earlier quoted context omitted.

I'm not sure that's a counter-example -- what assembly do you think should be emitted for floating-point math on an AVR microcontroller?

It means that "a += 1` is easy to understand as incrementing a numeric value by 1" is not true and instead "it can be really difficult to map between the original source and the machine code". More examples of non-trivial mapping from C code to generated code: https://godbolt.org/z/jab6vh6dM

All of those look pretty straightforward to me -- again, what assembly would you expect to be emitted in those cases?

For contrast, here's the assembly generated for Haskell for integer addition: https://godbolt.org/z/vdeMKMETT

And here's assembly for C++: https://godbolt.org/z/dedcof9x5

Post reply on HN