Live data from Hacker News

Ubuntu 24.04 LTS will enable frame pointers by default

ubuntu.com

61–70 of 121 posts

Re: Ubuntu 24.04 LTS will enable frame pointers by default

#61
post #2

Frame pointers are such a destructive micro-optimization to omit by default, I am beyond excited about this collaboration with the folks at Canonical to make Ubuntu debuggable by default!

> to make Ubuntu debuggable by default!

How's that, you'd still need the debug symbols

Also has anyone else noticed that running stuff through Valgrind is really only possible if the program was made with Valgrind in mind? For example, Python and its many extensions generate numerous errors and warnings, so many that any real problem becomes hidden.

I'd say that modern Linux systems are very far from being debuggable.

Re: Ubuntu 24.04 LTS will enable frame pointers by default

#62
This is a good idea for the short-term. As of now, frame pointers are the most reliable way to ensure that software can be profiled by tools like perf*. The core issue is that the kernel must be the one to unwind the userspace stack, and it only knows how to unwind stacks with frame pointers**. The .eh_frame data will never be supported by the kernel, because it involves a turing-complete program that must be executed to compute the necessary unwind info***.

For the long term, the more exciting option that's emerging is SFrame[1]. This is a new data section which would be generated by the compiler and contains unwind tables which the kernel will be able to understand. Unlike DWARF/.eh_frame, these tables would remain in the final binary (i.e. not be stripped away), and on exec(), the kernel would store them for use during profiling. Since the format is quite similar to ORC(*), and Steven Rostedt is quite invested in the format, it seems a safe bet that support will land in the kernel.

My hope isn't necessarily that a distribution completely disables frame pointers once this format becomes available... though it could be an interesting thing to try. Rather, there can be a conscious choice about whether frame pointers are used, or SFrame, which would be useful for cases like Python, where it's mentioned that frame pointers may still have a significant performance impact. The kernel should be able to fall back to frame pointers when SFrame is unavailable, which means that either will be acceptable. Ideally, in a few years time we'll be able to go back to forgetting about frame pointers for most cases :)

---

* Ironically, the kernel itself tends not to use frame pointers! It has its own unwind format called ORC, which gets generated by an in-kernel program called "objtool" which essentially reverse engineers the assembly generated by the compiler. It's x86_64-specific and frequently needs adjustment when the compiler changes code generation. It can't be used for userspace programs.

** it also knows how to unwind kernel stacks with ORC (see above)

*** There is an option to allow perf to unwind with DWARF, but it's a total hack (though a very effective one). By passing --call-graph=dwarf, you can instruct the kernel to copy the userspace stack (by default, 8k bytes!) into the perf event buffer with each sample (this can be as many as 100 or 1000 samples per second, per CPU...). Later, the perf userspace program will use that info, along with information about each process's address space, and the debuginfo for each program, to unwind the stacks. This has huge performance overhead, and it requires that you have easy access to debuginfo, which may not be the case, especially for container workloads.

[1] https://lwn.net/Articles/940686/

Re: Ubuntu 24.04 LTS will enable frame pointers by default

#63
post #53

Earlier quoted context omitted.

Well, for a start you probably mean compiled for an arch with 16 registers. It doesn't actually matter what arch the compiler ran on (assuming modern cross-compiler like GCC). If a program uses 16 registers then it needs 16 registers. But note it's the program itself that decides to use a frame pointer, it's not being reserved by the operating system or something. Programs don't even have to use the stack pointer as…

Ohh, I misunderstood what Ubuntu was doing. So the only change is that GCC and its toolkit will compile programs using a register as frame pointer by default? That seems like a very reasonable change. If having an extra general purpose register is critical for performance of then this can be disabled in that program's the makefile.

Correct and there are already known exceptions such as the python interpreter in which the “interpret function” function actually falls into this case and so for the foreseeable future python is going to continue to be compiled omitting frame pointers. But by default this destructive micro optimization is off by default until proven a performance bottleneck, just like any performance issue should be!

Re: Ubuntu 24.04 LTS will enable frame pointers by default

#64
post #38

Earlier quoted context omitted.

There are two key effects of this decision. The first effect is that it makes one additional general-purpose integer register unavailable for use for code. x86-64 has 16 general-purpose registers, but one of these is the stack pointer and basically can't be used for any other purpose; this would add a second reserved register for the frame pointer. This effect may cause slowdowns if the 15th register was critical for…

Could programs compiled in architectures with 16 general purpose registers fail in one with 15?

Not really? I mean, there are architectures with fewer than 16 GP registers (IA32 is one of them); if you can compile some C code on that architecture, then surely you can also compile it on x86_64 with 15 (well, 14, really, due to the stack pointer also being reserved) rather than 16 (15).

If someone is writing in assembly, then they've already decided if they're going to allocate a register for the frame pointer, and Ubuntu's change isn't going to affect that, as this is about compiled code, not assembled assembly.

The only real issue is performance: if a program has a particular hot-path function (or just many functions overall) that really benefit from having that extra register available, and would otherwise have to spill data into memory, then this change could have a big negative impact. But that's not really a big deal; the packager can decide to omit frame pointers just for that particular app or library.

Re: Ubuntu 24.04 LTS will enable frame pointers by default

#65
post #6

Earlier quoted context omitted.

This option was almost certainly a holdover from the bad old 32-bit x86 days, when disabling frame pointers gave you a seventh valuable general-purpose register. It's no longer beneficial on x86_64 -- even with rbp locked down, you still have fourteen registers there.

That much is true -- the difference between 6 and 7 registers is much larger than the benefit of going from 14 to 15. However, even under zero register pressure having a frame pointer is still an extra register that needs to be touched on every function invocation, extra instructions taking space in the I-cache, etc. It's a small thing, but it's still a cost that has to be paid by all compiled code. I'm not going to…

Ye. Bytecode interpreters very much benefit from the extra register. And any function that looks similar to one.

I mean e.g. getter and setter functions get alot of extra code to run.

Re: Ubuntu 24.04 LTS will enable frame pointers by default

#66
So the whole world should take a 1-2% performance penalty on everything so some users can maybe run a profiler?

Wouldn't it make more sense to just have an 'apt reinstall all --with-frame-pointers' command that power users could run before they wanted to profile something?

Re: Ubuntu 24.04 LTS will enable frame pointers by default

#67

This is a good idea for the short-term. As of now, frame pointers are the most reliable way to ensure that software can be profiled by tools like perf*. The core issue is that the kernel must be the one to unwind the userspace stack, and it only knows how to unwind stacks with frame pointers**. The .eh_frame data will never be supported by the kernel, because it involves a turing-complete program that must be execute…

We’ve also figured out an alternative format to use from within eBPF to unwind stacks (we happen to only support dwarf at the moment but theoretically any source information could work): https://www.polarsignals.com/blog/posts/2022/11/29/dwarf-bas...

Re: Ubuntu 24.04 LTS will enable frame pointers by default

#68
post #29

hasn't this already been in fedora for almost a year?

Yeah. To be fair there is a stronger argument for it being enabled in fedora than an LTS release (RHEL or Ubuntu) since it has more cutting edge software that needs more frequent debugging, is less likely to be used in production where the (minor, but uneven) performance hits may matter, and has so many upstream developers using it as their daily driver.

Agreed that cutting-edge software likely needs more frequent debugging, but I don't think that means LTS releases shouldn't be easier to debug.

Consider that you're a big company deploying software to hundreds or thousands of machines, and you hit a difficult-to-diagnose performance issue, crash, etc. You'll very much appreciate if the OS has made it easier for you to debug things.

Put another way, Fedora users/developers might appreciate having frame pointers because they have to debug more frequently, but RHEL/LTS release users might appreciate frame pointers because on the less-frequent occasion when they need to debug, the stakes are much higher.

Re: Ubuntu 24.04 LTS will enable frame pointers by default

#69
post #61
post #2

Frame pointers are such a destructive micro-optimization to omit by default, I am beyond excited about this collaboration with the folks at Canonical to make Ubuntu debuggable by default!

> to make Ubuntu debuggable by default! How's that, you'd still need the debug symbols Also has anyone else noticed that running stuff through Valgrind is really only possible if the program was made with Valgrind in mind? For example, Python and its many extensions generate numerous errors and warnings, so many that any real problem becomes hidden. I'd say that modern Linux systems are very far from being debuggable…

You offent need to make excemption file.

My tip is to run some thing one program execution, and then run it twice during the same program execution, and diff the reports.

Re: Ubuntu 24.04 LTS will enable frame pointers by default

#70
post #61
post #2

Frame pointers are such a destructive micro-optimization to omit by default, I am beyond excited about this collaboration with the folks at Canonical to make Ubuntu debuggable by default!

> to make Ubuntu debuggable by default! How's that, you'd still need the debug symbols Also has anyone else noticed that running stuff through Valgrind is really only possible if the program was made with Valgrind in mind? For example, Python and its many extensions generate numerous errors and warnings, so many that any real problem becomes hidden. I'd say that modern Linux systems are very far from being debuggable…

Ubuntu hosts a debuginfod server that you can automatically discover debuginfo from any binary from using the binary’s build id.

https://sourceware.org/elfutils/Debuginfod.html

Post reply on HN