Finding a CPU Design Bug in the Xbox 360 (2018)

The PSX's CPU is semi-custom. The core is a reasonably stock R3000 CPU, but the MMU is slightly modified and they attached a custom GTE coprocessor.... I guess you can debate if attaching a co-processor counts as custom or not (but then the ps4/xbone/ps5/xbs use unmodified AMD jaguar/zen2 cores)

IMO, the N64's CPU counts as off-the-shelf... however the requirements of the N64 (especially cost requirements) might have slightly leaked into the design of the R4300i. But the N64's RSP is a custom CPU, a from scratch MIPS design that doesn't share DNA with anything else.

But the Dreamcast's CPU is actually the result of a joint venture between Hitachi and Sega. There are actually two variants of the SH4, the SH4 and SH4a. The Dreamcast uses the SH4a (despite half the documentation on the internet saying it uses the SH4), which adds a 4-way SIMD unit that's absolutely essential for processing vertices.

We don't know how much influence Sega's needs had over the whole SH4 design, but the SIMD unit is absolutely there for the Dreamcast, I'm pretty sure it's the first 4-way floating point SIMD on the market. The fact that both the SH4/SH4a were then sold to everyone else, doesn't mean they were off the shelf.

Really, the original Xbox using an off-the-shelf CPU is an outlier (technically it's a custom SKU, but really it's just a binned die with half the cache disabled).

The difference is that on PowerPC, 32bit mode on 64bit processors (clearing the SF bit in the MSR) is just enabling a hardware 32bit mask on the effective address before it gets translated into a virtual address.

Unlike on x86-64 and arm64, there's no free (or even that cheap) way to do an ILP32 abi purely in software. x86 and arm allow encodings for memory reference instructions that only use the bottom half of the registers (the E* registers on x86, and the W* registers on arm64). No such encoding exists on PowerPC for memory reference instructions, so you'd be stuck manually masking each generated pointer.

Because of that, the compiler hacks you're talking about are kind of the opposite from what you're describing. The hacks are because on the upstream gcc PowerPC backend, having a 32bit pointers in hardware and having operations that operate on 64bit quantities had the same feature flag despite technically being able to be separately enabled on actual hardware. It was just very rare to do so. So the goal of the hacks was to describe to the compiler that the target has 32 hardware pointers, but still can issue instructions like ldd to operate on the full 64bit GPRs.