Sorry, false memory - I remembered the full story in all of its gory details after sleeping on it.
We did use magenta and colors near it for reserved pixels that would never be seen onscreen, but in our case it was for color animation. The Mac couldn't do full-screen palette animation in a way sanctioned by the OS, because Apple arbitrarily inserted an internal wait for the vsync monitor refresh interval in all of its palette functions, with no way to disable it or directly access the low memory variables that controlled the color lookup table (CLUT) like on the PC. So an empty main loop with palette animation ran at 60 fps, but doing any draw calls at all caused a timing miss which dropped it to 30 fps, while the CPU sat at about 50% idle. Our games redrew the whole screen anyway, so we opted to translate pixel colors on the fly via our own lookup table instead.
Apple also didn't provide OS calls for page flipping (to draw the next frame of animation while the current one is shown to double the frame rate), probably by design to maintain the Mac's image as a "professional" desktop computer, because such tricks were well-understood in the gaming industry. Or video resolutions below 640x480 (sometimes 512x384 on certain models).
Apple also tended to ship machines with half-width busses (supposedly to reduce cost) like the Mac LC, which reduced memory bandwidth so much that full-screen scrolling was difficult to achieve.
Those decisions prevented the Mac from becoming a performant gaming system, even though the RISC-like 68k chip with its numerous registers, predictable instruction set format and unsegmented memory were far superior to PC architecture at the time IMHO.
Later PowerPC chips like the 603e had a cache misalignment issue where double-width 8 byte memory copies that weren't 8 byte aligned ran at about half speed (probably using 2 copies internally) so I think we had to drop down to single-width 4 byte copies or use a cache hint function to disable caching while copying image buffers, which ran slightly slower.
Notable snafus included years-long delay of support for newer OpenGL versions, so we were stuck with fixed-pipeline 1.x calls long after the PC was exploring shaders. Then iOS only supported OpenGL ES, with no real reason not to offer an ES compatibility layer on desktop, necessitating support of 2 codepaths. Instead of remedying that stuff, they introduced Metal, which nobody asked for.
Not to mention deprecating wide swaths of the OS, forcing rewrites from MacOS 8 to 9 (Carbon), then from 9 to X (Cocoa), then from MacOS to iOS (Objective-C and Swift). Don't forget the 68k to PowerPC to Intel to ARM chip migrations, which forced developers to be aware of endianness issues, which greatly increased the complexity of reading/writing binary files.
Combining all of those permutations, MacOS software would often only survive perhaps 3 years before needing a rewrite. I probably have 10 times as many applications (mostly old games) on my Mac with a no-smoking sign through them as runnable applications.
I'm reminded of the expression "lemons for the price of peaches". The outer elegance of the Mac obfuscated the underlying byzantine layers. Denial became woven into the Mac experience, so much so that developers took a certain level of trauma to keep up appearances. I know I did. That's why I got out of the biz in the early 2010s after so many of our games that we put so much work into turned out to be commercial failures. We might have made 10 times more money targetting the PC, and conceivably 100 times more if we had cross-platform resources like Unity and Steam.
I bring this stuff up because rose colored glasses often obscure what really happened. Especially now with political insiders and the media producing so much revisionist history. Stuff we remember as cutting-edge manifested because the state of the art at the time was so abysmal.
I look around today and I see a whole lot of assumptions being made that this is all there is. That the current path of tech is the one true way. When nothing could be further from the truth. We're ruled by powerful duopoly forces presenting the illusion of choice, when all eggs are in the GPU basket. But do they use GPUs on Star Trek? Probably not.