Personally, a voxel space renderer for the PlayStation (Comanche terrain style rendering).
In theory, it's really not suited for this because (at least back then) this was designed for computers with:
- relatively large amounts of RAM (uncompressed color+heightmap data takes a lot of space),
- fast CPUs with data caches (it's a 2D trapezoidal walk of a top-down projected fustrum with plenty of additions, multiplications and divisions),
- memory-mapped linear framebuffers (for software rendering).
The PlayStation has none of these things: only 2 MiB of main RAM; an in-order scalar 33 MHz MIPS III processor without a FPU or a data cache, where any CPU load from main RAM stalls the pipeline for 5 cycles, multiplications have a 6 to 13 cycles latency and divisions have a 36 cycles latency; and the only way to touch VRAM is to send commands to the 2D GPU.
What it does have is 4 KiB of I-cache (direct-mapped), 1 KiB of scratchpad with no wait states and a fixed-point GTE coprocessor which is mostly geared towards projecting 3D points onto a 2D screen and not general matrix/vector processing. Meaning that if I get my hot loop within 1024 instructions (and no function calls), fit my working set+stack within 1 KiB and can pipeline the GTE for transformations and the CPU for map scanning/GPU submissions, it might just work.
So far I'm getting decent framerates (like 10-15 FPS at 256x240) with just software projection and GPU line rendering. Right now I'm wrestling with the GTE and various ways to trick it into transforming more points than it's supposed to be able to do, by looking at the fixed-hardware math equations for the various instructions and trying to fit my equations into them (doing stuff like putting relative altitude into X/Y vector registers instead of coordinates and massaging the rest of the values to get screen Y coordinates out of it).
Kind of a hardcore topic to pick for my first homebrew ever, but I craved a simpler, low-level optimization challenge as a pipe cleaner, after working on ghidra-delinker-extension for so long.