QMK supports ARM very strangely, but the more I think about it, the more I like it. They use ChibiOS as an RTOS to basically schedule USB code alongside QMK, and then they get ChibiOS's USB support for free. (Note that on many of these ARM chips, the USB is actually handled in hardware; you just need to get the right bytes into the right memory addresses at the right time. You don't have to be awake at the exact instant to toggle a USB pin at exactly the instant the protocol demands. So it works quite well; ChibiOS is not preempting your matrix scan at random times or anything.)
Having said that, I think it's reasonable to write your own firmware. QMK feels very very legacy right now. It is hard to use modern dev tools to work on the code (like clangd), and the newer chips are supported in a very janky manner because of assumptions made for the original AVRs that it ran on. I always thought it was weird that the boards that OLKB use are among the most jankily-supported, even though they fund the development of QMK. They also don't really maintain all of the in-tree examples, so you'll copy someone else's outdated keyboard code and be starting with code that barely works. (I made an STM32F103-based keyboard recently, and the example code actually has their own debouncing algorithm, rather than calling QMK's newish built-in debouncing algorithm. So you get latency by default, although it hasn't bothered me at all so I haven't fixed it.)
The model of flashing firmware to change the key layout also seems crazy to me, in an era where you can easily enumerate as a flash drive and just drag a JSON-encoded keymap onto it to adjust the layout. (Or just edit it live, a la CircuitPython.) I watch new users of QMK-based keyboards struggle and give up on a somewhat regular basis. Seems strange to me.