Like you said, my M1 Mac mini “officially” supports two monitors plugged in directly.
It’s nice that DisplayLink connections let an M1 machine have unlimited displays, and that the performance of their driver under Big Sur and above has been fantastic (for me) for the past half year driving 2 extra displays. (Four total, 1 is vertical)
I sometimes add an iPad in Sidecar mode, but that needs to connect before DisplayLink or it will error out, likely since the mini is flabbergasted at the number of monitors it sees attached at that moment. Since my DisplayLink connections are both on the same ThinkPad dock (1 built-in, 1 USB), I just toggle the dock off, connect the iPad with Sidecar, and turn it back on. After a moment everything is working as expected.
BetterTouchTool has a newish feature to capture your current window layout across all open apps and displays and lets you assign a shortcut to put everything back where you intended. I use it after first boot if needed, and keep one for ‘4 monitors’ and one for ‘5 monitors’ (when using Sidecar with iPad)
On-topic: Thankfully the Apple Silicon USB ports are more than fast enough for DisplayLink adapters, which (in my experience) are often using an early version of USB 3.0 aka USB 3.1 Gen 1 aka USB 3.2 Gen 1.