I don't know much about AVIF (I guess the same concern applies though), but for JPEG XL there are actually two knobs for compression: distance or quality (`-quality` in ImageMagick) and effort (`-define jxl:effort`).
Distance is the target perceptual metric (Butteraugli), while effort controls a variance of the perceptual metric of the encoded image---which can be never accurate. Higher effort means more consistent quality but more computation as well. The default for cjxl, and probably also ImageMagick, is 7 ("squirrel"), which is very high and very slow as you noted. This is much like Brotli defaulting to the highest setting (-11) because both assume that you can spend much more time in compression. If it's not your assumption you should change that.
I do feel that this aspect of JPEG XL is not well communicated. Normally you have a single knob (quality) and increasing quality means more time spent for compression, but in JPEG XL this can be easily decoupled---very useful concept but also relatively alien one. A common mistake is to set the highest effort (9, nicknamed quite appropriately, "tortoise") for speed benchmark, which suggests that there will be people doing this in the production and complaining that JPEG XL compression is very slow, not realizing they can live with much lower efforts.