>how can I have a package that sets every "the" in red colour, without it interfering with a package that sets every "the" in blue or titlecase?
Exactly.
Broadly, the things you might want a package/plugin to do can be categorised as "local" (e.g., add some new type of content in a fixed-size box that the layout engine can then treat the same way as it treats any other such box) or "global" (e.g., change where floats appear). Making local effects play together can be easily handled with standard PL ideas like lexical scoping, but doing the same for global things is much harder: it strongly depends on exactly what knobs (API) the base typesetting engine provides. We now have design patterns like Observer to help make creating such "global" effects simpler, assuming that their effects are genuinely orthogonal, but what if they aren't?
A plugin that sets each "the" to red clearly conflicts with a plugin that sets each "the" to blue: at most one of them can "win", so which is it? Does it depend on which plugin was loaded first? If so, that's no better than LaTeX, and will become an ever-growing headache as the ecosystem grows.
OTOH, a plugin that sets each "the" in italics can sensibly interoperate with a package that sets each "the" in bold -- and can even produce the same results regardless of which was loaded first, because these effects "commute". These effects could be implemented by having the base engine expose an event that can be listened to by any interested plugin.
ETA: The main reason LaTeX is a pain is because it makes no real attempt (that I can see) to manage any of the inevitable complexity of "global" effects. (Admittedly, this is a tough design problem.) I don't yet see signs of anything better from Typst, thus I assume it will become roughly as painful as LaTeX in time.