What is the line is a good question. I'm strongly pro-user agency, but I still think consent is more around what actually impacts the user. If there's some safeguards, for exceptional cirumstances that users can get into if they need it, I feel like the software has to be doing more, having more of an impact on you for there to be an offense or problem here.
That said, I do want to amplify agency. I don't immediately know what to expect for disarming this. If a website starts hitting the API heavily and my machine's fans are spinning up, where am I at, and what do I expect? It feels like the web is close already, with a pretty sophisticated permissions model, where we go to look for things. I'm interested in an evolved permissions model for the web, where even when permissions are on by default, it's the same flow to turn them off. I think that would remove a lot of the grounds for "I don't want this" that seems so persistently abundant these days.
Even it feels like the risk is so low/non-existent, if the user's demanding less agency from the their user agent, in principle I guess we ought give them the less that they asked for. Usually. But that always has some kind of practical limit too. CSS made some people mad! It's ok for this not to be the software for you, for you to go need to go somewhere else.
I believe that relatively inert capabilities like this, where mostly it's taking up some storage space and joules, is generally not really altering the contract, and is fine.