>LLM responses are certainly not idempotent, as they are not even deterministic.
Isn't that more due to an optimization and not how the LLM itself runs?
Like a MoE LLM run on a single input should give the same output each time. But this is inefficient, as any given token is hitting 1 (or maybe 2 or 3) experts at a time, meaning all the other experts are doing absolutely nothing. So you upgrade it to take in multiple requests. But then any given expert can become a bottleneck, so when too many requests need a given expert, some of them are routed to a second or third best expert instead. Within the context of any single request, this looks like non-determinism, but it is still deterministic when considering the full batch.
For everyday users and everyday use cases, that is enough to treat it as non-deterministic (the harness might also send in unique data like current time which means one can never have the exact same request twice), but when talking about LLMs more theoretically, I think we need to consider they can still be ran deterministically even if that isn't as optimized.
Similar with temperature. 0 means deterministic, but anything higher with a seeded value is deterministic. If anything, temperature is us purposefully adding non-determinism to agents because they were too deterministic.