There are known side effects (short term cognitive decline). In addition - there are unresolved questions about whether those effects are persistent or not, and whether age plays a role.

What you said is: "[anaesthetic agent] has no observed side effects in general" and that's just not true.

As an aside - my grandmother had hip surgery at 86 and was never the same mentally afterwards. She could walk, but she couldn't remember who her family was. It was not particularly unexpected (she had an existing diagnoses of alzheimers) but the change post-surgery was notable enough that the entire family discussed it. It was like she had been slipping a bit, but the surgery threw her off a cliff. The difference between knowing who you are, but discussing the weather every half hour or so, to having no idea who the people around her were.

Personally - I would firmly place anesthetics into a fairly risky category of drug that we have a very poor understanding of. Which is decidedly not where you are placing it.