On an object level it's not a big deal, but you're not thinking about it with the correct framing.
Being invited as a keynote speaker lends you credibility and status. It means you have been acknowledged as someone worthy of giving a keynote. You may not care about this, but other people do.
To downgrade someone from a keynote speaker to regular speaker is extremely disrespectful. Especially someone that you invited.
It makes it seem like you are toying with this person, and based on JT's account someone on the Rust team was toying with JeanHeyd because they went behind the rest of the team's back to downgrade his talk.
It's like giving a child a toy then snatching it back. It's just cruel. Whether it was a mistake or not doesn't change that.