I also have three Chromecasts and occasionally 7-8 mobile devices that are connected to them in multi-AP network. I have mostly Sony devices and I think they have a rather aggressive powersave profile.
The last 3-6 month or so, the mobile devices have been losing connections to Chromecast while playing, and the network have become markedly more flaky also for other devices.
I actually got a Chromecast Ultra with an ethernet connection in case the wifi was bad, but it didn't really make a difference.
At some point a D-Link AP that is also the main router started to hang at least once per day, occasionally every 5 minutes. I installed Lede on it in order to rule out hardware problems - and then it stopped crashing at which point I stopped bothering, but I suppose the hardware can still be cooked.
I've noticed that the network occasionally is really slow for a short while - I managed to measure the speed to an external provider to be around 4-5 mbit/s instead of 100, but I haven't considered the possibility that it's my own network that goes bananas.
The WAF is declining, but it's so much junk that can go wrong so it will take a considerate amount of time to debug.
Edit: I must say though, that I expect my routers and APs to survive 100.000 packages...