All in all after watching several of their videos I feel like I do a good job of recognizing the drowning person, but I’m amazed at how quickly the lifeguards spot it and dive into action. Even knowing that in this short clip there is absolutely someone drowning I still have doubt, but the lifeguard who doesn’t have that context is already half way to the person by the time I’m sure.
How long of a shift do lifeguards even have? Because I wouldn't feel confident in being able to observe a bunch of people in a pool for hours for what are quite subtle clues.
Guards typically have discrete areas they cover, and within those boundaries you can check for a certain number of people and boundary crossings. By doing things like mentally running through the list/number of people in your area and noting the higher-risk ones, it's a little less mentally taxing than you might think. Combined with some overlap between guard stations and regular rotations between areas, coverage can be pretty good. The real difficulty comes in when you start adding numbers of people- the boundaries become less clear, the number of high-risk swimmers increase, and the total cycle time through everyone you're covering increases such that your margin of error between a situation presenting and your time to react starts decreasing.
Anecdotally, I always found the behavior changes from a normal to tired swimmer to be one of the easier parts. Lack of forward motion, intense focus on the activity of swimming itself, falling behind from a group. Again, it's more difficult the more crowded an area becomes.