Quite a while ago, I learned the best way to follow the ball or puck at a sporting match. You don't try to look for the little object itself, you just observe what all the relevant players are doing, where they're looking, where they're running. You figure it out really quick with them helping!
Likewise, if I am waiting for a bus or something, I am really alert and on guard if I'm alone at the stop, because that bus will blow right by. Once there are a few other passengers around, I kick back and just watch them, because unless they're all impatient and pacing, their attitude and behavior will indicate when the bus is visibly arriving.
SDCs that coldly and analytically examine the traffic around them will probably never get the human nuances that we can pick up from second-and-third-order clues. For example, if I'm paranoid about whether a car's going to turn into me in an intersection, I don't watch their turn signal or lateral motion, I watch their front tire, because that's the first indicator of a turn. Is that sort of thing baked into AI? If several people are stopped at an intersection, eye contact and gestures are often essential in negotiating a solution. AI? Same with pedestrians. Make eye contact, signal, work something out. I'll often stop on a corner and wave cars through rather than insist on my right-of-way. Of course the motorist also has to decide if that's safe to do, and sometimes you can fool a motorist into making an unsafe move just by signaling your permission...