Even a small part of a sign should be enough. They're designed to be easy to spot.
It seems like we just set neural networks up to recognise all objects and assume they'll recognise simple objects too. However typically how humans learn is through simple cartoons first and then layer on top of that rather than the other way round.
Edit: should have done some googling before opening my mouth [0] [1]
[0]: https://amundtveit.com/2017/07/13/traffic-sign-detection-wit...
[1]: http://www.bartlab.org/Dr.%20Jackrit's%20Papers/ney/1.TRAFFI...