Let's see...
1.) You need a shit-load of people to play baseball anything like for-real. At minimum 15-20 people for two teams. Pretty tough to do as a kid, unless you're free to roam free all over town to gather a crew, which approximately 0% of kids have license/protection from CPS to do.
2.) You need a big enough area for a baseball diamond, where you don't have to worry about knocking out a window with a foul ball or an errant throw. Mostly, you aren't allowed to use the actual baseball fields to just play pickup. Compare with soccer or football, where any old open area is good enough, or basketball, which needs a hoop but probably takes the least space.
3.) Everybody needs at least a baseball glove. You also need a bat and a ball or two. Compare with sports that really don't require any specialized equipment, like soccer, football(two-hand-touch or recess-tackle style), and basketball.
4.) If you live in the northern half of the US, the season for playing baseball is almost exclusively during the summer months, when school isn't in session.
5.) Baseball on TV is maybe a half-step better than watching paint dry, and you have to pay for the privilege. Since they cleaned up the steroids, it's boring as hell.