I have a 4 year old, in my mid 40s, and think about this a lot for some reason. In the 70s during the summer, when I was around 7, we would wake up, eat breakfast, go out and about wherever we wanted, come back for lunch, out again, come back for dinner, and then out again past dark to play flashlight tag. I think I would be charged with child endangerment these days if my kid (3 years in the future) was doing stuff…
What part of the US do you live in? I have 3 kids ages 7 to 12 and they and all of the neighborhoods kids wander together over a distance of several miles. We make sure they have a phone and a time they are expected to return. The kids always group up as there is more safety in numbers. Life isn't worth living if you don't actually live.
We used to live in suburban Washington DC and I didn't let my kids out of sight of the house.
Now we live in suburban Denver. We cared about play a lot so we used it as the main criterion of where we bought a house. We drove through neighborhoods on Saturday mornings until we found one where kids were out riding bikes and playing without adult supervision. We found a neighborhood that was both well designed (pools, a network of walking trails away from roads, school in walking distance) and full of people who were culturally supportive of free-range kids.
If you want to find or help to foster such a neighborhood, I recommend the book we used, "Playborhoods": http://playborhood.com