That weird valley is essentially where you become a high school teacher. Most of the time if you're talking to that few people, it's because you're trying to give a lesson in some form or other.
I've spoken in front of large groups of students of all disciplines, large groups of business types and small to mid size groups of scientists. Large in this case is several hundred.
Undergrads get bored easily, but you can make it fun. Scientists are hard, simply because they know their stuff and get a knack for asking really annoying questions, the ones you didn't quite prepare for. Business people are simple in principle - everything has to be dirt simple (could your Mum understand it?) and usually to the point (what are you selling and why?). The hard part for business talks is that usually you can't be remotely technical because people will tune out.
With very few people it's easy to be flexible - odds are you know them reasonably well or you know enough about them that you can throw in a bit of banter, make it fun and respond to their queries well.
Larger groups you're just a drop in the ocean. The bigger the talk, usually the less time there is for interaction. So you can do your spiel, have a couple of short questions and leave. I find they're fairly easy as everyone blurs together.
In between is hard. You're acutely aware if people get bored - they won't hide it like small groups. You may have to answer fairly involved questions too. The only way, as ever, is to practice and make sure you know what you're talking about back to front.