The point is that when you use "very", you reveal that you have not attained that level of intellect which realizes that "very" is extraneous almost every time. This insight usually comes with a large amount of time invested in reading challenging books. It has little to do with expansive vocabulary, and more to do with understanding word choice and avoiding pretension.
We pick "very" when we have not trained ourselves to be conscious of word choice. When we become conscious of it, we stop saying "very" quite naturally.
And we start using annoying words like "quite".
For anyone trying to sound like they have a better grasp of English, definitely lose the "very".
Here's an example: which is better? "gregarious" or "very outgoing"?
(It's a trick question - they're both bad... the correct answer is "outgoing")
If you want to sound actually smart, remove words like "very", and don't change "outgoing" to "gregarious". The only people who use "gregarious" in a normal conversation are people who are trying to sound smart. A smart person is conscious of how to avoid using pretentious words in normal conversation. Smart people usually don't _need_ to demonstrate that they are smart.
To sound intelligent, simply use the tone of your voice to indicate that you could say "very", but instead you just say "outgoing" - perhaps with some inflection or cadence variation. Smart people notice when someone omits words like "very", because they have been faced with the same issue.