Caveat Emptor: All the analysis below pertains mostly to type system features and other "surface observable" aspects of the lanugage. Newly maturing compilation techniques are certainly another reason for the recent explosion (e.g. Rust, Haskell). But I'll stick to what I (ostensibly) know.
> I have no particular background in programming languages design and theory
Well then, good news: most of these new languages -- especially those you mentioned -- were invented primarily with software engineers and programming in mind. That said, all are informed by ideas which emerged as PL design principles in the 1960's-1980's and became well-established in PLT academia throughout the 70's, 80's, 90's and 00's. Haskell is definitely an exception is many ways, but at least the essential ideas driving the type system design were there in the 80's. (with the possible exception of Haskell, where the "old ideas finally getting to market" analysis is a bit less true).
> I've been also on an exploration lately into the history of computer science and reading about the Lisp family and Smalltalk as they seem to viewed as the better designed ones.
I don't know about better designed. A better characterization is that they capture some essence -- lisp, smalltalk, SML, Haskell, etc. were all designed and implemented to demonstrate the feasibility of a certain programming style or discipline (as well how that approach makes certain problems really easy when they weren't easy before.)
> So what I don't understand and hope somebody here could shed some light on it is what's with all the new languages?
> How many of them really bring something new to the table, a better way than the old one?
> How is Go or Rust better than C C++ Ruby Python Lisps Java Smalltalk Erlang and whatnot.
A detailed answer would consider each pair. But broadly:
* These languages typed, which contrasts them from the dynamic family (including lisp).
* These languages tend to favor composition over inheritance, which differentiates them from (canonical) Java.
* These languages tend to make typed functional programming first-class (syntactic and compiler support for lambdas; pattern matching; etc.)
* The examples you've provided -- Rust, Go, Swift -- are more systems-oriented than Java and are not based on a VM.
* Lots of smaller things. E.g. apparently avoiding C++'s slow builds were a major design point for Go.
> Are those languages designed for very specific cases where older languages can't cope with.
Yes. All are designed to address some significant flaw with existing languages. Most were created because for an important set of language requirements, there exists a language which fulfills each requirement but no single language which fulfills all requirements. (Again, Haskell stands out as an experiment with laziness if I understand the history correctly).
> When I read about Smalltalk or Lisp or Haskell people regard them as the pinnacle of programming language design and yet their popularity isn't really proportional to those statements.
> How do languages get popular?
This is an area of active research (search for SocioPLT [1]). The common wisdom is "library support + important problem niche". The library thing strikes me as tautological.
> Money, syntax, portability?
The first is certainly a major reason the # languages exist :-)
> Why did PHP rule the 90' and not Common Lisp or Erlang or whatever.
Oh dear. Let's just agree that "quality" does not equal "popularity". Bieber > Vienna Philharmonic?
> Why do I read so much bad stuff about C++ from smart people yet it's one of the most popular languages. Why isn't Objective-C more popular since it is too C with classes? Why Java and not Self?
You'll receive lots of conjectures. I'll leave that business to others.
[1]http://www.eecs.berkeley.edu/~lmeyerov/projects/socioplt/pap...
edits: formatting, adding link to splt