Semitic languages never had a neuter gender. That's one of the prominent differences between classic Semitic and Indo-European languages.
Otherwise this description is pretty correct, but it omits an important part. Middle English was highly influenced not by Anglo-Norman (the dialect of French spoken by the Norman nobility who conquered and settled in England), but also by the Old Norse spoken by the Vikings who conquered and settled the Danelaw before them.
English has gotten about half of its vocabulary from Anglo-Norman (either in French or Latin form) and this is truly a tremendous influence. It is interesting to note, however, that Anglo-Norman mostly supplemented — not replaced! — native Old English words, and often provided a more aristocratic alternative to them. Thus the peasants who raised chicken, cows, pigs, calves and sheep for meat used Old English words to refer to the animals, but the richer classes who actually ate the meat used Anglo-Norman words to refer to it: poultry, beef, pork, veal and mutton.
Old Norse, on the other hand, probably doesn't comprise more than a small percentage (2-5%) of Modern, English vocabulary, but the parts it affected are much more core to the language, and it often replaced the Old English word, which often had a very similar sound: e.g. sister (Old Norse: systir, Old English: sweoster), egg (OE: ǣġ, pronounced 'ey'), sky (ON: ský, OE: heofon - which became restricted to 'heaven'). Even the plural pronoun "they" comes from Norse (ON: their, OE: hīe).
The loss of grammatical gender and case was probably almost complete by the 1066, when William the Conqueror invaded England. The Middle English that we start seeing a little after the Norman conquest is already missing most of the case and gender endings, which led linguists to believe that this change happened earlier, but was not reflected in the conservative writing. As you say, what keeps language unchanging is the ruling class, but this is ONLY true for the written language, which can be policed. The nobles and clergymen can't very much execute every single peasant who drops a grammatical case. But once the Old English ruling class was replaced with a ruling class who only cared about preserving French, and the writing standards for Old English lost hold, the preexisting changes started surfacing in writing.
As to why Old Norse affected Old English grammar so strongly while it had a smaller impact (in sheer size, if not importance) on the vocabulary than French, there seem to be two factors:
1. Contact between Old English and Old Norse was much higher than contact between Middle English and Anglo-Norman, since the Viking migration brought in both lords and peasants, and intermarriage was common. Language contact was especially a daily thing around the border area, which happened to be just around London. This turned out to be important, since other English dialects took more time to lose the grammatical endings, but once became the capital, the London dialect became the more important during the 14th and 15th centuries, once the royal court gradually started replacing French with English, since the English of the kings and court was most influenced by the London dialect.
2. The two Germanic languages were mutually intelligible in vocabulary, but their grammatical endings were strikingly different. The easiest way for English and Norse speakers to understand each other was to ignore the grammatical endings, and thus they lost their meaning.