The idea that expertise matters is fine.
Where meritocracy gets bad isn't the "merit" part, it's the "-ocracy" bit - the idea that people who have the most merit get to call the shots.
It's problematic in all sorts of ways. First, who gets to define merit? Well, probably the meritocrats. If we assume they're rational agents (which we must, by definition), then they will choose to define merit in the most self-serving manner possible, in order to consolidate their own position, status, and power. And even if we don't assume they're perfectly rational agents (which is more likely, if contradictory to the basic idea behind meritocracy) we should still expect to see self-serving definitions of merit that grow organically out of everyone's bias for their own perspective. Consider the history of racial bias in IQ testing, for example.
Second, by the people who merit the highest in a meritocracy have different backgrounds, and therefore slightly different interests, from everyone who's lumped together in the lumpy part of the bell curve. This is inevitable - in a world where education costs money, you'll have a hard time getting to be the most educated at anything without coming from a position of relatively greater economic privilege than others. Pythagoras was no peasant, nor was Galileio or Newton or Einstein or Feynman. Therefore, we should not assume that they will make decisions in accord with the best interests of society as a whole. And their current circumstances are probably sharply different from most others' too. Probably the person who knows more than anyone else about investment banking is himself (I think we can safely assume it's a he - see below) an investment banker. That by no circumstances means that this person is the most fit to dictate investment banking regulation. It's true that he's got the expertise to draw up a set of rules that will encourage almost any outcome, but the outcome we should all be most worried about is the one where investment bankers get outlandishly wealthy at the expense of everyone else.
Finally, it's impressive how effectively people can use their pre-existing merit to influence others' ability to acquire merit. Often their ability to do so is so well-honed that they can wield it almost constantly without even realizing they're doing so. See: White men in academia, white men in fraternities, white men in software development, etc.