The short answer to your question is that larger engines have very few advantages and many disadvantages.
Specific impulse is the most obvious parameter of a rocket engine, but it's relatively less important for a first stage where thrust/weight is also a very important concern. Specific impulse depends on the chemistry, combustion cycle, combustion efficiency and nozzle design. There isn't any obvious reason why it would scale either way with engine size.
Thrust/weight is very important for engines on a first stage because of gravity losses. Imagine if the rocket sitting on the pad had a thrust/weight of just 1.1; that would mean that at liftoff 91% of the thrust would be wasted just countering gravity. You both want a high thrust/weight at liftoff and a small dry mass to give the second stage the most momentum possible at stage separation. High thrust also helps for reuse in making the burn time of the first stage shorter (for fixed specific impulse and propellant load), which means that at stage separation the first stage isn't too far downrange and is easier to return to the launch site.
Note that there's both the thrust/weight of the rocket as a whole, and of the engines themselves. The engines comprise a significant proportion of the first stage's weight, though, so looking at engine weight (and thrust to weight) is also useful. Consider the RS-25, Merlin 1D, and Raptor. Their specific impulses are 366s, 282s, and 330s (at sea level), so the RS-25 looks pretty good. But their thrust/weight ratios are roughly 60, 185, and 200 (target). The Merlin 1D was, I believe, the highest thrust-weight ratio liquid-fueled engine ever. This is one of the primary reasons, in addition to cost, it is considered so good, despite having such anemic specific impulse.
You can optimize thrust/weight several ways. Firstly, you can try to cram more thrust out of an engine of a fixed size. This is done by driving up chamber pressures as high as you can, which might be easier with a smaller engine due to square-cube scaling. Secondly, you can make the engine smaller, without changing its thrust, and try to cram as many of them in as possible. Consider thrust/area as another important metric -- for fixed rocket dimensions, being able to cram more engines into the base is an easy way to get more thrust and improve the total thrust/weight of the stack.
Another consideration, which has all too often been ignored in the rocketry business, is cost. More specifically $/thrust, if we're looking at a first stage. Here, smaller engines have a clear advantage in that your tooling doesn't need to be as large and expensive, and you're going to need more of them so you can start to leverage economies of scale rather than having each engine being an individual, artisan-produced artifact. That obviously has a limit -- there's a point at which more engines would make things more expensive, but judging from the thrust and cost of the Raptor, I would guess it's right around the optimal point.
Finally, it's worth noting that, all other considerations aside, going bigger in rocket engines tends to make the engineering more difficult. The square-cube law makes heat flux in the combustion chamber scale roughly linearly with engine size, which makes cooling more difficult. And the larger the combustion chamber, the more at risk it is of combustion instabilities. Look, for example, at the Russian RD-170 engine. It looks like four engines but is actually one engine with four combustion chambers. They did that because while it looks more complex, it actually makes things easier.