Object lifetime is a much more important factor than class size for most server request / response style processing.
Typically there are three lifetimes for objects in server processes. Those that are allocated around startup and are never deallocated; those that are allocated per-request and become garbage once the response goes out; and lifetimes that span multiple requests, like objects in caches.
The first are normally ultra-cheap to "collect": with a generational GC, you simply don't scan them at all, because they haven't changed.
The second group, per-request, are also fairly cheap to collect. Every so often, you GC the youngest generation, and you only need to keep track of references in registers and on the stack. Ideally many requests will have occurred between collections, and the only objects that get kept alive are objects that are in-flight for the current request. And this is why you need at least three generations; you really don't want to have to scan the oldest generation to collect these ephemeral objects after they've built up over a number of youngest generation collections.
It's the third group that kills you. You can save on the cost of scanning the whole heap, using write barriers to track new roots buried in the oldest generation; but that adds accounting costs, and eventually overtakes the cost of a whole heap GC. These guys can also cause the fragmentation you're worried about - they need to be compacted down, copied possibly multiple times. On the CLR, last time I checked, you need a full gen2 GC in order to get rid of them, as they've likely survived a gen1 collection.
With these guys, it's worthwhile doing the big object thing. In fact, it may be worthwhile not having any GC heap storage for them at all, and refer to them using different techniques, like ephemeral keys that look up in Redis, or native pointers stored in statically allocated arrays.
In app servers I've designed, I've never seen GC CPU usage over 5% or so, even with heavy usage of tiny short-lived objects. But you need to care about lifetime.