Earlier quoted context omitted.
CRLs can be sharded, the cert carries the URL for the relevent CRL inside it. So they wouldn't need to have grown as huge as you suggest. But, this sharing carries a cost for user privacy, if I shard certs 16 ways then each CRL download gives me 4 bits of info about which sites you were visiting. OCSP effectively takes this to the extreme, each lookup is tiny because it's just for one cert, but it gives away exactly…
Besides leaking data by on demand CLR checking, you also have a difficult fail open v fail closed decision. Failing closed means failure of a third party immediately breaks your site. Failing open means a MitM can simply block the CRL check. OCSP stapling and the 'must staple' header are a lot better for privacy, and OCSP responses have some validity so at least a 5 hour outage of your CA doesn't bring your site down…
Apache and nginx both shipped OCSP stapling implementations that are very bad, awful enough that for almost anyone I'd say "No, don't enable that" rather than try to explain how they need to use it and get them to a place where it's useful and safe. Adam Langley wrote years ago about how to do this correctly, and there does seem to be a little bit of movement in the correct direction at Apache, but the situation remains pretty poor.