Earlier quoted context omitted.
One of the known things is that babies who get seriously sick somewhere in their first year have a better immune system for life. So yes, failing to be exposed to viral diseases can be bad for you.
Are you sure you aren't falling for survivor bias?
See https://www.cnn.com/2010/HEALTH/12/06/daycare.kids/index.htm... for a random news article reporting on a similar result only with kids getting sick in daycare. Being exposed to diseases early builds immunity later.