In some sense your eye might not care, but it really do seem that either eye or the brain cares. Not necessarily about the background or foreground itself unless the background is really bright, but something more like the relative contrast of our entire field of vision, and interacting with the size of elements.
A good black and white (true grayscale CRT display) in a properly lit room can be eminently readable and cause very little eye strain. However, as soon as you start to use color and images, and/or the room isn't properly lit, it seems the level of contrast necessary for make colour look right - especially in images - also tend to make white backgrounds far too bright. According to my experience the effect get worse the bigger the screen is.
It does seem to be an issue, but not really a problem with the text versus background color, rather because we handle gamut/contrast/color the same way in both UI elements, on screen text, and images.
While this might partially be down to the last decades dominance of LCD screens with rather poor color rendering that nobody has cared to fix it, the color profiles of UI, text, and images should really be adjusted separately.
So why does dark background seemingly alleviate this issue for some fraction of people? I don't know for sure, it could be the brain "averages" light over a small area, when determining some global brightness/contrast measure, thus making might on black seem less contrasty while still allowing use of brightness/color range that also works reasonably well for images?
I notice my daughter brought up with screens around seem to care less, but that could be a coincidence.