My understanding is that justified text is harder to read as in order to create the justification, uneven spaces are generated between the words which disrupts the flow of your eye.
Additionally, just like it's harder for the eye to make out a word in all caps due to greater regularity in the image, it's also likely harder to distinguish differences in a paragraph if it's all a 'block' to the reader.
I'd imagine some people rely more heavily on the shapes of words rather than the letters or spelling itself. And then this heuristic might also carry through into the shape of a paragraph.