"What about the factory workers who also practiced prudent money management and did not end up millionaires?"
Then he at least ended up better off than he would have if he had not practiced prudent money management. Anything else would require stretching the definition of "prudent" beyond anything that fits.
It is a fallacy to believe all of life's outcomes are purely due to skill, one that will harm anyone who acts on it. But it is also a fallacy to believe that all of life's outcomes are purely based on luck and privilege, and of the two, this one is far more harmful to anyone who acts on it than the first one.
It is also a fallacy to believe that all possibly skill improvements are equal, a fallacy that helps people over-attribute things to luck when they see "Well, these two people both worked on skills and look at their divergent outcomes!" But if one guy was carefully tuning the skills to things that the market wanted and would lead to success, and the other guy followed, say, a socially-approved educational outcome that produced a very shiny credential, even perhaps in the same field, but did not provide real things that anybody else wanted, you will get divergent outcomes. But that will be less due to luck than someone who assumes that the skills obtained by both were the same would think.