Neither, mostly. Or both. Let me see if I can usefully elucidate.
In the short term, the stress originates in the fact that the pay sucks; even if you work as hard as you can all the time and get all the hours you can handle, you still barely get by in the best of times - and one serious medical issue, for example, is more than enough to wipe you out financially for good.
In the long term, these jobs entail extensive physical labor, and it takes very little time to realize that there's eventually going to be a hard limit on your ability to continue to do such work - especially if you work as hard as you can all the time and get all the hours you can handle. Sooner or later, you're just going to be too old and too beat up to continue, and the longer you do this kind of work, the higher your risk of the kind of injury whose repair costs will wipe you out as I describe above.
And even if you beat all the odds and never get sick or get hurt, what happens when you can't work any more? Funding even a minimally feasible retirement runs to the mid- to high six figures at best, and doing this kind of work all your life, you're lucky to top out at fifty thousand a year - and that's if you can get such work at all, when illegal day labor is so much more cost-effective.
So you're pretty much screwed both ways, when this kind of work is the best your abilities enable you to find. You don't make enough money to provide a stable life for yourself and your family, and when you run up against the physical infirmities that come to us all but most quickly to those who use their bodies hardest, you cease being able to make even that much, and then what the hell do you do?