Independent contractors where not doing anything different than larger agencies. Same service, just with more attractive pricing.
The difference between a small business and agency is that agency can actually afford to properly avoid paying tax.
The taxes have been changed anyway, so the mythical small salary + dividend, does not offer meaningful advantage any more when it comes to tax.
The real problem is that IR35 removes profit from the company and the company has no way of offsetting legitimate business costs against tax (like your office, accountant, insurance, tools, software etc.) and you also can't retain part of the invoice and taking lower salary and dividend to have a cushion for the times out of clients or when you want to build capital for a start up idea.
It has completely killed entrepreneurship. It means less competition, less jobs and smaller growth of the economy. Not sure how is that a good idea.