The main issue is that currently there's no way around publishing at the established journals and conferences, because of the reputation they've built up. Funding and career advancement hinge on publications in these venues. If we accept that publishers don't offer much at all in return for the publication fees/library subscription costs (barely any editing, reviewers work pro bono, hosting PDFs is very cheap nowadays), the main issue is one of "reputation transfer".
One way to cut out the middle man would be to convince journal editors to run sibling journals alongside the existing journals, so for each "Transactions on XYZ" there would be an "Open Transactions on XYZ" (as close in title as is legal). Importantly, each sibling journal would be run by the exact same academics (who are doing the real work on tax money anyway), and according to the same process as the original journal, just without involvement from a traditional publisher. PDFs would be hosted on a site like arXiv. The goal would be that submitting to the open "sibling" would be the obvious rational choice (same people, same decision-making, no fees, open access), which in time even the funding agencies and tenure committees would have to acknowledge.