First, publishing costs money, even in the digital age. It costs money to comb through submissions and decide which ones are worth pursuing. It costs money to hassle scientists into reviewing the submissions. It costs money to convert every submission into the same format. It costs money to develop and host a website to disseminate the articles. All of these things cost money.
Now, who is going to pay for it? Traditionally these costs were put onto the research institutions in the form of library subscription fees. Open access shifts this burden onto the author, and ideally grants would include that into the budget.
Even if grants include that in their budget (and many don't yet), there's a finite amount of money available for research. Shifting the cost of publishing onto grants will make funding available for actual research even smaller than it is now. In some fields publishing costs are entirely negligible compared to the cost of research, but in others it's not.
Also, open access would mean that you have to have funding in order to publish a paper. As it stands right now you don't actually need funding to do research in certain fields. A math professor at a university can devote some of his spare time to a project over several years and publish a paper on it with no costs at all. This happens all the time, not every paper has funding behind it.
I'm not necessarily arguing against open access, I just think people haven't fully explored the downsides of moving away from our current system.