I'm not a lawyer, so take the following with much salt. Furthermore, I'm arguing more from an "ideal legal world" viewpoint rather than the current American legal environment.
A contract must consist of:
1. A "meeting of the minds." This means both sides come to a common understanding of what's in the contract. Basically, it means there's an offer by one side and acceptance by the other without changes.
2. Intention to be legally bound. Basically, this means that the context of a contract matters. It seems to boil down to commercial agreements generally being enforceable while others (e.g. promising money to family) are not.
3. Consideration. This is some sort of exchange of value. In short, a contract cannot be one-way. This is why you sometimes see e.g. things being sold for $1 rather than being just plain given away for free.
The way I see it, there are serious problems for click-through agreements for both 1 and 3. 2 is not a big deal, as despite the name, it doesn't appear to actually apply to the case where you click Agree without the intention to follow the contract.
One problem with #1 is that essentially nobody ever reads these things. In general, "I didn't read the contract I signed" is not an excuse, of course. However, I think this does (or should) change when everybody doesn't read it, because it means that the other party knows, or should know with reasonable certainty, that you're unaware of the contents. Normally you presume that each party is aware of the contents of the contract, but you can't do that with click-through agreements since it's widely-known that they go unread. A contract which I don't read and which you are well aware I didn't read should not be enforceable. For high-value, important contracts like buying a house, a person familiar with the contract will go over each page with you and have you initial it to affirm that you read it, precisely to avoid this problem. The combination of widespread failure to read the contracts and a complete lack of effort to ensure that you, personally read the one in question means that, IMO, there is no meeting of the minds.
Another problem with #1 is that there is either no opportunity to propose changes, or the other side never notices or reviews them. With a normal contract, I can cross out clauses, add or change wording, etc., at which point the other party can accept or reject my changes, or propose further changes. Where this really runs into trouble, I think, is when you edit the contract locally before clicking Agree. This is generally trivial using a DOM inspector for any click-through agreement found on the web. Imagine I take the ToS in question here and alter it to read, "In exchange for mikeash's use of the Google API, Google promises to pay mikeash the sum of ten million dollars." Then I click Agree. Should Google be bound by this agreement? I imagine you immediately and strongly say "No!" But why should I be bound by the original while they are not bound by the alteration? In a normal contractual setting, they have the opportunity to say yes or no to my proposed changes. They will, of course, say no. In this setting, they don't say no! They simply grant me access to their service after I "Agree" to the altered contract. Does this not imply acceptance on their part? If not, what's the difference between their situation and mine?
Finally, #3 seems to completely destroy the concept of any click-through agreement that isn't part of some sort of payment process. Terms of service presented as part of a checkout process would seem to be fine in this respect, but when presented as part of a free web site, it doesn't seem to work. They give you access in exchange for... nothing. Similarly, click-through software licenses seem to fail here because they show up after the money has been exchanged. I pay for a copy of software which I obtain, and only later am I presented with the EULA. Since the money and product already changed hands, there's no further consideration, and so the EULA should not have force.
That's my thinking. I welcome dissenting opinions.