My work involves designing interfaces and this discussion could help me understand this topic better; but like the author of the article, your post shows some intense frustration towards rollover-only elements, but none made it very evident to me as to why it's so frustrating and why it's so "stupid". We all experience things differently, and I'd love to picture a perspective where it's so frustrating.
Here's how I see it (and as you'll see, I'll be taking various assumptions. Please feel free to tell me I'm wrong (but make a case)):
We like to have everything at hand, but we also like (it's more like a need actually) to focus. Too many elements are a distraction. That's basically why we put things in boxes even if we don't intend to transport them anywhere: out of our sight. Our brains want/need to focus. That would be my answer to your question "What is the affordance of invisibility?".
The example of Google - if that really is a delete contact button, I don't use that app - I would agree is bad design, because its effort to find it outweights the benefits of having it hidden (partly because its placement is not apparently logical and that makes it extra hard to find).
In the first example in the article (github) however, I would tend to think that that specific solution is OK. The author of the article forgot to explain what that button does. A quick look at github, a rollover and a click (three actions) shows me that its function is it edits the description of the repository. How often do you edit the description of a repository? Not that often. Probably very rarely. And the more you use github, the smaller its usage gets proportionally to the rest of the app. The day you need it, you'll find it relatively quickly. Because you've experienced this practice for a while in other sites, and even more so because you've seen github act like that every now and then. Your brain is able to predict that if you don't see it, it might very well be rollover-only. The first logical place you try is the within the repository dashboard, over the description of the repository. Not too hard. I think in this case the benefits of every time I read that page not having that element there and not having to neutralize it myself with my brain by ignoring it vastly outshine the effort of looking for it when I rarely need it. It's a good deal. [1]
I would love to understand a little more about why this concept didn't work out for you.
edit: typos, formatting and deleted some unneeded parts
--
[1] The github element in the case of a touch interface is not great, I would agree, because it's harder to predict that by touching the white area of the description the button will show up. But it works (as in: you can still access it). And you have to cut interface designers some slack (or instead give us the strength we need): we're still trying to figure out what to do exactly with the current mess of having two very different types of clients (mouse and touch) accessing our web interfaces. After we figure out if we should unify, we need to figure out how. Killing rollover interactions might be a necessary casualty in that road (but that would make part of me sad, there are wonderful usages of rollover, see for example the custom sliders in here: http://worrydream.com/LadderOfAbstraction/), either way I'd love to understand a little more, and I assumed that in this discussion we're focusing on rollover, not touch.