All of Kurzweil's predictions are based on extrapolating exponential growth. That's all very well, but exponential growth in physical systems is usually restricted within limits. In a such a system the negative feedback may also be growing exponentially, which means although initially it may be too small to be noticed, after the growth passes some boundary the negative feedback becomes relevant and the overall growth…
"The question is how close are we to the limit and that is something we are only likely to know when we reach it." You've posed the question and then immediately explained why it's fruitless to ask. Even if you are right that there must be limits; if you have no idea when his models are likely to break down then your skepticism is no more solid than his prediction.
But the onus is on the predictor