> I've always held that the numbers themselves are perfectly precise. It's the operations that don't do what you expect.
I think that this is an oversimplification, and misses a key point.
Floating point numbers can be interpreted in (at least) two distinct ways.
In one use model, floating point numbers allow a sparse representation of selected points on the (extended) real number line. In this model, the numbers themselves are perfectly precise, and the basic operations do exactly what you'd expect -- but because the result of the expected real-number operation may not be one of the selected representable points, each operation is potentially paired with a second, implicit, rounding operation, and the consequences of the cumulative effects of this rounding operation take some thought.
This first model is often comfortable for those who have done low-level numerical program with sized integers, which are likewise a selected sparse representation of selected points on the integer line. While rounding comes up in fewer operations here, and is simpler, wrap-around becomes an issue.
However, a second model of floating point numbers is that each possible bit sequence represents an /interval/ on the (extended) real number line. In versions of this model, while the set of representable intervals is still very restricted, every point on the real number line falls into a representable interval, which is a property that sometimes eases (and sometimes seems to ease) analysis. In exchange, the operations are much more complex, and their limitations are much more obvious -- in particular, not only can no disjoint intervals be represented, but for any given point on the real line only one interval that contains it is representable -- there's no way to represent intervals of different sizes containing (some of) the same points.
This second model is what people are mostly thinking of when they say that floating point arithmetic is "not precise." In particular, both the inputs and outputs of operations have imprecise interpretations -- that is, the input interval contains infinitely many indistinguishable points, and the output interval contains infinitely many indistinguishable points, even if a real number equivalent of the problem would have a single number in the desired range.
When extended to true interval arithmetic, the second model can be very useful for understanding error propagation and uncertainty, but sadly this is rarely done.