I think that matrices seem difficult because they are taught by teachers who don't know how they play a role in geometry. Even points and vectors can be hard if the teacher doesn't show how to connect their algebra and their geometry.
Example: a parametric line segment: P(t) = t * A + (1 - t) * B: If t is 0, you get A. If it's 1 you get B. If it's 0.5, you get halfway between A and B. If negative, you're 'before' A. If t > 1, you're 'beyond' B. Then it's easy to see that restricting t > 0 gives you a ray. The geometry matches the algebra nicely.
Example: in ray tracing, you have to test whether a ray intersects a sphere. This gives you a quadratic equation. If the equation has two real roots, the ray hits the sphere twice. If just one real root, it touches it once (one hit point: ray is tangent to sphere). If no real roots, the ray misses (no hit points). I love how the algebra and the geometry correspond so nicely.
Once you tell a student that applying a matrix will change an object's shape, it's easy to get them to see how applying two matrices means applying two shape changes. Then if you apply the changes in opposite order, you get a different matrix product, and a different final shape. Again, the algebra corresponds to the geometry. And you've just taught them about non-commutative multiplication!
And then I show them a translation matrix, and the inverse of the translation matrix (I don't teach them how to invert matrices; why bother them?). And I show them that the inverse makes sense: the translation amounts are the negatives of the original. And I show them that the product of those two matrices is the identity, which again matches the geometric fact: move, then un-move, is the same as doing nothing.
I talk about rotations using my body: I turn around the X axis, then the Y axis. The I repeat it in opposite order, and show that rotations usually don't commute.
What I'm trying to say is that matrices are confusing because teachers SUCK at explaining them. For the small subset of linear algebra that graphics needs, there are a lot of geometric intuitions that make it simple to explain the material.