The reason that 4x4 matrices are used so much in graphics is their versatility: you can use them to implement modeling (placing an object into the world), viewing (changing a world point into camera coords), projection and clipping (putting a camera-coord point into clipping coords), color transformations (after all, both x,y,z,w and r,g,b,a are coordinates of a 4D point). That's why there's matrix multipliers built into all graphics hardware.
I understand where you're coming from: translating a point is just 3 additions, so why do a full matrix x point multiply? Scaling is just 3 multiplies, and so on. But if you've got a hardware matrix-point multiplier, all those transformations cost the same, so you're not saving time by devoting special code to each kind of transformation.
Moreover, you don't have to learn full linear algebra: to do graphics, you don't have to know how to invert a matrix, nor solve a system of linear equations, nor diagonalize a matrix. Graphics just needs a tiny subset of linear algebra.
When I teach graphics, I describe points, vectors, lines, circles, matrix-point multiplication, and matrix-matrix multiplication. It takes maybe two lectures, and it provides tools for lots of other parts of the graphics pipeline.
Once you have those tools, you can talk about the viewing transformation (just a special matrix), composite transformations (matrix-matrix multiplication), hierarchical objects (more matrix-matrix multiplication), projection (another special matrix), color transformations (another special matrix).
I also find that when describing ray tracing, you need to talk about going from (row col) pixel coordinates to a 3D point in space, where the ray starts. That's partly the INVERSE view transformation, and (yay) it's just a matrix.
When I was a wee lad in 1981 I did indeed try what you suggest: I worked out the algebra of projection, knowing just where the image plane was (using some points and vectors), and where the world point was. This give me an image point. I was very proud to be able to squeeze this into 50 opcodes on a TI-57 calculator. But when I became a man, and put away childish things, I saw that matrices describe the same process much more cleanly and simply. I wish I had known about them much earlier.
Hell, I wish high school math taught some points and vectors, and some simple matrix operations as I described above (not full-blown linear algebra). Then, they could use those tools to make pictures starting from just numerical descriptions of shapes! It would give many kids more enthusiasm for mathematics, and would start them on the way towards operators, groups, linear spaces, and all those things they might see later. All we would have to do is get education departments to teach the math teachers all this stuff. It would be exhilarating for everyone involved.