I used to think this way, particularly because I didn't have much money, so I needed to buy the cheapest computer I could at a given time, resulting in paying more over the lifetime of the computer (which might not be that long).
It was only once I began working with data centers when I realized what a huge amount the power and cooling bills could be. In a datacenter, the power and cooling requirements of a high-end system can easily be more than the cost of buying the system initially. A great example is Western Digital's new "Green" 2TB drives. The drive costs more $/GB initially than two 1TB drives, even when you factor in the cost of housing the drives; but if you look at the power required by the hard drives (and hard drives really do pull a lot of energy) the 2TB drives can come out cheaper. The same analysis applies to SSD drives.
If you want to be smart with your money, and have more in the long run, don't forget to account for ALL the costs you'll incur when compared to another solution. This applies to just about everything, not just computers.