Earlier quoted context omitted.
I can think of at least 113 other reasons as well. I'd imagine a fatal crash would have had much more impact on consumer attitudes towards the Concorde than it would towards other, more "conventional" plane models. http://en.wikipedia.org/wiki/Air_France_Flight_4590
It's strange. The Concorde crash was clearly due to a cascade of safety protocol violations, from Continental's runway debris and subsequent skipped runway inspection, to Air France's overweight takeoff, yet the Concorde itself, despite a spotless record, became the target of public outrage.
The real problem was that the Concorde was a 1960's design in 2003 with only 20 units produced. As such it did not continue to receive upgrades and other refinements that other aircraft do. Compare what a 737-100 looked like and what a 737 you'd fly on today looks like. That's just the visual changes, think of the improvements you can't see. Like improved cockpit controls. With only 20 units parts were expensive. Airbus didn't want to support the aircraft either.
I'm sure there was some decrease in passengers after the crash. But I don't think there was a public outrage. With time the numbers would have recovered. They retired the plan not even a full year after restoring service.
As I understand it Air France was selling tickets at less than their cost to operate it. While both BA and AF announced the cancelations at the same time, I wouldn't be surprised if Air France was the group that decided to pull the plug first. Taking out half the planes would have made the maintenance even more expensive for BA.
Finally, Richard Branson tried to buy the BA Concorde fleet and continue operating it. Branson is not an idiot at business, he obviously thought there was a market there. For some reason that's not clear BA refused to sell to him. Which leads to the other possible reason for retirement, the airlines figured out they could make more profit selling subsonic first class than Concorde tickets. Not selling to Branson prevented him from having the supersonic competitive advantage.