Presumably antimatter would be your energy source, but not your propellant. The gamma rays from a small number of annihilations would heat up a much larger amount of normal matter.
At least for the first generation, you likely also wouldn't be using antimatter as the main source of energy, but rather as a method of initiating some other reaction. For example where in a conventional fission reaction you get a relatively clean split of a nucleus into two halves plus a few extra neutrons to drive a chain reaction, an antiproton will blast apart such a nucleus like a billiard break, allowing fission reactions with much less than a conventional critical mass. A quick burst of positrons hitting the surface of some lithium deuteride would be able to replace a fission primary and make a pure-fusion explosion. Either of these options could be used as either incredibly low-mass nukes for an orion drive or as a light weight reactor for a more conventional nuclear propulsion method. While about 600 times less energy dense than pure antimatter, you're still talking 10 million times better energy density than our best current rocket fuels, while using several orders of magnitude less antimatter.