I've not the read Greaber's book, though I intend to, but Adam Smith is regularly misinterpreted in an extreme right-wing way (e.g. the Adam Smith Institute) so it's possibly to disagree with that interpretation without disagreeing with Adam Smith.
https://aeon.co/essays/we-should-look-closely-at-what-adam-s...
> The context of Smith’s intervention in The Wealth of Nations was what he called ‘the mercantile system’. By this Smith meant the network of monopolies that characterised the economic affairs of early modern Europe. Under such arrangements, private companies lobbied governments for the right to operate exclusive trade routes, or to be the only importers or exporters of goods, while closed guilds controlled the flow of products and employment within domestic markets.
> As a result, Smith argued, ordinary people were forced to accept inflated prices for shoddy goods, and their employment was at the mercy of cabals of bosses. Smith saw this as a monstrous affront to liberty, and a pernicious restriction on the capacity of each nation to increase its collective wealth. Yet the mercantile system benefited the merchant elites, who had worked hard to keep it in place. Smith pulled no punches in his assessment of the bosses as working against the interests of the public. As he put it in The Wealth of Nations: ‘People of the same trade seldom meet together, even for merriment and diversion but the conversation ends in a conspiracy against the public, or in some contrivance to raise prices.’
> The merchants had spent centuries securing their position of unfair advantage. In particular, they had invented and propagated the doctrine of ‘the balance of trade’, and had succeeded in elevating it into the received wisdom of the age. The basic idea was that each nation’s wealth consisted in the amount of gold that it held. Playing on this idea, the merchants claimed that, in order to get rich, a nation had to export as much, and import as little, as possible, thus maintaining a ‘favourable’ balance. They then presented themselves as servants of the public by offering to run state-backed monopolies that would limit the inflow, and maximise the outflow, of goods, and therefore of gold. But as Smith’s lengthy analysis showed, this was pure hokum: what were needed instead were open trading arrangements, so that productivity could increase generally, and collective wealth would grow for the benefit of all