How could it be consensual between all adults? The marriage "license" in polygamy is between man and women, yet by secondary effects the women are fully entangled with one another. Yet if you watch any show on polygamy you'll see that a guy will keep adding new, younger wives while holding their current wives hostage (due to their lack of another option in terms of housing, money, etc). It's absurd to think that any thinking person would get in on the wife side of the polygamy pact when in the end they have their equality divided by X wives.
Smart folks have said that the reason homosexual marriage even became a thought was because of the transformation of marriage from a bond of ownership of man over woman to a shared partnership. A polygamous arrangement (as per the mormon church, not the polyamorous relationships of the hippies, etc) ends up with a hierarchy of power with one man controlling multiple wives, with the power to divorce, ruin, etc the others if they don't allow him to wed again.
I think an argument for polyamorous marriage could be made, but polygamy is essentially a raw deal for whoever isn't the hub of the marriage wheel (one man in the case of traditional mormonism, islam, etc.).