Here's an argument:
If you think about it, "mass" actually means two different things. One one hand, there is mass as the "charge" of gravitational interaction, that is, the gravitational field generated by a body is proportional to its mass. Let's call this concept "gravitational mass". On the other hand, you have mass in the sense of "intertial mass", a body's resistance to change of velocity. Although we kinda take it for granted, it's weird that these two concepts coincide, and not something that would be necessarily true a priori.
Asserting this equivalence as a principle is the starting point for Einstein's theory of General Relativity, which is one of the most beautiful and thoroughly tested theories in modern physics.
On the other hand, you can show that the inertial mass cannot be negative (else the Lagrangian would not be bounded from below and therefore you could not minimise it; the whole of physics then comes crumbling down). Therefore, if that equivalence stands, the gravitational mass also cannot be negative.