Formalism is one way out of the straight jacket :)

The second way might be 'taste', but of course this is just a triviality (to get it right, simply know what is right!), but of course with experience you can get better at knowing what works -- and, passed the test of formalism, what is relevant at all, or better yet interesting.

The third way is of course some kind of consensus indeed.

In the end, pretty much every scientist (or philosopher, or artist!) hopes for an "eventual consensus" -- where eventually his creations will be appreciated or have some value (unless you're really just making it to yourself; that sounds more like a hobby though?).