Sure, there's lots of trial and error. But consider something like Universal Sentence Encoders (USE) versus facebooks InferSent. USE is superior, mostly because its new, but still under performs infersent in a few areas. This is the kind of thing where actually building a more specialized model for question answering or textual similarity could see huge performance boosts for companies, but nobody is doing it. If they are, its under lock and key. Anyone looking to perform these tasks is mostly just pulling the code, tweaking the data, messing with the heads of the networks, and then calling it a day.
EDIT:
copying and pasting this from another answer:
I think its way different than that, those would just be precursors, and in cases like real analysis, superfluous. Instead it would look something like, I have a Universal Sentence Encoder architecture, but its not performing well on my data, aside from tweaking the training set, how can I take this architecture and change it to work better with my individual problem? The number of people on the planet that can do this successfully, without wasting months of time messing around with tensorflow is extremely small. But this is where the value is. These massive catch all models only work for the people creating them, just jamming them into any NLP model will always produce sub par and probably unusable results