Big Tech obviously has a massive moat of human interaction data. But as a founder looking at the unit economics of training, I keep wondering why we are so obsessed with human data given how inconsistent, biased, and noisy it is.
In theory, a startup could bypass that bottleneck by generating synthetic datasets that are actually better: internally consistent, fully labeled, and optimized for specific reasoning patterns. If we can define the rules of logic, math, and syntax, why not generate "ideal" training data at scale and avoid the long tail of human errors entirely?
For those who have actually tried this strategy, what breaks? Does it collapse diversity? Do you overfit to the hidden biases of the generator model? Or is "human messiness" actually a feature required for robustness?
Curious to hear from founders or engineers who have experimented with 100% synthetic pipelines—is the "Human Data Wall" real, or just a lack of better filtering?