This is so indicative of Google culture missing the point. The idea of spending $10M training a single model is treated as a casual reality. But “tHaNk GoOdNeSs those generous open source people published their HiGh QuAlItY datasets of ten thousand examples each. Otherwise we’d have no way of creating datasets like that…” :| the sustainable competitive advantage has been and will continue to be HUGE PROPRIETARY DATASETS. (Duh - this is as true for new AI as it was for old AI = ad targeting). It was the _query+click pairs_ that kept Google dominant all these years, not the brilliant engineers. They had all of humanity labeling the entire internet with “when I click on this page/ad for this query I do/don’t search again” a billion+ times a day for a decade. For good measure they’ve also been collecting your email, your calendar, and your browsing habits for nearly as long. The fact that they’ve managed to erase that historic advantage from their collective consciousness (presumably because AI researchers would rather not spend time debugging data labeling UI) is strange to me. It at least deserves a mention in a strategy memo like this. Not vague platitudes about “influence through innovation.” Spend that $10M you were going to spend on a training run as $9.9999M on a private dataset, then the remaining $100 on training. Better still, build products that gets user behavior to train your models for you. Obviously.
We’re going to watch the biggest face plant in recent economic history if they can’t get this one together. I can’t decide if that makes me happy about an overdue changing of the guard in the Valley or sad about the fall of a once great company.
It’s not about the models! Model training is a commodity! It’s about the data! Come on guys.