It's amazing how much industry changed CS in the last decade. Significantly more than Academia. Resources is part of it of course, but I also think that the majority of best people in CS choose research labs in industry over academia. No surprise here of course - if openai offers you more than $500K a year, why would you go to academia, make peanuts in comparison and worry about politics for your tenure? Then when they get bored at big tech they raise money and start a company.
The Google employees who created transformers
171–180 of 258 posts
Re: The Google employees who created transformers
#172Earlier quoted context omitted.
What is OpenAI today? Can you elaborate? Google is a varied $trillion company. OpenAI sells access to large generative models.
Sure, Google is worth more but they essentially reduced themselves to an ad business. They aren't focused on innovation as much as before. Shareholder value is the new god.
An ad business is all Google ever was. But they made 20 billion in profits last quarter. Why mess with a good thing? Innovator’s Dilemma.
Also, to be fair, OpenAI’s huge valuation is only a promise right now. Hopefully they will eventually turn profitable…
Re: The Google employees who created transformers
#173Earlier quoted context omitted.
Their wildest success at Google would have meant the company's stock price doubling or tripling and their stock grants being worth a couple million dollars at most. Meanwhile all of them had blank checks from top VCs in the valley and the freedom to do whatever they wanted with zero oversight. What exactly could Sundar have done to retain them?
It would have gone much better for Google if the Brain team had been permitted to apply their work, but they were repeatedly blocked on stated grounds of AI safety and worries about impacting existing business lines through negative PR. I think this is probably the biggest missed business opportunity of the past decade, and much of the blame for losing key talent and Google's head start in LLMs ultimately resides wit…
Funnily enough, the same AI safety teams that held Google back from using large transformers in products are also largely responsible for the Gemini image generation debacle.
It is tough to find the right balance though, because AI safety is not something you want to brush off.
Re: The Google employees who created transformers
#174In Google's heyday, around 2014, I was talking with Uszkoreit about a possible role on his then NLP team. I asked "What would you do if you had an unlimited budget?" He simply said, "I do"
Those were fun times! (& great to see you again after all these years). It's astonishing to me how far the tech has come given what we were working on at the time.
Re: The Google employees who created transformers
#175Earlier quoted context omitted.
Conflating illegal and legal immigration is not a useful contribution to this conversation.
The phrasing was "welcoming to immigrants", not "welcoming to the ever shrinking definition of good immigrants established by a bunch of octogenarian plutocrats". "Illegal" is a concept - it's not conflating to assume that it's not the bedrock of the way people think.
Illegal is a status more than it is a concept. Immigrating illegally is not the central definition of immigration, any more than shoplifting is the central definition of customer.
America is much more welcoming of immigration, by which I mean legal immigration, than Japan or China. This is not in dispute.
It is also, in practice, quite a bit more slack about illegal immigration than either of those countries. Although I hope that changes.
Re: The Google employees who created transformers
#176In Google's heyday, around 2014, I was talking with Uszkoreit about a possible role on his then NLP team. I asked "What would you do if you had an unlimited budget?" He simply said, "I do"
Hahaha... I worked at Borg The quota system can kick in at whatever time the limits are reached. And GPUs are scattered across borg cells, limiting the ceiling. That's why XBorg was created so that a global search among all Borg cells for researchers. And data center Capex is around 5 billion each year. Google makes hundres of billions of revenue each year. You are asking what people would do in impossible situation.…
> I cannot even understand what I do stands for in the context of your question
That he had a higher budget than he knew what to do with. When I worked at Google I could bring up thousands of workers doing big tasks for hours without issue whenever I wanted, for me that was the same as being infinite since I never needed more, and that team didn't even have a particularly large budget. I can see a top ML team having enough compute budget to run a task on the entire Google scrape index dataset every day to test things, you don't need that much to do that, I wasn't that far from that.
At that point the issue is no longer budget but time for these projects to run and return a result. Of course that was before LLMs, the models before then weren't that expensive.
Re: The Google employees who created transformers
#177I thought level 4+ self-driving was going to be the face of "modern AI"... instead we got this thing
All modern AI is based on this, including self driving. Check out Waymo's research papers. They even have something called "Wayformer".
Re: The Google employees who created transformers
#178Attention models? Attention existed before those papers. What they did was show that it was enough to predict next word sequences in a certain context. I'm certain they didn't realize what they found. We used this frame work in 2018 and it gave us wildly unusual behavior (but really fun) and we tried to solve it (really looking for HF capability more than RL) but we didn't see what another group found: that scale in…
I’m studying neuroscience but very interested in how ai works. I’ve read up on the old school but phrases like memory graph and energy minimization are new to me. What modern papers/articles would you recommend for folks who want to learn more?
Someone put this link up on another discussion the other day and I found it really fascinating:
I believe energy minimization is literal, just look at the size of that thing and imagine the power bill.
Re: The Google employees who created transformers
#179Attention models? Attention existed before those papers. What they did was show that it was enough to predict next word sequences in a certain context. I'm certain they didn't realize what they found. We used this frame work in 2018 and it gave us wildly unusual behavior (but really fun) and we tried to solve it (really looking for HF capability more than RL) but we didn't see what another group found: that scale in…
I’m studying neuroscience but very interested in how ai works. I’ve read up on the old school but phrases like memory graph and energy minimization are new to me. What modern papers/articles would you recommend for folks who want to learn more?
He is most likely referring to some sort of free energy minimization.
Re: The Google employees who created transformers
#180And none of them still work for Google. It’s truly baffling that Google’s CEO still has a job after how badly he fumbled on AI.
Google doesn't have to try to be the best at AI now, it can just buy whoever is the winner later - a lesson learned from google video.
No Google can't just buy the winner.
Google purchased YouTube for $1.65 billion (and I recall that seemed overpriced at the time).
The way VC works for private companies now is completely different than in 2006.