Live data from Hacker News

The Google employees who created transformers

wired.com

111–120 of 258 posts

Re: The Google employees who created transformers

#111
post #68

And none of them still work for Google. It’s truly baffling that Google’s CEO still has a job after how badly he fumbled on AI.

I think the issue is that there is no future for a trustworthy AI that doesn't completely cannibalize their ad revenue cash cow.

Like, who wants to use an AI that says things like, "... and that's why you should wear sunscreen outside. Speaking of skin protection, you should try Banana Boat's new Ultra 95 SPF sunscreen."

Re: The Google employees who created transformers

#113
post #71

Earlier quoted context omitted.

First generation immigrants are still a tiny minority of the population. The fact that the entire team consists effectively of first generation immigrants says something, probably both about higher education and American culture.

Isn't some of it have to do with it being a self selecting sample? If you come to America to study, you were a good student in your resident country leading to more chances of success than the average local citizen. Their children might be smarter on average. Alternatively if you are coming fleeing persecutions you are enterprising enough to set up something for your children. That hard work inculcates a sort of work…

I think all those are true, but if so the percentages of first-generation immigrants should increase as you ascend the educational pyramid. I believe it does from undergrad to Phd, but not from general population to higher education, so clearly there are at least two very different worlds.

There is a motivation that comes with both trying to make it and being cognizant of the relative opportunity that is absent in the second-generation and beyond.

There are also many advantages given to students outside the majority. When those advantages land not on the disadvantaged but on the advantaged-but-foreign, are they accomplishing their objectives? How bad would higher education have been in Europe? What is the objective, actually?

Re: The Google employees who created transformers

#114

In Google's heyday, around 2014, I was talking with Uszkoreit about a possible role on his then NLP team. I asked "What would you do if you had an unlimited budget?" He simply said, "I do"

Hahaha...

I worked at Borg

The quota system can kick in at whatever time the limits are reached.

And GPUs are scattered across borg cells, limiting the ceiling. That's why XBorg was created so that a global search among all Borg cells for researchers.

And data center Capex is around 5 billion each year.

Google makes hundres of billions of revenue each year.

You are asking what people would do in impossible situation. Like "what you do after you are dead", literally I could do nothing after I am dead.

I cannot even understand what I do stands for in the context of your question. The above is my direct reaction in the line that he assumes he had unlimited budget.

Re: The Google employees who created transformers

#115

Earlier quoted context omitted.

It's 2024; do we really have to point out the fact that publications take liberties with headlines? It's been that way forever. Besides that, are you denying that transformers are the fundamental piece behind all the AI hype today? That's the point of the article. I think the fact that a mainstream publication is writing about the Transformers paper is awesome.

OK, so the title should be changed to "8 Google Employees Invented Modern AI Hype"?

8 Google Employees authored the Transformer paper, which is the common architecture of all LLMs

Re: The Google employees who created transformers

#116

Attention models? Attention existed before those papers. What they did was show that it was enough to predict next word sequences in a certain context. I'm certain they didn't realize what they found. We used this frame work in 2018 and it gave us wildly unusual behavior (but really fun) and we tried to solve it (really looking for HF capability more than RL) but we didn't see what another group found: that scale in…

I’m studying neuroscience but very interested in how ai works. I’ve read up on the old school but phrases like memory graph and energy minimization are new to me. What modern papers/articles would you recommend for folks who want to learn more?

Re: The Google employees who created transformers

#117
post #71

Earlier quoted context omitted.

Why would that be of note, especially in America? I think it would be an interesting observation in China or Japan, or some other country which is generally less welcoming to immigrants than the US

First generation immigrants are still a tiny minority of the population. The fact that the entire team consists effectively of first generation immigrants says something, probably both about higher education and American culture.

I think you have completely the wrong takeaway here...

The US population is around 330 million. The world population is 8.1 billion people. What is that 4%? If you took a random sampling of people around the world, none of them would be Americans. You're going to need a lot more samples to find a trend.

Yet when you turn around and look at success stories, a huge portion of this is going to occur in the US for multiple reasons, especially for the ability to attract intelligence from other parts of the globe and make it wealthy here.

Re: The Google employees who created transformers

#118
post #106
post #68

And none of them still work for Google. It’s truly baffling that Google’s CEO still has a job after how badly he fumbled on AI.

I'm not 100% sure I agree... Google is an ad company at the end of the day. Google is still making obscene amount of money with ads. Currently Gen AI is a massive expense in training and running and is only looking like it may harm future ad revenue (yet to be seen). Meanwhile OpenAI has not surpassed that critical threshold where they dominate the market with a moat and become a trillion dollar company. It is typica…

>it may harm future ad revenue (yet to be seen)

It harms ad revenue at the moment.

For some specific queries (like code-related), I now go to chat.openai.com before google.com, I'm sure I'm not the only one.

AI is the only thing that can put an end to Google's dominance of the web. I also have no idea how Sundar still has a job there.

Re: The Google employees who created transformers

#120

Looking forward to the inevitable movie about how they invented it and the company failed to productize it because of it's culture and management

This is a story that could be written over and over again. Companies with big R&D budgets but not nimble enough any more to take advantage.
Post reply on HN