Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Francois Chollet is leaving Google

developers.googleblog.com

Post reply on HN
Print view View original

21–30 of 182 posts

Re: Francois Chollet is leaving Google

#21

Post by paxys »

geor9e wrote: post #19

If I were to speculate, I would guess he quit Google. 2 days ago, his $1+ million Artificial General Intelligence competition ended. Chollet is now judging the submissions and will announce the winners in a few weeks. The timing there can't be a coincidence.

More generally, there is unlimited opportunity in the AI space today, especially for someone of his stature, and staying tied to Google probably isn't as enticing. He can walk into any VC office and raise a hundred million dollars by the end of the day to build whatever he wants.

Re: Francois Chollet is leaving Google

#22

Post by fastball »

trott wrote: post #14
max_ wrote: post #3

I wonder what he will be working on? Maybe he figured out a model that beats ARC-AGI by 85%?

> Maybe he figured out a model that beats ARC-AGI by 85%? People have, I think. One of the published approaches (BARC) uses GPT-4o to generate a lot more training data. The approach is scaling really well so far [1], and whether you expect linear scaling or exponential one [2], the 85% threshold can be reached, using the "transduction" model alone, after generating under 2 million tasks ($20K in OpenAI credits). Perh…

I like the idea of ARC-AGI and think it was worth a shot. But if someone has already hit the human-level threshold, I think the entire idea can be thrown out.

If the ARC-AGI challenge did not actually follow their expected graph[1], I see no reason to believe that any benchmark can be designed in a way where it cannot be gamed. Rather, it seems that the existing SOTA models just weren't well-optimized for that one task.

The only way to measure "AGI" is in however you define the "G". If your model can only do one thing, it is not AGI and doesn't really indicate you are closer, even if you very carefully designed your challenge.

[1] https://static.supernotes.app/ai-benchmarks-2.png

Re: Francois Chollet is leaving Google

#23

Post by paxys »

mistrial9 wrote: post #12
dools wrote: post #10

Earlier quoted context omitted.

FTA "With over two million users, Keras has become a cornerstone of AI development, streamlining complex workflows and democratizing access to cutting-edge technology. It powers numerous applications at Google and across the world, from the Waymo autonomous cars, to your daily YouTube, Netflix, and Spotify recommendations."

sure -- all true in 2018; right about then pyTorch passed TensforFlow in the raw numbers of research papers using it.. grad students later make products and product decisions.. currently, pyTorch is far more popular, the bulk of that is with LLMs source: pyTorch Foundation, news

The existence of a newer, hotter framework doesn't mean all legacy applications in the world instantly switch to it. Quite the opposite in fact.

Re: Francois Chollet is leaving Google

#24

Post by blackeyeblitzar »

modeless wrote: post #9
tadeegan wrote: post #2

I guess they realized muilti-backend keras is futile? I never liked the tf.keras apis and the docs always promosed multi backend but then I guess they were never able to deliver that without breaking keras 3 changes. And even now.... "Keras 3 includes a brand new distribution API, the keras.distribution namespace, currently implemented for the JAX backend (coming soon to the TensorFlow and PyTorch backends)". I don't…

Why would you interpret this as Google disliking Keras? Seems a lot more likely he was poached by Anthropic.

Where did you see that he was poached by Anthropic?

Re: Francois Chollet is leaving Google

#25

Post by TheDudeMan »

fastball wrote: post #22
trott wrote: post #14

Earlier quoted context omitted.

> Maybe he figured out a model that beats ARC-AGI by 85%? People have, I think. One of the published approaches (BARC) uses GPT-4o to generate a lot more training data. The approach is scaling really well so far [1], and whether you expect linear scaling or exponential one [2], the 85% threshold can be reached, using the "transduction" model alone, after generating under 2 million tasks ($20K in OpenAI credits). Perh…

I like the idea of ARC-AGI and think it was worth a shot. But if someone has already hit the human-level threshold, I think the entire idea can be thrown out. If the ARC-AGI challenge did not actually follow their expected graph[1], I see no reason to believe that any benchmark can be designed in a way where it cannot be gamed. Rather, it seems that the existing SOTA models just weren't well-optimized for that one ta…

What you're calling "gamed" could actually be research and progress in general problem solving.

Re: Francois Chollet is leaving Google

#26

Post by nl »

fastball wrote: post #22
trott wrote: post #14

Earlier quoted context omitted.

> Maybe he figured out a model that beats ARC-AGI by 85%? People have, I think. One of the published approaches (BARC) uses GPT-4o to generate a lot more training data. The approach is scaling really well so far [1], and whether you expect linear scaling or exponential one [2], the 85% threshold can be reached, using the "transduction" model alone, after generating under 2 million tasks ($20K in OpenAI credits). Perh…

I like the idea of ARC-AGI and think it was worth a shot. But if someone has already hit the human-level threshold, I think the entire idea can be thrown out. If the ARC-AGI challenge did not actually follow their expected graph[1], I see no reason to believe that any benchmark can be designed in a way where it cannot be gamed. Rather, it seems that the existing SOTA models just weren't well-optimized for that one ta…

> The only way to measure "AGI" is in however you define the "G"

"I" isn't usefully defined either.

At least most people agree on "Artificial"

Re: Francois Chollet is leaving Google

#28

Post by perturbation »

minimaxir wrote: post #20
magicalhippo wrote: post #13

Earlier quoted context omitted.

As someone who hasn't really used either, what's pytorch doing that's so much better?

A few things from personal experience: - LLM support with PyTorch is better (both at a tooling level and CUDA level). Hugging Face transformers does have support for both TensorFlow and PyTorch variants of LLMs but... - Almost all new LLMs are in PyTorch first and may or may not be ported to TensorFlow. This most notably includes embeddings models which are the most important area in my work. - Keras's training loop…

I think a lot of these may have improved since your last experience with Keras. It's pretty easy to override the training loop and/or make custom loss. The below is for overriding training / test step altogether, custom loss is easier by making a new loss function/class.

https://keras.io/examples/keras_recipes/trainer_pattern/

> - Keras's training loop assumes you can fit all the data in memory and that the data is fully preprocessed, which in the world of LLMs and big data is infeasible.

The Tensorflow backend has the excellent tf.data.Dataset API, which allows for out of core data and processing in a streaming way.

Re: Francois Chollet is leaving Google

#30

Post by trott »

fastball wrote: post #22
trott wrote: post #14

Earlier quoted context omitted.

> Maybe he figured out a model that beats ARC-AGI by 85%? People have, I think. One of the published approaches (BARC) uses GPT-4o to generate a lot more training data. The approach is scaling really well so far [1], and whether you expect linear scaling or exponential one [2], the 85% threshold can be reached, using the "transduction" model alone, after generating under 2 million tasks ($20K in OpenAI credits). Perh…

I like the idea of ARC-AGI and think it was worth a shot. But if someone has already hit the human-level threshold, I think the entire idea can be thrown out. If the ARC-AGI challenge did not actually follow their expected graph[1], I see no reason to believe that any benchmark can be designed in a way where it cannot be gamed. Rather, it seems that the existing SOTA models just weren't well-optimized for that one ta…

> But if someone has already hit the human-level threshold

There is some controversy over what the human-level threshold is. A recent and very extensive study measured just 60.2% using Amazon Mechanical Turkers, for the same setup [1].

But the Turkers had no prior experience with the dataset, and were only given 5 tasks each.

Regardless, I believe ARC-AGI should aim for a higher threshold than what average humans achieve, because the ultimate goal of AGI is to supplement or replace high-IQ experts (who tend to do very well on ARC)

---

[1] Table 1 in https://arxiv.org/abs/2409.01374 2-shot Evaluation Set

Post reply on HN