Viewing profile — mikeknoop

Fun memory trip. Learned assembly on those old Z80s in middle school. I had to go re-dig up SafeGuard, a program I made by reverse engineering TI's TestGuard, to stop admins from w…

One must now ask whether research results are analyzing pure LLMs (eg. gpt-series) or LLM synthesis engines (eg. o-series, r-series). In this case, the headline is summarizing a pa…

I think we agree; to clarify, sharp messaging isn't inaccurate messaging. And I believe the story is not overhyped given the evidence: the benchmark resisted a $1M prize pool for ~…

Correct, fine-tuning is not new. It's long been used to augment foundational LLMs with private data. Eg. private enterprise data. We do this at Zapier, for instance. The new and su…

> I'd heartily recommend maybe taking down the marketing vibrance down a notch and keep things a bit more measured, it's not entirely a meme, though some of the more-serious resear…

Author here -- six months ago we launched ARC Prize, a huge $1M experiment, to test if we need new ideas for AGI. The ARC-AGI benchmark remains unbeaten and I think we can now defi…

Context: ARC Prize 2024 just wrapped up yesterday. ARC Prize's goal is to be a north star towards AGI. The two major categories of this year's progress seem to fall into "program s…

I met my Zapier co-founder bryanh through HN 15 years ago when someone made a similar service to OP called "hacker newsers". We were the only two people in Missouri at the time whi…

I personally am slightly surprised at o1's modest performance on ARC-AGI given the large leaps in performance on other objectively hard benchmarks like IOI and AIME. Curiosity is t…

I bet pretty well! Someone should try this. It's likely expensive but sampling could give you confidence to keep going. Ryan's approach costs about $10k to run the full 400 public …

Author here. Which aspects are misleading? How can it be improved?

High efficiency "search" is necessary to reach AGI. For example, humans don't search millions of potentially answers to beat ARC Prize puzzles. Instead, humans use our core experie…

ARC isn't perfect and I hope ARC is not the last AGI benchmark. I've spoken with a few other benchmark creators looking to emulate ARC's novelty in other domains, so I think we'll …

(ARC Prize co-founder here). Ryan's work is legitimately interesting and novel "LLM reasoning" research! The core idea: > get GPT-4o to generate around 8,000 python programs which …

Yes there is a secondary leaderboard called ARC-AGI-Pub (in beta) with no limitations: https://arcprize.org/leaderboard

(You can direct link to a task like this: https://arcprize.org/play?task=009d5c81 in case you want to share!)

Here is some published research on the human difficulty of ARC-AGI: https://cims.nyu.edu/~brenden/papers/JohnsonEtAl2021CogSci.p... > We found that humans were able to infer the un…

That is correct for ARC Prize: limited Kaggle compute (to target efficiency) and no internet (to reduce cheating). We are also trialing a secondary leaderboard called ARC-AGI-Pub t…

I agree, $1M is ~trivial in AI. The primary goal with the prize is to raise public awareness about how close (or far today) we are from AGI: https://arcprize.org/leaderboard and we…

Hey folks! Mike here. Francois Chollet and I are launching ARC Prize, a public competition to beat and open-source the solution to the ARC-AGI eval. ARC-AGI is (to our knowledge) t…

(Zapier co-founder) Perhaps the least known feature of Zapier is that you can use the dev platform ( https://zapier.com/developer ) to make your own private apps for any API (inclu…

Couldn't find your contact info. Email me?

Very cool. Congrats on getting this launched @gogwilt!