Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Anthropic's original take home assignment open sourced

github.com

Post reply on HN
Print view View original

201–210 of 394 posts

Re: Anthropic's original take home assignment open sourced

#201

Post by alcasa »

saagarjha wrote: post #137
alcasa wrote: post #115

Earlier quoted context omitted.

No the 2 hours is their time limit for candidates. The thing is that you are allowed to use any non-human help for their take homes (open book), so if AI can solve it in below 2 hours, it's not very good at assessing the human.

4 hours but AI help is (was?) allowed. I assume it was retired because of Opus basically oneshotting it

Fair enough. I feel like designing AI-proof take-homes is getting ever more futile. Given the questions need to be sufficiently low context to be human-doable in a short time and timespans for AI tasks increasing, I'm not sure take homes can actually serve any filtering function whatsoever, besides checking if applicants are willing to put in a minimal amount of effort.

Re: Anthropic's original take home assignment open sourced

#202

Post by ramraj07 »

heavyset_go wrote: post #33
pclmulqdq wrote: post #26

Earlier quoted context omitted.

I generally have a policy of "over 4 hours and I charge for my time." I did this in the 4-hour window, and it was a lot of fun. Much better than many other take-home assignments.

I don't do take home assignments, but when I did, I would offer to do it at my hourly rate, even if it was just an hour. It's time I would otherwise spend making money. Anyone worth working with respected that and I landed several clients who forwent the assignment altogether. It's chump change in the grand scheme of things, and often a formality. Does help that I have a very public web presence and portfolio, though…

I have foregone our take home for exceptional candidates, but let me ask you, do you also demand compensation for in person or zoom call 1-1 interviews? Surely thats the same time of your life.

Re: Anthropic's original take home assignment open sourced

#203

Post by mettamage »

lawrencechen wrote: post #92
languid-photic wrote: post #66

Naively tested a set of agents on this task. Each ran the same spec headlessly in their native harness (one shot). Results: Agent Cycles Time ───────────────────────────────────────────── gpt-5-2 2,124 16m claude-opus-4-5-20251101 4,973 1h 2m gpt-5-1-codex-max-xhigh 5,402 34m gpt-5-codex 5,486 7m gpt-5-1-codex 12,453 8m gpt-5-2-codex 12,905 6m gpt-5-1-codex-mini 17,480 7m claude-sonnet-4-5-20250929 21,054 10m claude-…

codex cli + gpt-5-2-codex-xhigh got to 1606 with the prompt "beat 1487 cycles. go." ~53 minutes.

Wow, is gpt-5-2-codex-xhigh really that good in general? Is this the 200$ per month version?

Re: Anthropic's original take home assignment open sourced

#204

Post by maccard »

forgotpwd16 wrote: post #150
mike_hearn wrote: post #143

Earlier quoted context omitted.

The question isn't clearly written down anywhere, that's why. Presumably actual candidates would have been given more info over the phone or email. Part of the "challenge" is reverse engineering their Python; unclear if that's intentional. If you look at the top of perf_takehome.py then there is a brief comment saying the challenge is to optimize a kernel. Kernel in GPU land means a program that computes on data in p…

This is nice writeup. Thanks. Another commenter said will've taken them 2h just to sketch out ideas; sans LLMs will've taken me more than 2h just to collect all this info let alone start optimizing it.

I've not written a VM before, but the comments in perf_takehome.py and problem.py explain the basics of this.

I gleaned about half of this comment in a few minutes of just skimming the code and reading the comments on the functions and classes. There's only 500 lines of code really (the rest is the benchmark framework).

Re: Anthropic's original take home assignment open sourced

#206

Post by lostmsu »

NitpickLawyer wrote: post #32

The writing was on the wall for about half a year (publicly) now. The oAI 2nd place at the atcoder world championship competition was the first one, and I remember it being dismissed at the time. Sakana also got 1st place in another atcoder competition a few weeks ago. Google also released a blog a few months back on gemini 2.5 netting them 1% reduction in training time on real-world tasks by optimising kernels. If t…

1% doesn't sound like a lot at all.

Re: Anthropic's original take home assignment open sourced

#207

Post by IsTom »

pvalue005 wrote: post #85
bird0861 wrote: post #68

Earlier quoted context omitted.

Which Gemini model did you use? My experience since launch of G3Pro has been that it absolutely sucks dog crap through a coffee straw.

/model: Auto (Gemini 3) Let Gemini CLI decide the best model for the task: gemini-3-pro, gemini-3-flash After ~40 minutes, it got to: The final result is 2799 cycles, a 52x speedup over the baseline. I successfully implemented Register Residency, Loop Unrolling, and optimized Index Updates to achieve this, passing all correctness and baseline speedup tests. While I didn't beat the Opus benchmarks due to the complexit…

Did you check that it did the things it claims it did?

Re: Anthropic's original take home assignment open sourced

#208

Post by b40d-48b2-979e »

mike_hearn wrote: post #143
fergie wrote: post #96

Earlier quoted context omitted.

I'm 30 years in, and literally don't understand the question.

The question isn't clearly written down anywhere, that's why. Presumably actual candidates would have been given more info over the phone or email. Part of the "challenge" is reverse engineering their Python; unclear if that's intentional. If you look at the top of perf_takehome.py then there is a brief comment saying the challenge is to optimize a kernel. Kernel in GPU land means a program that computes on data in p…

    Sounds like a fun exercise :)
I'll be honest, that sounds like the opposite of fun since the worst parts of my job are touching the parts of a Python codebase that are untyped. The sad part is this work codebase isn't even that old, maybe a few years, and the developers definitely should have known better if they had anyone capable leading them. Alas, they're all gone now.

Harder than figuring out the instruction set for some exotic CPU are definitely the giant untyped dicts/lists common in data science code.

Re: Anthropic's original take home assignment open sourced

#210

Post by piokoch »

tayo42 wrote: post #55

I wonder if the Ai is doing anything novel? Or if it's like a brute force search of applying all types of existing optimizations that already exist and have been written about.

How something that generates next token, given a list of previous tokens, can do something novel?
Post reply on HN