Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Anthropic's original take home assignment open sourced

github.com

Post reply on HN
Print view View original

301–310 of 394 posts

Re: Anthropic's original take home assignment open sourced

#301

Post by NitpickLawyer »

myahio wrote: post #237
NitpickLawyer wrote: post #32

The writing was on the wall for about half a year (publicly) now. The oAI 2nd place at the atcoder world championship competition was the first one, and I remember it being dismissed at the time. Sakana also got 1st place in another atcoder competition a few weeks ago. Google also released a blog a few months back on gemini 2.5 netting them 1% reduction in training time on real-world tasks by optimising kernels. If t…

Sakana is a grift from what I understand

Eh. I'd call them overly enthusiastic :) I know they publish hype-y stuff, they jumped the gun on a few things, I get that. But their recent result was on a "live" contest, and they did share agent traces, so that's likely a legit result.

Re: Anthropic's original take home assignment open sourced

#302

Post by legel »

bradlys wrote: post #286
game_the0ry wrote: post #284

Earlier quoted context omitted.

How do you know if one candidate happened to see the problem on leetcode and memorized the solution versus one who struggled but figured it out slower?

It doesn’t matter. It’s about looking for candidates who have put in the time for your stupid hazing ritual. It signals on people who are willing to dedicate a lot of time to meaningless endeavors for the sake of employment. This type of individual is more likely to follow orders and work hard - and most importantly - be like the other employees you hired.

Once upon a time, the "stupid hazing ritual" made sense.

Now it means company is stupid.

Re: Anthropic's original take home assignment open sourced

#303

Post by seamossfet »

I'm getting flashbacks from my computer engineering curriculum. Probably the first place I'd start is replacing comparison operators on the ALU with binary arithmetic since it's much faster than branch logic. Next would probably be changing the `step` function from brute iterators on the instructions to something closer to a Btree? Then maybe a sparse set for the memory management if we're going to do a lot of iterations over the flat memory like this.

Re: Anthropic's original take home assignment open sourced

#304

Post by fc417fc802 »

HarHarVeryFunny wrote: post #296
fc417fc802 wrote: post #288

Earlier quoted context omitted.

They do. They provide reference_kernel which shows the algorithm itself, build_mem_image which shows the data format you will be working with, and finally reference_kernel2 which implements said algorithm on said data format. They then provide you with a very naive implementation that runs on their (very simple) VLIW architecture that you are to optimize. If at the end of that someone is still lost I think it is safe…

Well, yes, they have a reference implementation as documentation, just as they have the simulator as documentation for the ISA ... The problem is about pipelining memory loads and ALU operations, so why not just give clear documentatation and state the task rather than "here's a kernel - optimize it"? \_(ツ)_/

Presumably that is only one of two purposes, with the other being to test your ability to efficiently read, understand, and edit low level code that you didn't write. I imagine you'd regularly run into raw PTX if you worked for them in the relevant capacity.

And perhaps a third purpose is to use the simulator to test your ability to reason about hardware that you are only just getting familiar with.

Re: Anthropic's original take home assignment open sourced

#306

Post by measurablefunc »

fc417fc802 wrote: post #280
measurablefunc wrote: post #267

Earlier quoted context omitted.

You're missing the point. There is no evidence to support their claims which means they are more than likely leaking the memory into the LLM prompt & it is cheating by simply loading constants into memory instead of computing anything. This is why formal specifications are used to constrain optimization. Without proof that the code is equivalent you might as well just load constants into memory & claim victory.

> There is no evidence to support their claims Do you make a habit of not presuming even basic competence? You believe that Anthropic left the task running for hours, got a score back, and never bothered to examine the solution? Not even out of curiosity? Also if it was cheating you'd expect the final score to be unbelievably low. Unless you also suppose that the LLM actively attempted to deceive the human reviewers…

This has nothing to do w/ me & consistently making it a personal problem instead of addressing the claims is a common tactic for people who do not know what it means to present evidence for their claims. Anthropic has not provided the necessary evidence for me to conclude that their LLM is not cheating. I have no opinion on their competence b/c that is not what is at issue. They could be incompetent & not notice that their LLM is cheating at their take home exam but I don't care about that.

Re: Anthropic's original take home assignment open sourced

#307

Post by eisbaw »

eisbaw wrote: post #221

I got to 1364 cycles for now, semi-manually: Using design space exploration organized via backlog.md project, and then recombination from that. 20 agents in parallel. Asked to generate drawio for the winner so I can grok it more easily, then I gave feedback. Edit: 1121 cycles

1023 cycles

Re: Anthropic's original take home assignment open sourced

#308

Post by pclmulqdq »

Aurornis wrote: post #238
pclmulqdq wrote: post #26

Earlier quoted context omitted.

I generally have a policy of "over 4 hours and I charge for my time." I did this in the 4-hour window, and it was a lot of fun. Much better than many other take-home assignments.

> I generally have a policy of "over 4 hours and I charge for my time. Worth mentioning that demanding to be paid to apply for a company is usually equivalent to rejecting the job. Most companies are going to end the interview there. Few HR departments would allow one applicant to be paid for the same interview loop as other candidates. I was helping out in a mentoring program during the ZIRP period when the idea of…

Yeah, I have told HR people this and been rejected. I do say this upfront because I don't want to send you a surprise bill. The main response I get is "OK, that's fine, don't spend more than 4 hours on it." The Anthropic recruiter told me, "no problem, it's a 4-hour test anyway."

Re: Anthropic's original take home assignment open sourced

#309

Post by fc417fc802 »

measurablefunc wrote: post #306
fc417fc802 wrote: post #280

Earlier quoted context omitted.

> There is no evidence to support their claims Do you make a habit of not presuming even basic competence? You believe that Anthropic left the task running for hours, got a score back, and never bothered to examine the solution? Not even out of curiosity? Also if it was cheating you'd expect the final score to be unbelievably low. Unless you also suppose that the LLM actively attempted to deceive the human reviewers…

This has nothing to do w/ me & consistently making it a personal problem instead of addressing the claims is a common tactic for people who do not know what it means to present evidence for their claims. Anthropic has not provided the necessary evidence for me to conclude that their LLM is not cheating. I have no opinion on their competence b/c that is not what is at issue. They could be incompetent & not notice that…

You are implying that you believe them to be incompetent since otherwise you would not expect evidence in this instance. They also haven't provided independent verification of their claims - do you suspect them of lying as well?

How do you explain the specific score that was achieved if as you suggest the LLM simply copied the answer directly?

Re: Anthropic's original take home assignment open sourced

#310

Post by ActorNightly »

lbreakjai wrote: post #70

I consider myself rather smart and good at what I do. It's nice to have a look at problems like these once in a while, to remind myself of how little I know, and how much closer I am to the average than to the top.

Yours is a good mentality to have because it creates the emotional drive to learn more, so don't lose that. That being said, this isn't really that complicated. Its just a matter of taking enough time to look at the code and understand how its structured. I feel like the thing that differentiates developer skill is pretty much being able to do that, specifically in the process of having the model of the program in your head.
Post reply on HN