Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Anthropic's original take home assignment open sourced

github.com

Post reply on HN
Print view View original

181–190 of 394 posts

Re: Anthropic's original take home assignment open sourced

#181

Post by bjackman »

sva_ wrote: post #174
bjackman wrote: post #87

Earlier quoted context omitted.

Lately with Gemini CLI / Jules it doesn't seem like time spent is a good proxy for difficulty. It has a big problem with getting into loops of "I am preparing the response for the user. I am done. I will output the answer. I am confident. Etc etc". I see this directly in Gemini CLI as the harness detects loops and bails the reasoning. But I've also just occasionally seen it take 15m+ to do trivial stuff and I suspect…

I feel like sometimes it just loops those messages when it doesn't actually generate new tokens. But I might be wrong

There are some other failure modes that all feel kinda vaguely related that probably help with building a hypothesis about what's going wrong:

Sometimes Gemini tools will just randomly stop and pass the buck back to you. The last thing will be like "I will read the code to understand " and then it waits for another prompt. So I just type "continue" and it starts work again.

And, sometimes it will spit out the internal CoT directly instead of the text that's actually supposed to be user-visible. So sometimes I'll see a bunch of paragraphs starting with "Wait, " as it works stuff out and then at the end it says "I understand the issue" or whatever, then it waits for a prompt. I type "summarise" and it gives me the bit I actually wanted.

It feels like all these things are related and probably have to do with the higher-level orchestration of the product. Like I assume there are a whole bunch of models feeding data back and forth to each other to form the user-visible behaviour, and something is wrong at that level.

Re: Anthropic's original take home assignment open sourced

#184

Post by [deleted] »

fergie wrote: post #96
lbreakjai wrote: post #70

I consider myself rather smart and good at what I do. It's nice to have a look at problems like these once in a while, to remind myself of how little I know, and how much closer I am to the average than to the top.

I'm 30 years in, and literally don't understand the question.

[deleted]

Re: Anthropic's original take home assignment open sourced

#185

Post by WithinReason »

fergie wrote: post #96
lbreakjai wrote: post #70

I consider myself rather smart and good at what I do. It's nice to have a look at problems like these once in a while, to remind myself of how little I know, and how much closer I am to the average than to the top.

I'm 30 years in, and literally don't understand the question.

After a quick look this is can be seen as a low level GPU/TPU optimization problem where you have to consider the throughput and depth of different arithmetic pipelines. If you want to hire people who understand how to do that you unfortunately have to give them such a convoluted task and emulate the relevant parts of HW. (In reality this is probably more like TPU since it has scalar pipelines, but the optimization methods are not that different)

The task is to parallelize tree traversal, which is embarrassingly unparallel so it's tricky.

Re: Anthropic's original take home assignment open sourced

#186

Post by fny »

ignoramous wrote: post #155
Onavo wrote: post #117

Earlier quoted context omitted.

If you ask an EE to debug react state management code without prior exposure they won't do too well either. But on the other hand they can easily pick up most of it after a week long crash course while training a performance engineer who can optimize code for a specific architecture would take months.

> EE to debug react state management ... easily pick up most of it after a week long crash course while training a performance engineer ... would take months Isn't that mostly because as you go up the abstraction layer, tools and docs to teach yourself the tricks of trade fast are in abundance (let alone a popular layer like React)? Which inturn is likely a function of incentives and opportunities.

It's because the higher up the stack you go, tools become more declarative and literate. Calling sort is far easier than understanding the algorithm for example.
Post reply on HN