Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

AGENTS.md outperforms skills in our agent evals

vercel.com

Post reply on HN
Print view View original

131–140 of 212 posts

Re: AGENTS.md outperforms skills in our agent evals

#131

Post by postalcoder »

Izkata wrote: post #127
postalcoder wrote: post #110

Earlier quoted context omitted.

That's not the only useful takeaway. I found this to be true: > "Explore project first, then invoke skill" [produces better results than] "You MUST invoke the skill". I recently tried to get Antigravity to consistently adhere to my AGENTS.md (Antigravity uses GEMINI.md). The agent consistently ignored instructions in GEMINI.md like: - "You must follow the rules in [..]/AGENTS.md" - "Always refer to your instructions…

An idea: The first two are obviously written as second-person commands, but the third is ambiguous and could be interpreted as a first-person thought. Have you tried the first two without the "you must" and "your", to also change them to sort-of first-person in the same way?

Solid intuition. Testing this on antigravity is a chore because I'm not sure if I have to kill the background agent to force a refresh of the GEMINI.md file so I just did it anyway.

  +------------------+------------------------------------------------------+
  | Success/Attempts | Instructions                                         |
  +------------------+------------------------------------------------------+
  | 0/3              | Follow the instructions in AGENTS.md.                |
  +------------------+------------------------------------------------------+
  | 3/3              | I will follow the instructions in AGENTS.md.         |
  +------------------+------------------------------------------------------+
  | 3/3              | I will check for the presence of AGENTS.md files in  |
  |                  | the project workspace. I will read AGENTS.md and     |
  |                  | adhere to its rules.                                 |
  +------------------+------------------------------------------------------+
  | 2/3              | Check for the presence of AGENTS.md files in the     |
  |                  | project workspace. Read AGENTS.md and adhere to its  |
  |                  | rules.                                               |
  +------------------+------------------------------------------------------+
In this limited test, seems like the first person makes a difference.

Re: AGENTS.md outperforms skills in our agent evals

#133

Post by BiteCode_dev »

tottenhm wrote: post #5

> In 56% of eval cases, the skill was never invoked. The agent had access to the documentation but didn't use it. The agent passes the Turing test...

You got me good with this one.

But seriously, this is my main answer to people telling me AI is not reliable: "guess what, most humans are not either, but at least I can tell AI to correct course and it's ego won't get in the way of fixing the problem".

In fact, while AI is not nearly as a good as a senior dev for non trivial tasks yet, it is definitely more reliable than most junior devs at following instructions.

Re: AGENTS.md outperforms skills in our agent evals

#135

Post by DanOpcode »

motoboi wrote: post #121

Models are not AGI. They are text generators forced to generate text in a way useful to trigger a harness that will produce effects, like editing files or calling tools. So the model won’t “understand” that you have a skill and use it. The generation of the text that would trigger the skill usage is made via Reinforcement Learning with human generated examples and usage traces. So why don’t the model use skills all t…

What's RL?

Re: AGENTS.md outperforms skills in our agent evals

#137

Post by jacobkg »

DanOpcode wrote: post #135
motoboi wrote: post #121

Models are not AGI. They are text generators forced to generate text in a way useful to trigger a harness that will produce effects, like editing files or calling tools. So the model won’t “understand” that you have a skill and use it. The generation of the text that would trigger the skill usage is made via Reinforcement Learning with human generated examples and usage traces. So why don’t the model use skills all t…

What's RL?

Reinforcement Learning https://en.wikipedia.org/wiki/Reinforcement_learning

Re: AGENTS.md outperforms skills in our agent evals

#138

Post by wahnfrieden »

DanOpcode wrote: post #135
motoboi wrote: post #121

Models are not AGI. They are text generators forced to generate text in a way useful to trigger a harness that will produce effects, like editing files or calling tools. So the model won’t “understand” that you have a skill and use it. The generation of the text that would trigger the skill usage is made via Reinforcement Learning with human generated examples and usage traces. So why don’t the model use skills all t…

What's RL?

Reinforcement learning

Re: AGENTS.md outperforms skills in our agent evals

#139

Post by alchemism »

devonkelley wrote: post #123
deaux wrote: post #122

Earlier quoted context omitted.

Just happen to post 2 comments within 30s on completely different posts, having all of the hallmarks of LLM output? With your other post being full of green accounts? With no account activity for 8 years? You're clearly posting comments straight from an LLM. It's not realistic to read the other post to a significant degree, think about it, and then type all of this: > The prompt injection concerns are valid, but I th…

I made an account years ago, never posted, and decided I want to be more active in the community. Green accounts probably bc I sent my post to some friends and users directly when I made it. Is that illegal on HN? I legit don't know how things work here. I was excited over my launch post. Anyways, not a fucking bot, my company is real, the commenters on my post are real and if it's a crime for me to rapid fire post a…

When was the last time you passed a Voight-Kampff test, friend?

Re: AGENTS.md outperforms skills in our agent evals

#140

Post by bonesss »

BiteCode_dev wrote: post #133
tottenhm wrote: post #5

> In 56% of eval cases, the skill was never invoked. The agent had access to the documentation but didn't use it. The agent passes the Turing test...

You got me good with this one. But seriously, this is my main answer to people telling me AI is not reliable: "guess what, most humans are not either, but at least I can tell AI to correct course and it's ego won't get in the way of fixing the problem". In fact, while AI is not nearly as a good as a senior dev for non trivial tasks yet, it is definitely more reliable than most junior devs at following instructions.

Key differences, though:

Humans are reliably unreliable. Some are lazy, some sloppy, some obtuse, some all at once. As a tech lead you can learn their strengths and weaknesses. LLMs vacillate wildly while maintaining sycophancy and arrogance.

Human egos make them unlikely to admit error, sometimes, but that fragile ego also gives them shame and a vision of glory. An egotistical programmer won’t deliver flat garbage for fear of being exposed as inferior, and can be cajoled towards reasonable output with reward structures and clear political rails. LLMs fail hilariously and shamelessly in indiscriminate fashion. They don’t care, and will happily argue both sides of anything.

Also that thing that LLMs don’t actually learn. You can threaten to chop their fingers off if they do something again… they don’t have fingers, they don’t recall, and can’t actually tell if they did the thing. “I’m not lying, oops I am, no I’m not, oops I am… lemme delete the home directory and see if that helps…”

If we’re going to make an analogy to a human, LLMs reliably act like absolute psychopaths with constant disassociation. They lie, lie about lying, and lie about following instructions.

I agree LLMs better than your average junior first time following first directives. I’m far less convinced about that story over time, as the dialog develops more effective juniors over time.

Post reply on HN