Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Effective context engineering for AI agents

anthropic.com

Post reply on HN
Print view View original

21–30 of 36 posts

Re: Effective context engineering for AI agents

#21

Post by CuriouslyC »

lyu07282 wrote: post #20

I think "output engineering" is equally as important, and steering with grammar (structured output with json schema or CFGs directly) is a huge win there I find: https://platform.openai.com/docs/guides/function-calling#con...

Oh yeah, this is huge! I instruct agents to do a few things in this vein that are big improvements:

1. Have agents emit chatter in a structured format. Have them emit hypotheses, evidence, counterfactuals, invariants, etc. The fully natural language agent chatter is shit for observability, and if you have structured agent output you can actually run script hooks that are very powerful in response to agent input/output.

2. Have agents summarize key evidence from toolcalls, then just drop the tool call output from context (you can give them a tool to retrieve the old value without recomputation, cache tool output in redis and give them a key to retrieve it later if needed). Tool calls dominate context expansion bloat, and once you've extracted evidence the original tool output is very low value.

Re: Effective context engineering for AI agents

#23

Post by ijk »

koakuma-chan wrote: post #9
CuriouslyC wrote: post #7

Earlier quoted context omitted.

It's pretty straightforward, different optimizers have different requirements. Some require example inputs/outputs, others will just optimize on whatever you've got. You can use codex/claude code to set it up in order to bootstrap quickly, they're decent at it.

Does dspy support structured outputs?

Yes, I was using it for structured outputs before the dedicated structured outputs got their act together.

Re: Effective context engineering for AI agents

#24

Post by ivape »

layer8 wrote: post #22
ivape wrote: post #3

I think any meaningful context engineering strategies will be trade secrets.

Why do you think that?

Competitive edge. Some agents will be better than others, therefore worth paying for. So for example, if one writes an AI trading agent, there’s no reason to share it similar to how it is at the moment with regular trading algos.

I’m not saying it won’t eventually be known, but not in these initial stages.

The only thing separating Claude, Gemini and ChatGPT is their context and prompt engineering, assuming the frontier models belong to the same class of capability. You can absolutely release a competitor to these things that could perform better for certain things (or even all things, if you introduce brand new context engineering ideas), if you wanted to.

Re: Effective context engineering for AI agents

#25

Post by ijk »

SOLAR_FIELDS wrote: post #12
saltyoldman wrote: post #6

Earlier quoted context omitted.

Maybe, but we'll be getting to a place where each LLM call gets cheaper, faster and has a larger context, it may not matter long term.

Context is often not the only issue. Really the issue is attention - context is a factor in how well the LLM handles attention to the broad scope of a task, but one can anecdotally easily observe the thing forget or go off the rails when only a fraction of the context window is being used. Oftentimes it’s effective to just say “don’t ever go above 20% of the max”

Some of that is, or at least was, down to the training: extending the context window but not training on sufficiently long data or using weak evaluation metrics caused issues. More recent models have been getting better, though long context performance is still not as good as short context performance, even if the definition of "short context" has been greatly extended.

RoPE is great and all, but doesn't magically give 100% performance over the lengthened context; that takes more work.

Re: Effective context engineering for AI agents

#26

Post by ivape »

lomase wrote: post #4
ivape wrote: post #3

I think any meaningful context engineering strategies will be trade secrets.

Imagine where we would be if academia or open source had this train of tougth. No algorithms, no Linux, no open protocols, maybe not even internet.

Sure, it’s a horrible attitude. With that said, there is a time and place for everything. At the very beginning of AI, which is where we are, it’s not necessarily evil to carve out your advantages and share later.

Re: Effective context engineering for AI agents

#28

Post by elpakal »

I’ve been playing around with Apple’s Foundation Models, their on device llm has a 4k context window. That’s really been an interesting exercise in context engineering coming from others like Claude and GPT. I think those larger context windows have made me take context engineering for granted.

Re: Effective context engineering for AI agents

#29

Post by layer8 »

ivape wrote: post #24
layer8 wrote: post #22

Earlier quoted context omitted.

Why do you think that?

Competitive edge. Some agents will be better than others, therefore worth paying for. So for example, if one writes an AI trading agent, there’s no reason to share it similar to how it is at the moment with regular trading algos. I’m not saying it won’t eventually be known, but not in these initial stages. The only thing separating Claude, Gemini and ChatGPT is their context and prompt engineering, assuming the front…

No, I mean why do you think that effective context engineering will remain a black art, instead of becoming something with standard practices that work well for most use cases?

Re: Effective context engineering for AI agents

#30

Post by ivape »

layer8 wrote: post #29
ivape wrote: post #24

Earlier quoted context omitted.

Competitive edge. Some agents will be better than others, therefore worth paying for. So for example, if one writes an AI trading agent, there’s no reason to share it similar to how it is at the moment with regular trading algos. I’m not saying it won’t eventually be known, but not in these initial stages. The only thing separating Claude, Gemini and ChatGPT is their context and prompt engineering, assuming the front…

No, I mean why do you think that effective context engineering will remain a black art, instead of becoming something with standard practices that work well for most use cases?

I can’t say it will remain a black art because the tech itself creates new paradigms constantly. An LLM can be fine tuned with context engineering examples, similar to Chain Of Thought tuning, and that’s how we get a reasoning loop. With enough fine tuning, we could get a similar context loop, in which case those keeping things hidden will be washed away with new paradigms.

Even if someone fine tuned an LLM with this type of data, Deepseek has shown that they can just use a teacher-student strategy to steal from whatever model you trained (exfiltrate your value-add, which is how they stole from OpenAI). Stealing is already a thing in this space, so don’t be shocked if over time you see a lot more protectionism (protectionism is something we already see geopolitically on the hardware front).

I don’t know what’s going to happen, but I can confidently say that if humans are involved at this stage, there will absolutely be some level of information siloing, and stealing.

——

But to directly answer your question:

”… instead of becoming something with standard practices that work well for most use cases?”

In no uncertain terms, the answer is because of money.

Post reply on HN