Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

The new skill in AI is not prompting, it's context engineering

philschmid.de

Post reply on HN
Print view View original

491–500 of 550 posts

Re: The new skill in AI is not prompting, it's context engineering

#491

Post by Davidzheng »

benreesman wrote: post #466
Davidzheng wrote: post #441

Earlier quoted context omitted.

OK I, like the other commenter, also feel stupid to reply to zingers--but here goes. First of all, I think a lot of the issue here is this sense of baggage over this word intelligence--I guess because believing machines can be intelligent goes against this core belief that people have that humans are special. This isn't meant as a personal attack--I just think it clouds thinking. Intelligence of an agent is a spectru…

Eh...kinda. The RL in RLHF is a very different animal than the RL in a Waymo car training pipeline, which is sort of obvious when you see that the former can be done by anyone with some clusters and some talent, and the latter is so hard that even Waymo has a marked preference for operating in July in Chandler AZ: everyone else is in the process of explaining why they didn't really want Level 5 per se anyways: all br…

I'm in agreement--RLHF won't lead to massively more intelligent beings than humans. But I said RL not RLHF

Re: The new skill in AI is not prompting, it's context engineering

#492

Post by Davidzheng »

benreesman wrote: post #446
Davidzheng wrote: post #437

Earlier quoted context omitted.

Ok I'm open (and happy to hear!) to being wrong on this. You are saying I can find tutorials which can train something like gpt3.5 level model (like a 7B model?) from scratch for under 1000 USD of cloud compute? Is there a guide on how to do this?

The literally watch it on a live stream version does in fact start with the GPT-2 arch (but evals way better): https://youtu.be/l8pRSuU81PU Lambda Labs full metas jacket accelerated interconnect clusters: https://lambda.ai/blog/introducing-lambda-1-click-clusters-a ... FineWeb-2 has versions with Llama-range token counts: https://huggingface.co/datasets/HuggingFaceFW/fineweb-2 Ray Train is one popular choice for goin…

Thanks for the links! Hopefully this doesn't come across as confrontational (this is really something I would like to try myself) but I don't think a gpt2 arch will get to close to gpt3.5 level intelligence? I feel like there was some boundary around gpt3.5 where the stuff started to feel slightly magical for me [maybe it was only the RLHF effect]. Do you think models in gpt2 size now are getting to that capability? I know sub 10B models have been getting really smart recently.

Re: The new skill in AI is not prompting, it's context engineering

#493

Post by zelphirkalt »

simonw wrote: post #477
PeterStuer wrote: post #472

Earlier quoted context omitted.

floating point rounding errors are still deterministic. Parallelism dynamics can impact results, but those are not specific to LLM's.

Here's something that isn't deterministic: a = 0.1, b = 0.2, c = 0.3 a * (b * c) = 0.006 (a * b) * c = 0.006000000000000001 If you are running these operations in parallel you can't guarantee which of those orders the operations will complete in. When you're running models on a GPU (or any other architecture that runs a whole bunch of matrix operations in parallel) you can't guarantee the order of the operations.

The order of completion doesn't necessarily influence the overall result of a parallelized computation. This depends on how the results are aggregated. For example for reducing floating point error in calculating a sum of floating point numbers, you could have a sorting step before calculating the sum and then start summing up from the lowest values to the higher ones. Then it doesn't matter at all which of the values is calculated first, because you need them all anyway, to sort them and once they are sorted, the result will always be the same, given same input values.

So you can see, completion time is a completely orthogonal issue, or can be made one.

And even libraries like tensorflow can be made to give reproducible results, when setting the corresponding seeds for the underlying libraries. Have done that myself, speaking from experience in a machine learning setting.

Re: The new skill in AI is not prompting, it's context engineering

#494

Post by v3ss0n »

zacksiri wrote: post #419
v3ss0n wrote: post #334

Earlier quoted context omitted.

At this point , due to non-deterministic nature and hallucination context engineering is pretty much magic. But here are our findings. 1 - LLM Tends to pick up and understand contexts that comes at top 7-12 lines.Mostly first 1k token is best understood by llms ( tested on Claude and several opensource models ) so - most important contexts like parsing rules need to be placed there. 2 - Need to keep context short . W…

Based on my testing the larger the model the better it is at handling larger context. I tested with 8B model, 14B model and 32B model. I wanted it to create structured json, and the context was quite large like 60k tokens. the 8B model failed miserably despite supporting 128k context, the 14b did better the 32B one almost got everything correct. However when jumping to a really large model like grok-3-mini it got it…

That is true too. But I found Qwen3 14B with 8bit quant fair better than 32B with 4b quant . Both kvcache at 8bit. ( i enabled thinking , i will try with /nothink)

Re: The new skill in AI is not prompting, it's context engineering

#495

Post by v3ss0n »

lblume wrote: post #365
v3ss0n wrote: post #334

Earlier quoted context omitted.

At this point , due to non-deterministic nature and hallucination context engineering is pretty much magic. But here are our findings. 1 - LLM Tends to pick up and understand contexts that comes at top 7-12 lines.Mostly first 1k token is best understood by llms ( tested on Claude and several opensource models ) so - most important contexts like parsing rules need to be placed there. 2 - Need to keep context short . W…

I have uploaded entire books to the latest Gemini and had the model reliably accurately answer specific questions requiring knowledge of multiple chapters.

Try multi-turn and agent-to-agent , it will breakdown , but Gemini is a lot better at larger context.

Re: The new skill in AI is not prompting, it's context engineering

#496

Post by v3ss0n »

fwn wrote: post #388
lblume wrote: post #365

Earlier quoted context omitted.

I have uploaded entire books to the latest Gemini and had the model reliably accurately answer specific questions requiring knowledge of multiple chapters.

That’s pretty typical, though not especially reliable. (Allthough in my experience, Gemini currently performs slightly better than ChatGPT for my case.) In one repetitive workflow, for example, I process long email threads, large Markdown tables (which is a format from hell), stakeholder maps, and broader project context, such as roles, mailing lists, and related metadata. I feed all of that into the LLM, which deter…

Gemini does a lot better at long context.

Re: The new skill in AI is not prompting, it's context engineering

#497

Post by bostik »

felipeerias wrote: post #234
JohnMakin wrote: post #10

> Building powerful and reliable AI Agents is becoming less about finding a magic prompt or model updates. Ok, I can buy this > It is about the engineering of context and providing the right information and tools, in the right format, at the right time. when the "right" format and "right" time are essentially, and maybe even necessarily, undefined, then aren't you still reaching for a "magic" solution? If the definit…

If someone asked you about the usages of a particular element in a codebase, you would probably give a more accurate answer if you were able to use a code search tool rather than reading every source file from top to bottom. For that kind of tasks (and there are many of those!), I don't see why you would expect something fundamentally different in the case of LLMs.

In my previous job I repeatedly told people that "git grep is a superpower". Especially in a monorepo, but works well in any big repository, really.

To this day I think the same. With the addition that knowing about "git log -S" grants you necromancy in addition to the regular superpowers. Ability to do rapid code search, and especially code history search, make you look like a wizard without the funny hat.

Re: The new skill in AI is not prompting, it's context engineering

#498

Post by daxfohl »

Seems like there'd be an opportunity for open source tooling here. Context visualized, summarizers, explorers, A/B testers, etc. Also LLM pre-caching of context summaries since IIUC any context change requires full N^2 recalculation of everything so adds a ton of latency and cost. And some optimizers since the previous N is actually (N-M) where M is the the first 0..M context tokens that were unchanged by your update. Though generally you probably want to summarize more of the beginning of the context, so M is likely small in most cases.

Anyway, seems like most of these algorithms are fairly ad hoc things built into all the various agents themselves these days, and not something that exist in their own right. Seems like an opportunity to make this it's own ecosystem, where context tools can be swapped and used independently of the agents that use them, similar to the LLMs themselves.

Re: The new skill in AI is not prompting, it's context engineering

#499

Post by rvz »

CamperBob2 wrote: post #480
rvz wrote: post #440

Earlier quoted context omitted.

> If you don't review the code your C compiler generates now, why not? That isn't a reason why you should NOT review AI-generated code. Even when comparing the two, a C compiler is far more deterministic in the code that it generates than LLMs, which are non-deterministic and unpredictable by design. > Compiler bugs still happen, you know. The whole point is 'verification' which is extremely important in compiler des…

If you can look at what's happening today, and imagine that code will still be generated the same way in 10-15 years as it is today, then your imagination beats mine. 99.9999% of code is not written with compilers that are "formally verified" as immune to code-generation bugs. It's not likely that any code that you and I run every day is.

> 99.9999% of code is not written with compilers that are "formally verified" as immune to code-generation bugs.

Again, that isn't a reason to never check or write tests for your code because an "AI-generated it" or even assuming that an AI will detect all of them.

In fact, it means you NEED to do more reviewing, checking and testing than ever before.

> It's not likely that any code that you and I run every day is.

So millions of phones, cars, control systems, medical devices and planes in use today aren't running formally verified code every day?

Are you sure?

Re: The new skill in AI is not prompting, it's context engineering

#500

Post by mountainriver »

arkmm wrote: post #174
mountainriver wrote: post #162

You can give most of the modern LLMs pretty darn good context and they will still fail. Our company has been deep down this path for over 2 years. The context crowd seems oddly in denial about this

What are some examples where you've provided the LLM enough context that it ought to figure out the problem but it's still failing?

if prompting worked then we would have reliable multi-step agents, the companies that are succeeding like Manus are doing alignment, which is intuitive
Post reply on HN