Live data from Hacker News

The new skill in AI is not prompting, it's context engineering

philschmid.de

441–450 of 550 posts

Re: The new skill in AI is not prompting, it's context engineering

#441
post #414

Earlier quoted context omitted.

yes except intelligence isn't like a car, there's no way to break the complicated emergent behaviors of these models into simple abstractions. you can understand a LLM by training one the same amount you can understand a brain by dissection.

I think making one would help you understand that they're not intelligent.

OK I, like the other commenter, also feel stupid to reply to zingers--but here goes.

First of all, I think a lot of the issue here is this sense of baggage over this word intelligence--I guess because believing machines can be intelligent goes against this core belief that people have that humans are special. This isn't meant as a personal attack--I just think it clouds thinking.

Intelligence of an agent is a spectrum, it's not a yes/no. I suspect most people would not balk at me saying that ants and bees exhibits intelligent behavior when they look for food and communicate with one another. We infer this from some of the complexity of their route planning, survival strategies, and ability to adapt to new situations. Now, I assert that those same strategies can not only be learned by modern ML but are indeed often even hard-codable! As I view intelligence as a measure of an agent's behaviors in a system, such a measure should not distinguish the bee and my hard-wired agent. This for me means hard-coded things can be intelligent as they can mimic bees (and with enough code humans).

However, the distribution of behaviors which humans inhabit are prohibitively difficult to code by hand. So we rely on data-driven techniques to search for such distributions in a space which is rich enough to support complexities at the level of the human brain. As such I certainly have no reason to believe, just because I can train one, that it must be less intelligent than humans. On the contrary, I believe in every verifiable domain RL must drive the agent to be the most intelligent (relative to RL award) it can be under the constraints--and often it must become more intelligent than humans in that environment.

Re: The new skill in AI is not prompting, it's context engineering

#442

Earlier quoted context omitted.

Yep, every AI call is essentially just asking it to predict what the next word is after: You are a helpful assistant. Why is the sky blue? Because of Rayleigh scattering. The blue light refracts more. Why is it red at sunset then? And we keep repeating that until the next word is ` `, then extract the bit in between the last assistant tags, and return it. The AI has been trained to look at ` ` differently to ` `, but…

This is why I enjoy calling AI "autocomplete" when people make big claims about it - because that's where it came from and exactly what it is.

AI is not autocomplete. LLMs are autocomplete.

Re: The new skill in AI is not prompting, it's context engineering

#443
post #167

The new skill is programming, same as the old skill. To the extent these things are comprehensible, you understand them by writing programs: programs that train them, programs that run inferenve, programs that analyze their behavior. You get the most out of LLMs by knowing how they work in detail. I had one view of what these things were and how they work, and a bunch of outcomes attached to that. And then I spent a…

Saying the best way to understand LLMs is by building one is like saying the best way to understand compilers is by writing one. Technically true, but most people aren't interested in going that deep.

[deleted]

Re: The new skill in AI is not prompting, it's context engineering

#444
post #428
post #370

Earlier quoted context omitted.

I agree with your point 2. I can't decide if I agree with your point 1 unless you can explain what "reason" means.

I found few definitions. "Reason is the capacity of consciously applying logic by drawing valid conclusions from new or existing information, with the aim of seeking the truth." Wikipedia This Wikipedia definition refers to The Routledge dictionary of philosophy which has a completely different definition: "Reason: A general faculty common to all or nearly all humans... this faculty has seemed to be of two sorts, a f…

heres mine..

0) theory == symbolic representation of a world with associated rules for generating statements

1) understanding the why of anything == building a theory of it

2) intelligence == ability to build theories

3) reasoning == proving or disproving statements using a theory

4) math == theories of abstract worlds

5) science == theories of real world with associated real world actions to test statements

If you use this framework, LLMs are just doing a mimicry of reasoning (from their training set), and a lot of people are falling for that illusion - because, our everyday reasoning jives very well with what the LLM does.

Re: The new skill in AI is not prompting, it's context engineering

#445
post #315

Earlier quoted context omitted.

Engineering tends to mean "the application of scientific and mathematical principles to practical ends". I'm not sure there's much scientific or mathematical about guessing how a non-deterministic system will behave.

The moment you start building evaluation pipelines and running experiments to validate your ideas it stops being guessing

You've heard of science versus pseudo-science? Well..

Engineering: "Will the bridge hold? Yes, here's the analysis, backed by solid science."

Pseudo-engineering: "Will the bridge hold? Probably. I'm not really sure; although I have validated the output of my Rube Goldberg machine, which is supposedly an expert in bridges, and it indicates the bridge will be fine. So we'll go with that."

"prompt engineer" or "context engineer" to me sounds a lot closer to "paranormal investigator" than anything else. Even "software engineer" seems like proper engineering in comparison.

Re: The new skill in AI is not prompting, it's context engineering

#446

Earlier quoted context omitted.

With modern high-quality datasets and the plummeting H100 rental costs it is 100% a feasible undertaking for an individual to train a model with performance far closer to gpt-4-1106-preview than to gpt-2, in fact its difficult to train a model that performs as badly as gpt-2 without carefully selecting for datasets like OpenWebText with the explicit purpose of replicating runs of historical interest: modern datasets…

Ok I'm open (and happy to hear!) to being wrong on this. You are saying I can find tutorials which can train something like gpt3.5 level model (like a 7B model?) from scratch for under 1000 USD of cloud compute? Is there a guide on how to do this?

The literally watch it on a live stream version does in fact start with the GPT-2 arch (but evals way better): https://youtu.be/l8pRSuU81PU

Lambda Labs full metas jacket accelerated interconnect clusters: https://lambda.ai/blog/introducing-lambda-1-click-clusters-a...

FineWeb-2 has versions with Llama-range token counts: https://huggingface.co/datasets/HuggingFaceFW/fineweb-2

Ray Train is one popular choice for going distributed, RunHouse, bumcha stuff (and probably new versions since I last was doing this): https://docs.ray.io/en/latest/train/train.html

tiktokenizer is indispensable for going an intuition about tokenization and it does cl100k: https://tiktokenizer.vercel.app/

Cost comes into it, and doing things more cheaply (e.g. vast.ai) is harder. Doing a phi-2 / phi-3 style pretrain is like I said, more like the resources of a startup.

But in the video Karpathy evals better than gpt-2 overnight for 100 bucks and that will whet anyone's appetite.

If you get bogged down building FlashAttention from source or whatever, b7r6@b7r6.net

Re: The new skill in AI is not prompting, it's context engineering

#447

Earlier quoted context omitted.

The moment you start building evaluation pipelines and running experiments to validate your ideas it stops being guessing

It’s validated and filtered but isn’t it still guessing at the core? Should we call it validated guessing?

[deleted]

Re: The new skill in AI is not prompting, it's context engineering

#448

> Building powerful and reliable AI Agents is becoming less about finding a magic prompt or model updates. Ok, I can buy this > It is about the engineering of context and providing the right information and tools, in the right format, at the right time. when the "right" format and "right" time are essentially, and maybe even necessarily, undefined, then aren't you still reaching for a "magic" solution? If the definit…

Models are Biases.

There is no objective truth. Everything is arbitrary.

There is no such thing as "accurate" or "precise". Instead, we get to work with "consistent" and "exhaustive". Instead of "calculated", we get "decided". Instead of "defined" we get "inferred".

Really, the whole narrative about "AI" needs to be rewritten from scratch. The current canonical narrative is so backwards that it's nearly impossible to have a productive conversation about it.

Re: The new skill in AI is not prompting, it's context engineering

#449
post #416
post #344

Earlier quoted context omitted.

That's not true in practice. Floating point arithmetic is not commutative due to rounding errors, and the parallel operations introduce non-determinisn even at temperature 0.

That's basically a bug though, not an important characteristic of the system. Engineering tradeoff, not math.

It's pretty important when discussing concrete implementations though, just like when using floats as coordinates in a space/astronomy simulator and getting decreasing accuracy as your objects move away from your chosen origin.

Re: The new skill in AI is not prompting, it's context engineering

#450

Earlier quoted context omitted.

I've seen a lot of cases where, if you look at the context you're giving the model and imagine giving it to a human (just not yourself or your coworker, someone who doesn't already know what you're trying to achieve - think mechanical turk), the human would be unlikely to give the output you want. Context is often incomplete, unclear, contradictory, or just contains too much distracting information. Those are all thi…

I don't see the usefulness of drawing a comparison to a human. "Context" in this sense is a technical term with a clear meaning. The anthropomorphization doesn't enlighten our understanding of the LLM in any way. Of course, that comment was just one trivial example, this trope is present in every thread about LLMs. Inevitably, someone trots out a line like "well humans do the same thing" or "humans work the same way"…

I agree with you completely about the trend which has been going on for years. And it's usually used to trivialize the vast expanse between humans and LLMs.

In this case though it's a pretty weird and hard job to create a context dynamically for a task, cobbling together prompts, tool outputs, and other LLM outputs. This is hard enough and weird enough that you can often end up failing to make text that even a human could make sense of to produce the desired output. And there is practical value to taking a context the LLM failed at and checking if you'd expect a human to succeed.

Post reply on HN