Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Context engineering

chrisloy.dev

Post reply on HN
Print view View original

41–50 of 70 posts

Re: Context engineering

#41

Post by dingnuts »

cadamsdotcom wrote: post #23
simonw wrote: post #18

Earlier quoted context omitted.

Yes, and we've also decided that they deserve the title "engineering" more than software engineering does. Most engineering disciplines have to deal with tolerances and uncertainty - the real world is non-deterministic. Software engineering is easy in comparison because computers always do exactly what you tell them to do. The ways LLMs fail (and the techniques you have to use to account for that) have more in common…

Yep. Consider woodworking - the wood you use might warp over time, or maybe part of it ends up in the sun or the thing you’ll make gets partly exposed to water. Can you make a thing that’ll serve its purpose and look good for years under those constraints? A professional carpenter can. We have it easy in software.

Woodworking is to civil engineering as being an IT help desk rep is to being a software engineer. Woodworking isn't engineering either. If you build a system with aspects you can measure and predictably tune, you're engineering. If you're making skilled alterations to an existing structure or system without applied math or science, you're partaking in a craft.

Software engineering blurs the lines, sure, but woodworking isn't engineering ever.

Re: Context engineering

#42

Post by Scipio_Afri »

aeve890 wrote: post #38
andai wrote: post #35

Earlier quoted context omitted.

>just set temp to 0 to make LLMs deterministic Does that really work? And is it affected by the almost continuous silent model updates? And gpt-5 has a "hidden" system prompt, even thru the API, which seemed to undergo several changes since launch...

Strictly speaking, it should work. We don't have a _real_ RNG yet and with the same seed any random function becomes deterministic. But behind the blackbox of LLM providers who know what's tunned processing your request. But my point stands. The non-deterministic nature of LLMs are implementation details, not even close to physical constraints as the parent comment suggest.

There is inherent non-determinism in all machine learning models unless you explicitly configure pytorch or other frameworks to do determinism (https://docs.pytorch.org/docs/stable/notes/randomness.html). However, this is very unlikely to be done in models that are being run in production due to performance and other issues.

Re: Context engineering

#43

Post by skylurk »

dingnuts wrote: post #40
simonw wrote: post #18

Earlier quoted context omitted.

Yes, and we've also decided that they deserve the title "engineering" more than software engineering does. Most engineering disciplines have to deal with tolerances and uncertainty - the real world is non-deterministic. Software engineering is easy in comparison because computers always do exactly what you tell them to do. The ways LLMs fail (and the techniques you have to use to account for that) have more in common…

The tools mechanical and civil engineers use are predictable. You're confusing the things these engineers design, which have tolerances and things like that, with the tools themselves. If an engineer built an internal combustion engine that misfired 60% of the time, it simply wouldn't work. If an engineer measured things with a ruler that only measured correctly 40% of the time, that would be the apt analogy. The too…

Civil engineers deal with contractors who would misfire 100% of the time if they could get away with it.

Re: Context engineering

#44

Post by andai »

simonw wrote: post #37
andai wrote: post #35

Earlier quoted context omitted.

>just set temp to 0 to make LLMs deterministic Does that really work? And is it affected by the almost continuous silent model updates? And gpt-5 has a "hidden" system prompt, even thru the API, which seemed to undergo several changes since launch...

It famously does not: https://thinkingmachines.ai/blog/defeating-nondeterminism-in...

>But why aren’t LLM inference engines deterministic? One common hypothesis is that some combination of floating-point non-associativity and concurrent execution leads to nondeterminism based on which concurrent core finishes first. We will call this the “concurrency + floating point” hypothesis for LLM inference nondeterminism.

Dang, so we don't even know why it's not deterministic, or how to make it so? That's quite surprising! So if I'm reading this right, it doesn't just have to do with LLM providers cutting costs or making changes or whatever. You can't even get determinism locally. That's wild.

But I did read something just the other day about LLMs being invertible. It goes over my head but it sounds like they got a pretty reliable mapping from inputs to outputs, at least?

https://news.ycombinator.com/item?id=45758093

> Transformer components such as non-linear activations and normalization are inherently non-injective, suggesting that different inputs could map to the same output and prevent exact recovery of the input from a model's representations. In this paper, we challenge this view. First, we prove mathematically that transformer language models mapping discrete input sequences to their corresponding sequence of continuous representations are injective and therefore lossless, a property established at initialization and preserved during training. Second, we confirm this result empirically through billions of collision tests on six state-of-the-art language models, and observe no collisions.

The distinction here appears to be between the output tokens versus some sort of internal state?

Re: Context engineering

#45

Post by dwaltrip »

sgt101 wrote: post #9

Why would I believe that any of this works? This is just some blokes idea of what people should do. There is no evidence offered. No attempt to measure the benefits.

Imagine the gall of someone who just goes on the internet and writes something.

Re: Context engineering

#46

Post by calebkaiser »

timr wrote: post #25
calebkaiser wrote: post #14

Earlier quoted context omitted.

Based on the comments, I expected this to be slop listing a bunch of random prompt snippets from the author's personal collection. I'm honestly a bit confused at the negativity here. The article is incredibly benign and reasonable. Maybe a bit surface level and not incredibly in depth, but at a glance, it gives fair and generally accurate summaries of the actual mechanisms behind inference. The examples it gives for…

There’s nothing particularly wrong with the article - it’s a superficial summary of stuff that has historically happened in the world of LLM context windows. The problem is - and it’s a problem common to AI right now - you can’t generalize anything from it. The next thing that drives LLMs forward could be an extension of what you read about here, or it could be a totally random other thing. There are a million monkey…

I don't really understand this line of criticism, in this context.

What would "generalizing" the information in this article mean? I think the author does a good job of contextualizing most of the techniques under the general umbrella of in-context learning. What would it mean to generalize further beyond that?

Re: Context engineering

#47

Post by calebkaiser »

amonks wrote: post #31
calebkaiser wrote: post #19

Earlier quoted context omitted.

I think it's fair to question the use of the term "engineering" throughout a lot of the software industry. But to be fair to the author, his focus in the piece is on design patterns that require what we'd commonly call software engineering to implement. For example, his first listed design pattern is RAG. To implement such a system from scratch, you'd need to construct a data layer (commonly a vector database), retri…

long shot, apropos of nothing, just recognized your name: If you are the cincinnatian poet Caleb Kaiser, we went to college together and I’d love to catch up. Email in profile. If you aren’t, disregard this. Sorry to derail the thread.

Hello friend!

Re: Context engineering

#48

Post by bdangubic »

timr wrote: post #26
mpalmer wrote: post #24

Earlier quoted context omitted.

Physical engineers might scoff good-naturedly at an attempt by project managers to refer to work scheduling as "logistics engineering". But they really shouldn't because obviously scheduling and logistics is difficult, involving a lot of uncertainty and tolerances.

Uncertainty and tolerance implies that you have a predictable distribution in the first place. Engineers are not just dealing with a world of total chaos, observing the output of the chaos, and cargo culting incantations that seem to work for right now [1]…oh wait nevermind we’re doing a different thing today! Have you tried paying for a different tool, because all of the real engineers are using Qwghlm v5 Dystopic n…

exactly why calling this engineering is downright criminal

Re: Context engineering

#49

Post by simonw »

dingnuts wrote: post #40
simonw wrote: post #18

Earlier quoted context omitted.

Yes, and we've also decided that they deserve the title "engineering" more than software engineering does. Most engineering disciplines have to deal with tolerances and uncertainty - the real world is non-deterministic. Software engineering is easy in comparison because computers always do exactly what you tell them to do. The ways LLMs fail (and the techniques you have to use to account for that) have more in common…

The tools mechanical and civil engineers use are predictable. You're confusing the things these engineers design, which have tolerances and things like that, with the tools themselves. If an engineer built an internal combustion engine that misfired 60% of the time, it simply wouldn't work. If an engineer measured things with a ruler that only measured correctly 40% of the time, that would be the apt analogy. The too…

> No, and you aren't qualified to even comment on it since your only claim to fame is a fucking web app

Whoa, where did that come from?

Re: Context engineering

#50

Post by skylurk »

simonw wrote: post #49
dingnuts wrote: post #40

Earlier quoted context omitted.

The tools mechanical and civil engineers use are predictable. You're confusing the things these engineers design, which have tolerances and things like that, with the tools themselves. If an engineer built an internal combustion engine that misfired 60% of the time, it simply wouldn't work. If an engineer measured things with a ruler that only measured correctly 40% of the time, that would be the apt analogy. The too…

> No, and you aren't qualified to even comment on it since your only claim to fame is a fucking web app Whoa, where did that come from?

I know, right? I did not predict that output either.
Post reply on HN