Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Context engineering

chrisloy.dev

Post reply on HN
Print view View original

51–60 of 70 posts

Re: Context engineering

#51

Post by simonw »

Scipio_Afri wrote: post #39
simonw wrote: post #37

Earlier quoted context omitted.

It famously does not: https://thinkingmachines.ai/blog/defeating-nondeterminism-in...

Hey Simon, do you have any posts diving into how one might be able to deal with evaluating LLMs or Machine Learning models in general when reproducibility is so difficult given non-determinism? Pytorch has an article on it https://docs.pytorch.org/docs/stable/notes/randomness.html but then doesn't really go into how one would then take this deterministic result, and evaluate a model that is in production (which would…

This is so hard! I don't yet have a great solution for this myself, but I've been collecting notes about this on my "evals" tag for a while: https://simonwillison.net/tags/evals/

The best writing I've seen about this is from Hamel Husain - https://hamel.dev/blog/posts/llm-judge/ and https://hamel.dev/blog/posts/evals-faq/ are both excellent.

Re: Context engineering

#52

Post by voidhorse »

empath75 wrote: post #33
voidhorse wrote: post #32

Earlier quoted context omitted.

I completely agree that much of software engineering is not engineering, and building systems around LLMs is no better in this sense. When the central component of your system is a black box that you cannot reason about, have no theory around, and have essentially no control over (a model update can completely change your system behavior) engineering is basically impossible from the start. Practices like using autosc…

This is completely backwards. Engineers built steam engines first through trial and error and then eventually the laws of thermodynamics were invented to explain how steam engines work. Trial and error and fumbling around and creating rules of thumbs for systems you don’t entirely understand is the purest form of engineering.

I would argue it's more correct to call that phase experimentation. I doubt the early manufacturers of steam machines would even call themselves engineers in a serious or precise sense. They were engineers in the sense of "builder of engine" as a specific object, but the term's meaning has evolved from that basic initial usage.

A discipline becomes engineering when we achieve a level of understanding. such that we can be mathematically precise about it. Of course experimentation and trial and error are a fundamental part of that process, but there's a reason we have a word to distinguish processes which become more certain and precise thereafter and why we don't just call anything and everything engineering of some form.

Re: Context engineering

#54

Post by quequon »

simonw wrote: post #18
aeve890 wrote: post #7

Are we still calling this things engineering?

Yes, and we've also decided that they deserve the title "engineering" more than software engineering does. Most engineering disciplines have to deal with tolerances and uncertainty - the real world is non-deterministic. Software engineering is easy in comparison because computers always do exactly what you tell them to do. The ways LLMs fail (and the techniques you have to use to account for that) have more in common…

Classic shilling behavior of the insufferably embarrassing: redefining words to the benefit of those who pay your bills to the confusion of everyone else.

The definition of engineering, according to people outside the pocket of the llm industry:

> The application of scientific and mathematical principles to practical ends such as the design, manufacture, and operation of efficient and economical structures, machines, processes, and systems.

How do these techniques apply scientific and mathematical principals?

I would argue to do either of those requires reproducibility, and yet somehow you are arguing the less reproducible something is that the more like "physical engineering" it becomes.

Re: Context engineering

#55

Post by Balgair »

alecco wrote: post #20

This looks AI generated slop.

I know this is a bit of a non sequitur but, on my feed just below your comment, some asked for the RSS for this blog. The juxtaposition of the two comments here is just soooo HN

Re: Context engineering

#56

Post by [deleted] »

quequon wrote: post #54
simonw wrote: post #18

Earlier quoted context omitted.

Yes, and we've also decided that they deserve the title "engineering" more than software engineering does. Most engineering disciplines have to deal with tolerances and uncertainty - the real world is non-deterministic. Software engineering is easy in comparison because computers always do exactly what you tell them to do. The ways LLMs fail (and the techniques you have to use to account for that) have more in common…

Classic shilling behavior of the insufferably embarrassing: redefining words to the benefit of those who pay your bills to the confusion of everyone else. The definition of engineering, according to people outside the pocket of the llm industry: > The application of scientific and mathematical principles to practical ends such as the design, manufacture, and operation of efficient and economical structures, machines,…

[deleted]

Re: Context engineering

#57

Post by simonw »

quequon wrote: post #54
simonw wrote: post #18

Earlier quoted context omitted.

Yes, and we've also decided that they deserve the title "engineering" more than software engineering does. Most engineering disciplines have to deal with tolerances and uncertainty - the real world is non-deterministic. Software engineering is easy in comparison because computers always do exactly what you tell them to do. The ways LLMs fail (and the techniques you have to use to account for that) have more in common…

Classic shilling behavior of the insufferably embarrassing: redefining words to the benefit of those who pay your bills to the confusion of everyone else. The definition of engineering, according to people outside the pocket of the llm industry: > The application of scientific and mathematical principles to practical ends such as the design, manufacture, and operation of efficient and economical structures, machines,…

Being accused of shilling for saying that context engineering is closer to traditional engineering than software engineering is a new one for me.

Re: Context engineering

#58

Post by CjHuber »

elteto wrote: post #2

Are there any open source examples of good context engineering or agent systems?

I‘d consider DSPy to be one. While the prompts it is using are not the most elaborate, they are well tested and reliable

Re: Context engineering

#60

Post by graemefawcett »

dingnuts wrote: post #40
simonw wrote: post #18

Earlier quoted context omitted.

Yes, and we've also decided that they deserve the title "engineering" more than software engineering does. Most engineering disciplines have to deal with tolerances and uncertainty - the real world is non-deterministic. Software engineering is easy in comparison because computers always do exactly what you tell them to do. The ways LLMs fail (and the techniques you have to use to account for that) have more in common…

The tools mechanical and civil engineers use are predictable. You're confusing the things these engineers design, which have tolerances and things like that, with the tools themselves. If an engineer built an internal combustion engine that misfired 60% of the time, it simply wouldn't work. If an engineer measured things with a ruler that only measured correctly 40% of the time, that would be the apt analogy. The too…

If you're claiming those to be the success ratios you're having with AI assisted engineering, perhaps the phrase context in, tokens out might help. The relationship is symmetrical I have found.

In general, the more constraints you apply on the solution space via context, the more likely the correct solution is to stabilize.

It also helps to engineer the solution in such a way that the correct solution is also the easiest and this the most likely.

It takes time, but like most skills can be learned.

Post reply on HN