Saw this the other day and it made me think that too much effort and credence is being given to this idea of crafting the perfect environment for LLMs to thrive in. Which to me, is contrary to how powerful AI systems should function. We shouldn’t need to hold its hand so much. Obviously we’ve got to tame the version of LLMs we’ve got now, and this kind of thinking is a step in the right direction. What I take issue w…
We shouldn't but it's analogous to how CPU usage used to work. In the 8 bit days you could do some magical stuff that was completely impossible before microcomputers existed. But you had to have all kinds of tricks and heuristics to work around the limited abilities. We're in the same place with LLMs now. Some day we will have the equivalent of what gigabytes or RAM are to a modern CPU now, but we're still stuck in t…
The new skill in AI is not prompting, it's context engineering
191–200 of 550 posts
Re: The new skill in AI is not prompting, it's context engineering
#192Earlier quoted context omitted.
Not really. Got some code you don't understand? Feed it to a model and ask it to add comments. Ultimately humans will never need to look at most AI-generated code, any more than we have to look at the machine language emitted by a C compiler. We're a long way from that state of affairs -- as anyone who struggled with code-generation bugs in the first few generations of compilers will agree -- but we'll get there.
>any more than we have to look at the machine language emitted by a C compiler. Some developers do actually look at the output of C compilers, and some of them even spend a lot of time criticizing that output by a specific compiler (even writing long blog posts about it). The C language has an ISO specification, and if a compiler does not conform to that specification, it is considered a bug in that compiler. You can…
Those programmers don't get much done compared to programmers who understand their tools and use them effectively. Spending a lot of time looking at assembly code is a career-limiting habit, as well as a boring one.
I do not know what prompt engineering will look like in the future, but without AGI, I remain skeptical about verification of different kinds of code not being required in at least a sizable proportion of cases. That does not exclude usefulness of course: for instance, if you have a case where verification is not needed; or verification in a specific case can be done efficiently and robustly by a relevant expert; or some smart method for verification in some cases, like a case where a few primitive tests are sufficient.
Determinism and verifiability is something we'll have to leave behind pretty soon. It's already impossible for most programmers to comprehend (or even access) all of the code they deal with, just due to the sheer size and scope of modern systems and applications, much less exercise and validate all possible interactions. A lot of navel-gazing about fault-tolerant computing is about to become more than just philosophical in nature, and about to become relevant to more than hardware architects.
In any event, regardless of your and my opinions of how things ought to be, most working programmers never encounter compiler output unless they accidentally open the assembly window in their debugger. Then their first reaction is "WTF, how do I get out of this?" We can laugh at those programmers now, but we'll all end up in that boat before long. The most popular high-level languages in 2040 will be English and Mandarin.
May I ask what your professional background and experience is?
Probably ~30 kloc of C/C++ per year since 1991 or thereabouts. Possibly some of it running on your machine now (almost certainly true in the early 2000s but not so much lately.)
Probably 10 kloc of x86 and 6502 assembly code per year in the ten years prior to that.
But I have no experience with LLMs or prompt engineering.
May I ask why not? You and the other users who voted my post down to goatse.cx territory seem to have strong opinions on the subject of how software development will (or at least should) work going forward.
Re: The new skill in AI is not prompting, it's context engineering
#193Honestly, GPT-4o is all we ever needed to build a complete human-like reasoning system. I am leading a small team working on a couple of “hard” problems to put the limits of LLMs to the test. One is an options trader. Not algo / HFT, but simply doing due diligence, monitoring the news and making safe long-term bets. Another is an online research and purchasing experience for residential real-estate. Both these tasks,…
Re: The new skill in AI is not prompting, it's context engineering
#194You can give most of the modern LLMs pretty darn good context and they will still fail. Our company has been deep down this path for over 2 years. The context crowd seems oddly in denial about this
Re: The new skill in AI is not prompting, it's context engineering
#195The only engineering going on here is Job Engineering™
It is really funny to see the hyper fixation on relabeling of soft skills / product development to " Engineering" in the AI space.
Re: The new skill in AI is not prompting, it's context engineering
#196Earlier quoted context omitted.
Not really. Got some code you don't understand? Feed it to a model and ask it to add comments. Ultimately humans will never need to look at most AI-generated code, any more than we have to look at the machine language emitted by a C compiler. We're a long way from that state of affairs -- as anyone who struggled with code-generation bugs in the first few generations of compilers will agree -- but we'll get there.
> Not really. Got some code you don't understand? Feed it to a model and ask it to add comments. Absolutely not. An experienced individual in their field can tell if the AI made a mistake in the comments / code rather than the typical untrained eye. So no, actually read the code and understand what it does. > Ultimately humans will never need to look at most AI-generated code, any more than we have to look at the mac…
If you don't review the code your C compiler generates now, why not? Compiler bugs still happen, you know.
Re: The new skill in AI is not prompting, it's context engineering
#197Earlier quoted context omitted.
Sometimes buzzwords turn out to be mirages that disappear in a few weeks, but often they stick around. I find they takeoff when someone crystallizes something many people are thinking about internally, and don’t realize everyone else is having similar thoughts. In this example, I think the way agent and app builders are wrestling with LLMs is fundamentally different than chatbots users (it’s closer to programming), a…
I agree - what distinguishes this is how rushed and self-aware it is. It is being pushed top down, sheepishly. EDIT: Ah, you also wrote the blog posts tied to this. It gives 0 comfort that you have a blog post re: building buzz phrases in 2020, rather, it enhances the awkward inorganic rush people are self-aware of.
And I wrote the first post before the meme.
Re: The new skill in AI is not prompting, it's context engineering
#198> Then you can generate a response. > > Hey Jim! Tomorrow’s packed on my end, back-to-back all day. Thursday AM free if that works for you? Sent an invite, lmk if it works. Feel free to send generated AI responses like this if you are a sociopath.
Re: The new skill in AI is not prompting, it's context engineering
#199Saw this the other day and it made me think that too much effort and credence is being given to this idea of crafting the perfect environment for LLMs to thrive in. Which to me, is contrary to how powerful AI systems should function. We shouldn’t need to hold its hand so much. Obviously we’ve got to tame the version of LLMs we’ve got now, and this kind of thinking is a step in the right direction. What I take issue w…
It may not be a silver bullet, in that it needs lots of low level human guidance to do some complex task. But looking at the trend of these tools, the help they are requiring is become more and more higher level, and they are becoming more and more capable of doing longer more complex tasks as well as being able to find the information they need from other systems/tools (search, internet, docs, code etc...). I think…
Re: The new skill in AI is not prompting, it's context engineering
#200Earlier quoted context omitted.
You can look at AnyTool, 2024 (16,000 tools) and start looking at newer research from there. https://arxiv.org/abs/2402.04253 For long contexts start with activation beacons and RoPE scaling.
I would classify AnyTool as a context engineering trick. It's using GPT-4 function calls (what we would call tool calls today) to find the best tools for the current job based on a 3-level hierarchy search. Drew calls that one "Tool Loadout" https://www.dbreunig.com/2025/06/26/how-to-fix-your-context....
This field, I swear...it's the PPAP [1] of engineering.
[1] https://www.youtube.com/watch?v=NfuiB52K7X8
I have a toool...I have a seeeeearch...unh! Now I have a Tool Loadout!" *dances around in leopard print pyjamas*