Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Beyond Recall and the Illusion of Competence

var0.xyz

Post reply on HN
Print view View original

11–20 of 34 posts

Re: Beyond Recall and the Illusion of Competence

#11

Post by jt2190 »

> AI can create the illusion that you understand a system because it can make the system work for you. You ask a question and get an answer. You encounter an error and get a patch. The patch doesn't work, so you provide the new error and get another patch. Eventually the machine finds something that works.

> From the outside, this looks like competence. But if you don't understand why the solution works, you're not actually becoming more capable. You're becoming dependent on the machine to maintain the illusion.

So many things in life operate exactly like this: We don’t need to understand the why, we just accept the solution as-is and move on. An experienced professional has another skill: keeping track of which things matter enough to dig in to and develop mastery, and which things can be “good enough”. This is also informed by the “intuition of how machines behave” that the author mentions.

Re: Beyond Recall and the Illusion of Competence

#12

Post by kayo_20211030 »

cyanydeez wrote: post #9
kayo_20211030 wrote: post #8

> The developers who stand out will be the ones who understand the systems. That has always been true. AI doesn't change it. It just rearranges some elements that come ahead of that understanding. The volume and velocity are challenges but the understanding and judgement is what separates the good and the great. BTW: "systems" in my world include not just the technology components but also the humans who operate, adj…

It is an interesting diametric inversion: before AI, you had people designing good code from the ground up for the love of the game and you had businesses cutting corner due to budgets. Now you have AI infilitrating both sectors, and well. Ok, so it's probably going to be the same problem. With AI, I can now explore more docs and more tests, and push out into the edge cases, all because I've got a local LLM and have…

I think so. Two traditional and _almost_ disjoint sets, the technology realm and the business realm, will start to (be forced to?) have much greater overlap. There have always been political battles along the boundary, but this upcoming one will be bloody and long. The current structures and control mechanisms will have to change appreciably for any of the promises of AI to be realized.

Re: Beyond Recall and the Illusion of Competence

#13

Post by zer00eyz »

cyanydeez wrote: post #9
kayo_20211030 wrote: post #8

> The developers who stand out will be the ones who understand the systems. That has always been true. AI doesn't change it. It just rearranges some elements that come ahead of that understanding. The volume and velocity are challenges but the understanding and judgement is what separates the good and the great. BTW: "systems" in my world include not just the technology components but also the humans who operate, adj…

It is an interesting diametric inversion: before AI, you had people designing good code from the ground up for the love of the game and you had businesses cutting corner due to budgets. Now you have AI infilitrating both sectors, and well. Ok, so it's probably going to be the same problem. With AI, I can now explore more docs and more tests, and push out into the edge cases, all because I've got a local LLM and have…

Im not sure thats why businesses are straying from more LLM use.

Finding where AI is moving the needle has been hard, if not impossible.

Because code, by it self has never moved the needle. It's the product, what the code does that moves the needle.

Building the wrong thing, faster, is just a speed run to a legacy code base. That isnt your development teams fault.

Re: Beyond Recall and the Illusion of Competence

#14

Post by draw_down »

fsnovask wrote: post #10

>Let the AI write the code, not design the solutions. I don't think design is necessarily out of reach for AI. As usual, the closer you get to the bleeding edge of what's been done vs. what's possible, the more thought needs to go into a design for it to be considered good. And TBH, even if you only make it to "it works", that's still laudable. There's plenty of profitable companies with terrible designs, technical d…

[dead]

Re: Beyond Recall and the Illusion of Competence

#15

Post by WillEMac »

I think there is some nuisance on the redundancy required for the software.

For example I've worked in clean tech where PLC programming controls turbines, complete understanding is required here. Every bug and LOC associated a human needs to be in the loop, it's worth the time.

Alternatively I do game development, and I don't think there is value in understanding the complete debugging process. For example I accidentally bound spacebar-release to two things in multiplayer causing a co-op glitch. I do not need to search for this needle in a haystack to debug, that is a bad use of time.

>If writing code becomes cheap and accessible to everyone, then writing code stops being much of a differentiator. The developers who stand out will be the ones who understand systems.

I agree especially for limits/boundary conditions. In the game I'm working on, fundamentally understanding 500 GPU vs. CPU controlled fish on screen and their limitations is required to be a good architect, or this game will run at 10fps.

Re: Beyond Recall and the Illusion of Competence

#16

Post by FrustratedMonky »

WillEMac wrote: post #15

I think there is some nuisance on the redundancy required for the software. For example I've worked in clean tech where PLC programming controls turbines, complete understanding is required here. Every bug and LOC associated a human needs to be in the loop, it's worth the time. Alternatively I do game development, and I don't think there is value in understanding the complete debugging process. For example I accident…

Scale of the impact.

If your game was being played by a million people paying a subscription. Then it would be worth debugging it.

Re: Beyond Recall and the Illusion of Competence

#17

Post by aetherspawn »

chrisjj wrote: post #3

> And the consequences become particularly serious when something happens that the machine can't solve Sadly the bot-gulled will always say "But the next model will be good enough to do that!".

Judging by the rate of things it probably will be

Re: Beyond Recall and the Illusion of Competence

#18

Post by jobuildsstuff »

_bobm wrote: post #5

In your closing thoughts, you don't mention what is this staying firmly in control? how do you stay firmly in control while reaping the benefit of the model while being more productive? because if you start a curriculum over what the model wrote then, one can argue, you are better off writing it yourself in the first place.

One concrete answer from my own small tool: move the things you must stay in control of out of your head and into tests. Our privacy rules (drafts never stored or logged) are executable, so any patch that quietly violates them fails the suite whether or not I reviewed that diff carefully that day. I still read the code, but the invariants no longer depend on my attention holding.

Re: Beyond Recall and the Illusion of Competence

#19

Post by kayo_20211030 »

zer00eyz wrote: post #13
cyanydeez wrote: post #9

Earlier quoted context omitted.

It is an interesting diametric inversion: before AI, you had people designing good code from the ground up for the love of the game and you had businesses cutting corner due to budgets. Now you have AI infilitrating both sectors, and well. Ok, so it's probably going to be the same problem. With AI, I can now explore more docs and more tests, and push out into the edge cases, all because I've got a local LLM and have…

Im not sure thats why businesses are straying from more LLM use. Finding where AI is moving the needle has been hard, if not impossible. Because code, by it self has never moved the needle. It's the product, what the code does that moves the needle. Building the wrong thing, faster, is just a speed run to a legacy code base. That isnt your development teams fault.

> It's the product, what the code does that moves the needle.

Never a truer word said. It's the only thing that moves the needle.

I'm not sure that businesses are straying from AI. I reckon everyone's on board. but AI is now accessible to the business folks - callow as they are to the requirements of the technology components - and it's bound to run up against the technology folks trying to wave them off when enthusiastic excesses threaten what the code does, now or in the future. There'll be adjustments on both sides.

Post reply on HN