Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

LLMs reward expertise

seangoedecke.com

Post reply on HN
Print view View original

191–200 of 607 posts

Re: LLMs reward expertise

#191

Post by krisoft »

I did a test a few months ago. A friend of mine wanted to develop what i understood to be a simple single page web app. But since she didn’t have any software engineering experience she asked me to help. Around that time everyone was talking about how literally anyone can develop software with LLMs i asked her if she could give it a try first, and if I could watch the attempt.

I was fully expecting that writing the code will pose no problem for the AI. But i was curious if the AI will realise that my friend is a novice and needs extra help with things like: copy pasting the code into a text file and saving it with an html extension, helping her host the file online so she can share it with others, buying a domain for it, etc. I assumed they will get there eventually, but i also assumed that it will take a lot of stumbling around and misunderstandings.

But i was completely wrong. They didn’t even get to that point. Because my friend didn’t have the vocabulary to ask the AI to write code. They were just going around in circles where the AI was brainstorming with her about possible features and getting thints more and more complicated. We terminated the experiment after one and a half hours and many many messages exchanged between her and the LLM.

Whereas to me who knows the terminology would have probably just taken a single exchange of messages to get the result she described to me. I would have prompted with something like “Please write to me an html page which does X, Y, Z.” But since she didn’t know the right terminology she got into a vortex of feature discusion, and she didn’t find a way to tip the AI into “just do it, write it now” mode. In other words in that case the LLM would have rewarded even just a little bit of expertise, but without it there was a confusion about goals between the human and the machine.

Re: LLMs reward expertise

#192

Post by matthew-wegner »

krisoft wrote: post #191

I did a test a few months ago. A friend of mine wanted to develop what i understood to be a simple single page web app. But since she didn’t have any software engineering experience she asked me to help. Around that time everyone was talking about how literally anyone can develop software with LLMs i asked her if she could give it a try first, and if I could watch the attempt. I was fully expecting that writing the c…

Are you describing a "chat window" experience here? This is apples to oranges.

Re: LLMs reward expertise

#193

Post by theturtletalks »

Love this idea of reading prompts that lead to new discoveries and figuring out how the person got the LLM there. It truly is an art and I’m always reminded of “I, Robot” and the scene about “you must ask the right questions.”

Re: LLMs reward expertise

#194

Post by Lerc »

Greed wrote: post #141
chr15m wrote: post #135

Earlier quoted context omitted.

Read the code.

What if the code sucks, because it was vibe coded by an LLM over a dozen disparate sessions?

If it performs the overall function but the code is a mess, then you can ask the model to describe parts of it, the structures and what the structures are for. When it shows you something that is messed up, suggest a better framing for the same thing and ask the model to implement the better framing without regressing on already passing tests.

I'm going through this right now on a very difficult to implement task, the original was python and very verbose. But had facilitated a rust implementation that produced byte identical outputs. Then I asked it for what data was being passed around, placing restrictions on what passed between interfaces I could tell it what parts should be immutable and what parts should have no presence outside it's local context. Placing those limitations while having a exemplar of what it should be doing gives it little choice but to make better code if it meets the conditions set while at the same time not regressing.

I say things like 'this field is a implementation detail in a declarative data structure, it should not exist here.

This can mean hours of work with no observable change in program function, yet it is directly addressing the limitations that prevents it from being used in larger tasks.

Re: LLMs reward expertise

#195

Post by Amekedl »

You got to know how to use the model+harness+prompt to achieve the results you want, but honestly for many projects and questions all the models already pump out their same best version of an answer.

Sometimes it is really akin to a git clone, although it was a LLM request.

This rewarding expertise is somewhat wishful thinking.

At the end of a day, it feels and is more like gambling, even with the recommended expertise and a good approach, don't delude yourself you're simply pulling the lever too, as any novice.

Re: LLMs reward expertise

#196

Post by Synthetic7346 »

matthew-wegner wrote: post #192
krisoft wrote: post #191

I did a test a few months ago. A friend of mine wanted to develop what i understood to be a simple single page web app. But since she didn’t have any software engineering experience she asked me to help. Around that time everyone was talking about how literally anyone can develop software with LLMs i asked her if she could give it a try first, and if I could watch the attempt. I was fully expecting that writing the c…

Are you describing a "chat window" experience here? This is apples to oranges.

Yeah I wonder if they had given their friend Claude code or Codex, would it have been more likely to create what she wanted?

Re: LLMs reward expertise

#197

Post by theturtletalks »

Domain knowledge will stand alone as the sole differentiator. Because LLM benefits can be reaped by almost anyone and it’s a force multiplier. Now those who have the strongest initial force will have a far bigger edge than before.

Re: LLMs reward expertise

#198

Post by ehnto »

theturtletalks wrote: post #197

Domain knowledge will stand alone as the sole differentiator. Because LLM benefits can be reaped by almost anyone and it’s a force multiplier. Now those who have the strongest initial force will have a far bigger edge than before.

Real world domain knowledge and experience cuts through the chaff too. LLMs are going to have people reinventing the wheel and wasting tonnes of time on stuff that won't work out. If you're a domain expert you are going to be much more aware of how to focus effort in the right places, and what's actually needed or been tried before in your niche.

A lot of this domain knowledge is not in any training data, it's locked up in companies in the industry. I suspect it will get even more important to guard it.

Re: LLMs reward expertise

#200

Post by QuercusMax »

natsucks wrote: post #8

I am feeling this a lot lately. Getting the most out of agents seems to require being able to ask the right question. And how can you ask the right questions without deep domain expertise?

I started developing webapps back in the late 90s when I was in high school using Perl, and I've worked with tons of technologies up till around 2014 or so when I shifted into almost pure backend work and lost touch with modern frontend development.

I'm now learning how modern frontend development is done (for both personal and professional projects), so I may not know the specific tools, technologies, or terms but I can say "whatever the equivalent of XYZ is" and the models will translate for me. If I say "run pytype" it will tell me "we're using mypy - i'll run that checker for you".

If you can express what problem you're trying to solve, that will get you most of the way - and then you can refine by asking questions. "I think I need something like Redis for caching things - do people still use that? Is there a simpler more modern version that is the new standard? Do we already have company docs suggesting what to use?"

Post reply on HN