Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

AI has a multiplying effect on existing technical skills

joshwcomeau.com

Post reply on HN
Print view View original

251–260 of 344 posts

Re: AI has a multiplying effect on existing technical skills

#251

Post by [deleted] »

ok_dad wrote: post #238
lelanthran wrote: post #180

Earlier quoted context omitted.

> What folks seem to avoid is that a Junior (in ANY subject) has the ability to LEARN so much faster with an AI research assistant, You don't learn by reading, you learn by doing. In this case, simply reading the output of an LLM isn't going to substantially educate you.

You’re not as senior as you think if you think reading code isn’t worth it. Do you think novelists just write novels from nothing? They read books. Software developers need to read software, too. When was the last time you read the code for the best open source software in your industry? I routinely read the libraries I use.

[deleted]

Re: AI has a multiplying effect on existing technical skills

#252

Post by runarberg »

keeda wrote: post #234
runarberg wrote: post #58

Earlier quoted context omitted.

> a Junior (in ANY subject) has the ability to LEARN so much faster with an AI research assistant This is a testable hypotheses with severe lack of citations. Intuition would argue the opposite. We learn by using our brains, if we offload the thinking to a machine and copy their output we don‘t learn. A child does not learn multiplication by using a calculator, and a language learner will not learn a new language by…

Actually, you're both right. Using AI as a supplementary learning aid -- i.e. students use AI as a personalized tutor but still do the assignments themselves -- produces better outcomes. But using AI as a crutch -- i.e. using it to do the assignments -- produces worse outcomes. There is even preliminary research evidence for this, e.g. https://www.mdpi.com/2076-3417/14/10/4115 and https://www.sciencedirect.com/scienc…

> students use AI as a personalized tutor but still do the assignments themselves.

So your first study actually concludes the opposite. It concluded that all AI users performed worse, but the effect was smaller for students which used AI as a tutor.

The second meta analysis I don‘t quite understand. I understand they conclude that using AI tutor shows significant improvement, but I don‘t understand the methodology. I may be misunderstanding but it seems to simply count papers which shows positive outcomes and reaches conclusion that way. I think that methodology is deeply flawed as it will amplify whichever biases are present in the studies it uses. I also think the lack of control groups is a major issues. If we are comparing AI tutor to nothing, off course the AI tutor is gonna perform better. We need to compare to traditional methods. And this is especially relevant in our discussion because junior developers usually have excellent access to senior developers (via peer review, pair programing, etc.), much better then student’s access to tutors for that matter.

So out of the meta-analysis I picked the paper with the strongest claim (trying to steel-man it) which is this one: https://online-journal.unja.ac.id/JIITUJ/article/view/34809/...

It claims the following in the abstract:

> The results indicated that students employing AI tutors shown significant improvements in problem-solving and personalized learning compared to the control group.

Now when I look at the control group it claims this (also in the abstract):

> Participants were allocated to a control group receiving conventional training and an experimental group utilizing AI technology,

But when I look into the methodology section I see this:

> The researchers classified the patients into two groups: MathGPT and Flexi 2.0

MathGPT and Flexi 2.0 are both AI tutors. Now I am confused, where is the control group and how was this “conventional training conducted”?

The methodology section actually tells a different story from the abstract:

> This research utilized a quantitative methodology via a quasi-experimental design.

By quasi-experimental design they mean that they tested the same students before and after AI intervention. And concluded that the AI tutor helped them improve. Now this is not what control group means, so the researchers are actually lying by omission in the abstract. This is a spectacularly bad experimental design and I wonder how it would pass peer review, so I look at the publisher Jurnal Ilmiah Ilmu Terapan Universitas Jambi. So not exactly a reputable journal.

I still stand by my no evidence for a testable hypotheses. I suspect that your first link is actually correct in that AI is bad for students and just less bad if it is used as a tutor.

Re: AI has a multiplying effect on existing technical skills

#253

Post by danielmarkbruce »

8note wrote: post #242
danielmarkbruce wrote: post #177

Earlier quoted context omitted.

It's not confusing. It makes sense.

no, it is confusing. the llm produced something the operator thought was garbage for the design too, and the operator iterated it from garbage to good. they could also have the llm iterate the underlying code from garbage to good, if they wanted. most likely a specialist would say its neither good nor bad, since its not considering the right things, and hasnt collected the right useability feedback, but making straig…

is functional but has bad UI/layout/etc is a thing.

It's only confusing because you don't know the field. Which is kind of the point.

Re: AI has a multiplying effect on existing technical skills

#254

Post by [deleted] »

adjejmxbdjdn wrote: post #130
Waterluvian wrote: post #6

I had an Iron Man moment last week where I was “vibe coding” a UI design with component tests live on the other screen. Iterating by asking it to move things, reduce emphasis of an element, exploring layout options, etc. The loop was near realtime and felt amazing. The code it generated was awful. The kind of garbage that people who don’t know any better would ship: it looked right and it worked. But it was instantly…

Could it be that the fact that the thing you’re an expert at looked like garbage to you, but the things you’re not an expert at, looked just fine, is not a coincidence? You can talk to a bunch of designers who will say the opposite. Claude Design Studio generated this garbage UI, that I fixed manually, but it created great code j never could have that made it work.

[deleted]

Re: AI has a multiplying effect on existing technical skills

#255

Post by joshwcomeau »

xnx wrote: post #12

An "elephant in the room" is a big topic that no one is talking about. Everyone is talking about AI. Better headline: "Why AI Multiplies Developer Skills Rather Than Replacing Them"

For context, this was a newsletter issue sent as part of a marketing campaign for my latest course. It was an elephant in the room in the sense that up until that point, my marketing campaign hadn't talked about AI.

This is the "view on web" link designed for people to read my emails if they don't display correctly in their email client . It’s not really intended for a broader audience.

Re: AI has a multiplying effect on existing technical skills

#256

Post by [deleted] »

simonw wrote: post #184
runarberg wrote: post #166

Earlier quoted context omitted.

This is also a testable hypothesis. I would like to see usage statistics before making assumptions here but my gut feeling is that an overwhelming AI usage (like > 90%) would fall into your category 1. But even with category 2. I think that still does not absolve AI as a cheating machine. Doing research is a skill and if you ask AI to do the research for you that is a skill a junior developer simply never learns.

This is interesting and relevant: https://www.sciencedirect.com/science/article/pii/S095947522... "The expertise reversal effect is present when instructional assistance leads to increased learning gains in novices, but decreased learning gains in experts." There's a whole lot of depth to the question of how AI tools support or atrophy learning for different levels of expertise.

[deleted]

Re: AI has a multiplying effect on existing technical skills

#257

Post by lobf »

reactordev wrote: post #143
adjejmxbdjdn wrote: post #130

Earlier quoted context omitted.

Could it be that the fact that the thing you’re an expert at looked like garbage to you, but the things you’re not an expert at, looked just fine, is not a coincidence? You can talk to a bunch of designers who will say the opposite. Claude Design Studio generated this garbage UI, that I fixed manually, but it created great code j never could have that made it work.

This is the juxtaposition the general public is in. They don’t have advanced tech skills to know any better so they see an output that they can’t produce from their skills and think it’s great. Maybe it is, maybe it isn’t. What does the code look like?

This is where I’m at. I’ve always been a computer tinkerer but a novice coder at best. I work in the film industry, so I don’t need to know how to code.

Where I’m at when building personal applications for my home / life is: does the code execute and perform the desired task?

If so, what do I care how shitty it is? I’m not publishing these projects (for the most part… I have one joke application up at songshift.reachnick.co) so efficient, clean, secure code are not really a priority for me.

Re: AI has a multiplying effect on existing technical skills

#258

Post by joshwcomeau »

bluefirebrand wrote: post #149
xnx wrote: post #12

An "elephant in the room" is a big topic that no one is talking about. Everyone is talking about AI. Better headline: "Why AI Multiplies Developer Skills Rather Than Replacing Them"

The outcome is the same, though Fewer developers required to achieve the same things means a lot of people are going to be unemployed It also means that the people who remain will likely be paid less. Why would you pay a senior salary when you could pay a junior salary plus AI subscription and get "the same result"? I think Software Devs are in for a rough time. I've been doing this for 15 years now, and I'm not look…

> Fewer developers required to achieve the same things means a lot of people are going to be unemployed

I don’t think that's a foregone conclusion. Every company I’ve worked for has had a huge list of tasks we'd do if we had more engineering resources. There's never been a shortage of worthwhile things we could do, it's always been ruthless prioritization to find the 10% of tasks that are the most important.

Look up Jevons' Paradox. This is a thing that has happened a bunch of times before.

Re: AI has a multiplying effect on existing technical skills

#259

Post by lobf »

danielmarkbruce wrote: post #253
8note wrote: post #242

Earlier quoted context omitted.

no, it is confusing. the llm produced something the operator thought was garbage for the design too, and the operator iterated it from garbage to good. they could also have the llm iterate the underlying code from garbage to good, if they wanted. most likely a specialist would say its neither good nor bad, since its not considering the right things, and hasnt collected the right useability feedback, but making straig…

is functional but has bad UI/layout/etc is a thing. It's only confusing because you don't know the field. Which is kind of the point.

> is functional but has bad UI/layout/etc is a thing.

Tell me about it… I was forced to use a program called Farmer’s Wife for a time. What a fucking nightmare of a UX.

Re: AI has a multiplying effect on existing technical skills

#260

Post by joshwcomeau »

block_dagger wrote: post #235

> You could give me Jimi Hendrix’s exact guitar but it would sound very different if I tried to play it! Guitars do not think. AI does. The analogies that try to paint AI as "just another inanimate tool" are way off base, and so is the conclusions of this article.

I guess that depends on how you define "think". I wouldn't say that AI thinks.

I admit the analogies aren't perfect, but the analogies are mostly used to help explain the empirical stuff I’m seeing in the real world. Are you seeing something different?

Post reply on HN