Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

The threat is comfortable drift toward not understanding what you're doing

ergosphere.blog

Post reply on HN
Print view View original

461–470 of 668 posts

Re: The threat is comfortable drift toward not understanding what you're doing

#461

Post by cvwright »

mojuba wrote: post #455
Wowfunhappy wrote: post #227

> Schwartz's experiment is the most revealing, and not for the reason he thinks. What he demonstrated is that Claude can, with detailed supervision, produce a technically rigorous physics paper. What he actually demonstrated, if you read carefully, is that the supervision is the physics. Claude produced a complete first draft in three days. It looked professional. The equations seemed right. The plots matched expecta…

> Which means we need people like Alice! We have to make space for people like Alice, and find a way to promote her over Bob The solution is relatively simple though - not sure the article suggests this as I only skimmed through: Being good in your field doesn't only mean pushing articles but also being able to talk about them. I think academia should drift away from written form toward more spoken form, i.e. confere…

I thought that was kind of how the hard sciences work already?

My grad school friend who was a physicist would write his talk just before his conferences, and then submit the paper later. My experience in CS was totally backwards from that.

Re: The threat is comfortable drift toward not understanding what you're doing

#462

Post by doug_durham »

Wowfunhappy wrote: post #227

> Schwartz's experiment is the most revealing, and not for the reason he thinks. What he demonstrated is that Claude can, with detailed supervision, produce a technically rigorous physics paper. What he actually demonstrated, if you read carefully, is that the supervision is the physics. Claude produced a complete first draft in three days. It looked professional. The equations seemed right. The plots matched expecta…

The article is a thought experiment. The author hypothesizes that Bob isn't getting the same benefit that Alice is getting. That hypothesis could be wrong. I don't know and the author doesn't know. It could be that Bob is going to have a very successful career and will deeply know the field because he is able to traverse a wider set of problems more quickly. At this point, it's just hypothesis. I don't think that we can say we need more Alices any more than we can say we need more Bobs. Unfortunately we will have to wait and see. It will be upon the academic community to do the work to enforce quality controls. That is probably the weakness to worry about.

Re: The threat is comfortable drift toward not understanding what you're doing

#463

Post by sandruso »

sd9 wrote: post #6

The thing is, agents aren’t going away. So if Bob can do things with agents, he can do things. I mourn the loss of working on intellectually stimulating programming problems, but that’s a part of my job that’s fading. I need to decide if the remaining work - understanding requirements, managing teams, what have you - is still enjoyable enough to continue. To be honest, I’m looking at leaving software because the job…

Programmers have ultimate (or they had) skill to solve anything if you have enough resources.

Now, you don't do thing and do other things when LLMs get stuck. There is no "given enough time I can do it".

I can't see how somebody would go solving slop bugs (slugs :)) in heavy AI generated codebase.

Hope, I'm wrong but that's somehing I personally encountered. Stay sharp.

Re: The threat is comfortable drift toward not understanding what you're doing

#464

Post by Warwolt »

stavros wrote: post #42
troupo wrote: post #35

Earlier quoted context omitted.

How can Bob produce novel things when he lacks the skills to do even trivial things? I didn't get to be a senior engineer by immediately being able to solve novel problems. I can now solve novel problems because I spent untold hours solving trivial ones.

Because trivial things aren't a prerequisite for novel things, as any theoretical mathematician who can't do long division will tell you.

That's not a good analogy. A good mathematician isn't necessarily dealing with calculations, i.e. long division, but rather with proof.

No-ones becomes a good mathematician without first learning to write simple proofs, and then later on more complex proof. It's the very stuff of the field itself.

Re: The threat is comfortable drift toward not understanding what you're doing

#465

Post by throw310822 »

Wowfunhappy wrote: post #227

> Schwartz's experiment is the most revealing, and not for the reason he thinks. What he demonstrated is that Claude can, with detailed supervision, produce a technically rigorous physics paper. What he actually demonstrated, if you read carefully, is that the supervision is the physics. Claude produced a complete first draft in three days. It looked professional. The equations seemed right. The plots matched expecta…

> the paradox is, the LLMs are only useful† if you're Schwartz, and you can't become Schwartz by using LLMs.

That you can't "become Schwartz" by using LLMs is an unproven assumption. Actually, it's a contradiction in the logic of the essay: if Bob managed to produce a valid output by using an LLM at all, then it means that he must have acquired precisely that supervision ability that the essay claims to be necessary.

Btw, note that in the thought experiment Bob isn't just delegating all the work to the LLM. He makes it summarise articles, extract important knowledge and clarify concepts. This is part of a process of learning, not being a passive consumer.

Re: The threat is comfortable drift toward not understanding what you're doing

#466

Post by techblueberry »

itmitica wrote: post #434
techblueberry wrote: post #239

Earlier quoted context omitted.

What if AI in the long run makes us slower and less effective. As someone who is one of the folks supercharged by these tools, I could see it. I think people are underestimating the level of experience and knowledge that’s required to prompt LLM’s. Not in the micro sense but in the macro. It seems so easy because it feels easy. But if you don’t have deep understanding of the domain, it will just feel impossible. The…

My experience is AI helping me unload things I am not built for. It allows me to be creative with less drag. Not everyone aims to be useless human automaton or useless human thesaurus. It allows me to exercise my intelligence freely.

We’re kind of saying the same thing. But If you only learn LLMs for work, and then you only do the work assigned with the LLM and you don’t see it as an opportunity to leverage creativity and curiosity, you won’t be adding value to the process and someone else will take your job.

Re: The threat is comfortable drift toward not understanding what you're doing

#467

Post by techblueberry »

therealdrag0 wrote: post #427
techblueberry wrote: post #337

Earlier quoted context omitted.

The problem is in most environments Bob won’t spend the rest of the year figuring out what the LLM did, because bob will be busy promoting the LLM for the next deliverable, and the problem is that if all bob has time for us to prompt LLMs, and not understand, there will be a ceiling to Bob’s potential. This won’t affect everyone equally. Some Bob’s will nerd out and spend their free time learning, but other Bob’s won…

Why would bob only have time to promote llms? Strange strawman. Many uni courses always had a level of you get out what you put in, it’s the same with LLMs.

Why would the university look at the amount of work a student get done, conclude the student can get 12x done because they can do a years work in a month and not make the student do 12x more work?

And it’s not strictly speaking university we’re talking about. The way we understand work is going to fundamentally change. And we’re not going to value the people who use LLMs to get 1x done.

But yes, university was always about how much work you put into it, and LLM’s are going to make that 10x more obvious.

The point is the Bob and Alice comparison is already a straw man, but I do squarely believe it’s the people with the best mental models and not the people who “get AI” who will win the new world. If you’re curious and good at developing mental models, you can learn “AI” in a week. But if you’re curious and good at developing mental models you’ve probably already lapped both Bob and Alice

Re: The threat is comfortable drift toward not understanding what you're doing

#468

Post by vinceguidry »

Wowfunhappy wrote: post #227

> Schwartz's experiment is the most revealing, and not for the reason he thinks. What he demonstrated is that Claude can, with detailed supervision, produce a technically rigorous physics paper. What he actually demonstrated, if you read carefully, is that the supervision is the physics. Claude produced a complete first draft in three days. It looked professional. The equations seemed right. The plots matched expecta…

I've been using ChatGPT to re-bootstrap my coding hobby. After the initial honeymoon wore off, I realized I was staring down the barrel of a dilemma. If I use AI to "just handle" the parts of the system I don't want to understand, I invariably end up in a situation where I gotta throw a whole bunch of work out. But I can't supervise without an understanding of what it's supposed to be doing, and if I knew what it was supposed to be doing, I could just do it myself.

So I settled on very incremental work. It's annoying cutting and pasting code blocks into the web interface while I'm working on my interface to Neovim, spent a whole day realizing I can't trust it to instrument neovim and don't want to learn enough lua to manage it. (I moved onto neovim from Emacs because I don't like elisp and gpt is even worse at working on my emacs setup than neovim, the end goal is my own editor in ruby but gpt damn sure can't understand that atm) But at least I'm pushing a real flywheel and not the brooms from Fantasia.

Re: The threat is comfortable drift toward not understanding what you're doing

#469

Post by foobarchu »

vinceguidry wrote: post #444
adriand wrote: post #436

Earlier quoted context omitted.

It’s a deliberate process powered by rightwing and capitalist interests designed to create a dumber, less educated and more distracted population. A war as stupid as the one with Iran would not have been possible three decades ago. As ill-advised as the Iraq war was, Bush at least spent months explaining the rationale and building support for it, successfully. Now that’s not needed. I saw interviews with young Americ…

I was there for that war. Kids weren't listening and didn't care back then either. If anything, Gen Z is the most politically-aware generation we've had since we started keeping track. Trump doesn't have to justify a single thing because the billionaires behind him know that every last bet is off and their very livelihoods are at risk, and his entire base of support up and down the chain are either complicit or foole…

I was there as well, the bush presidency lasted my entire middle and high school career, and I got the chance to vote for Obama in my senior year.

I remember things very differently. Everyone cared about the Iraq war, gay-straight alliance was one of the most up and coming clubs, and political music was everywhere. Green Day had their big second wave with American Idiot, System Of A Down was on top of the world, Rock Against Bush was huge, anarcho-punk like Rise Against was getting big.

I'm not a teenager anymore obviously, so it's entirely possible I'm just missing it, but I've seen very little that compares to those sort of movements. On the other hand, most millennials I know are still wildly politically active.

Re: The threat is comfortable drift toward not understanding what you're doing

#470

Post by bigstrat2003 »

sd9 wrote: post #6

The thing is, agents aren’t going away. So if Bob can do things with agents, he can do things. I mourn the loss of working on intellectually stimulating programming problems, but that’s a part of my job that’s fading. I need to decide if the remaining work - understanding requirements, managing teams, what have you - is still enjoyable enough to continue. To be honest, I’m looking at leaving software because the job…

Agents may not go away, but they are going to fall off significantly once people wake up to how bad they are at making software. It's like in the early 00s when business execs were stoked about the idea that they could cut costs by hiring bottom rate Indian contractors: it turned out to be a disaster for quality, and eventually there was a shift back towards having staff in the US. The same thing is going to happen with LLMs.
Post reply on HN