[flagged]
The threat is comfortable drift toward not understanding what you're doing
511–520 of 668 posts
Re: The threat is comfortable drift toward not understanding what you're doing
#512I dunno. Claude helped me implement a new memory allocator, compacting garbage collector and object heap for my programming language. I certainly understood what I was doing when I did this. The experience was extremely engaging for me. Claude taught me a lot. I think the real danger is no longer caring about what you're doing. Yesterday I just pointed Claude at my static site generator and told it to clean it up. I…
This seems contradictory at first glance, if you didn't actually implement it then how well have you actually understood it? It's known from learning theory that engagement or even self-reported understanding doesn't imply that the student can actually solve problems presented to him.
If someone claims to have "understood [middle school] algebra" but they aren't able to solve equations by themselves, you'd be skeptical. Of course past some point of familiarity it's simply faster to throw things into a CAS, but if you remove the initial manual struggle, then have you wired up your brain for understanding? There was a post on HN a few days back about how familiarity with a tool leads to a sense of "embodied understanding" [1], and I think the initial struggle is an intrinsic part of learning to get to the "unconscious competence' level.
Re: The threat is comfortable drift toward not understanding what you're doing
#513The thing is, agents aren’t going away. So if Bob can do things with agents, he can do things. I mourn the loss of working on intellectually stimulating programming problems, but that’s a part of my job that’s fading. I need to decide if the remaining work - understanding requirements, managing teams, what have you - is still enjoyable enough to continue. To be honest, I’m looking at leaving software because the job…
> So if Bob can do things with agents, he can do things. The problem arrises when Bob encounters a problem too complex or unique for agents to solve. To me, it seems a bit like the difference between learning how to cook versus buying microwave dinners. Sure, a good microwave dinner can taste really good, and it will be a lot better than what a beginning cook will make. But imagine aspiring cooks just buying premade…
> The problem arrises when Bob encounters a problem too complex or unique for agents to solve.
Or even sooner, when Bob’s internet connection is down, or he ran out of tokens, or has been banned from his favourite service, or the service is down, or he needs to solve a problem with a machine unable to run local models, or essentially any situation where he’s unable to use an LLM.
Re: The threat is comfortable drift toward not understanding what you're doing
#514Earlier quoted context omitted.
> At the type of place where I taught until recently—a small, selective, private liberal-arts college—administrators can go quite far in limiting AI use, if they have the guts to do so. They should commit to a ruthless de-teching not just of classrooms but of their entire institution. Get rid of Wi-Fi and return to Ethernet, which would allow schools greater control over where and when students use digital technologi…
> If students want to type notes in class or papers in the library, they can use digital typewriters, which have word processing but nothing else. Only, replacing the guts of such a machine to contain a local LLM is damn easy today. Right now the battery mass required to power the device would be a giveaway, but inference is getting energetically cheaper. > Colleges that are especially committed to maintaining this t…
Are you suggesting we should do nothing if the solution has any flaws?
Re: The threat is comfortable drift toward not understanding what you're doing
#515> Schwartz's experiment is the most revealing, and not for the reason he thinks. What he demonstrated is that Claude can, with detailed supervision, produce a technically rigorous physics paper. What he actually demonstrated, if you read carefully, is that the supervision is the physics. Claude produced a complete first draft in three days. It looked professional. The equations seemed right. The plots matched expecta…
My cynical take is that the 20th century exhausted the major scientific discoveries that move various global needles, and now doing well at fostering science matters less.
In the past, if you neglected your institutes and academia, what happened? A rival state got electricity/cars/nuclear weapons first and you would be SOL.
These days, what happens? They invent faster phones? Higher res video?
This take may be a bit hyperbolic but I find it's a good thought exercise prompt.
Re: The threat is comfortable drift toward not understanding what you're doing
#516Earlier quoted context omitted.
If you don't consider it "production quality" and I don't consider it "production quality" and the paying client does consider it "production quality," which two of these people are wrong?
Why does that matter and not you know... the actual outcomes and damages that are effecting real lives here? Are you seriously arguing that the only thing that determines right from wrong is someone buying the thing? I mean that would explain most of the sickness that is neoliberalism currently infecting the US.
I'm trying to warn that you had better pay attention to which incentives you select and where they lead you, because no one is wholly virtuous, and if you go around believing your moral worth is invariant over your behavior - as distinct from your own evaluation of that behavior - then that moral worth will rapidly diminish toward the negative. I would also like to see a clearer distinction drawn between economics and ethics. But if you imagine yourself to lack either agency or responsibility, over where and to how and for what and to whom you sell your labor, then no comment I could make will aid you.
Re: The threat is comfortable drift toward not understanding what you're doing
#517> Schwartz's experiment is the most revealing, and not for the reason he thinks. What he demonstrated is that Claude can, with detailed supervision, produce a technically rigorous physics paper. What he actually demonstrated, if you read carefully, is that the supervision is the physics. Claude produced a complete first draft in three days. It looked professional. The equations seemed right. The plots matched expecta…
Sadly I don’t see how our current social paradigm works for this. There is no history of any sort of long planning like this or long term loyalty (either direction) with employees and employers for this sort of journeyman guild style training. AI execs are basically racing, hoping we won’t need a Schwartz before they are all gone. But what incentives are in place to high a college grad, have them work without llms fo…
Well, the astrophysics situation is special because, as the article notes, there aren't breakthroughs that can be externally verified.
Other projects' success will be proportional to their number of Schwartz' and so it seems unlikely they disappear. But they may disappear for areas in which there is no immediate money.
Re: The threat is comfortable drift toward not understanding what you're doing
#518> I'm not worried about the machines. The machines are fine. I'm worried about us.
We're collectively racing towards the last scene of "Up", justifying each milestone along the way.
I've used this analogy more times than I thought I ever would; no regrets.
We're entering a post-thought era where knowledge itself is devalued and, worse, _seeking_ that knowledge is discouraged.
There's no time to stumble through learning a codebase when Claude can ship features faster than you can think. There's no time to learn.
Pixar warned us. Asimov warned us. Orwell warned us. This phenomenon isn't new; we just now have the technology to finally execute on this vision.
Re: The threat is comfortable drift toward not understanding what you're doing
#519> Schwartz's experiment is the most revealing, and not for the reason he thinks. What he demonstrated is that Claude can, with detailed supervision, produce a technically rigorous physics paper. What he actually demonstrated, if you read carefully, is that the supervision is the physics. Claude produced a complete first draft in three days. It looked professional. The equations seemed right. The plots matched expecta…
> Which means we need people like Alice! We have to make space for people like Alice, and find a way to promote her over Bob, even though Bob may seem to be faster.
If you are a massive company that owns all of the knowledge and all of the technology needed to apply that knowledge, then you don't need Alice. You don't _want_ Alice. You want more Bobs. It looks better on the books.
Tale as old as time.
Re: The threat is comfortable drift toward not understanding what you're doing
#520Earlier quoted context omitted.
One time when I was a kid I was playing with my older sister's graphing calculator. I had accidentally pressed the base button and now was in hex mode. I did some benign calculation like 10+10 and got 14. I believed it! I went to school the next day and told my teacher that the calculator says that 10+10 is 14, so why does she say it's 20? So she showed me on her calculator. She pressed the hex button and explained w…
Totally on a tangent here, but what kind of calculator would have a hex mode where the inputs are still decimal and only the output is hex..?
I probably got the actual numbers wrong in telling the story. But I do remember seeing a shift key on her calculator that would let you input abcde.