Live data from Hacker News

The threat is comfortable drift toward not understanding what you're doing

ergosphere.blog

611–620 of 668 posts

Re: The threat is comfortable drift toward not understanding what you're doing

#611
> Centuries of pedagogy, defeated by a chat window.

To be fair to pedagogy, it's also being defeated by eons of human brain wiring that says: "People in stories are somewhat real, and the author is always real."

We are disarmed by how well these things mimic all of typical indicators of thought, and it takes work— work our brains can't really sustain—to not fall into assumptions that there's another humanoid bearing some of the responsibility for catching problems.

Re: The threat is comfortable drift toward not understanding what you're doing

#612

Earlier quoted context omitted.

> I’ve now ostensibly understood what a derivative does and what it’s used for, yet I have zero idea how to mathematically do it. Does that make any results I gain from this intuitive understanding any less valuable? From a science standpoint, I'd say whatever "results" you got are completely worthless. > I’ll generally have some sort of hypothesis of what kind of result I’m expecting, given that my understanding is…

Asking all of these questions after (allegedly) reading my entire comment either means you didn't pay attention, in which I'm not going to spend any more effort responding either; or you've completely missed the point, in which case I can probably save myself the effort anyway. In any case, if you're genuinely interested in answers to your questions instead of merely posturing, I suggest you re-read carefully and the…

The problem I have with your logic is that you are hedging your arguments so much that the whole point become meaningless. If you are trying to argue that young aspiring scientists will be able to use LLMs to learn new concepts instead of doing the hard work themselves, then you also need to explain how they will be able to develop the skills to analyze and "run more thorough verification" INDEPENDENTLY of LLMs.

Re: The threat is comfortable drift toward not understanding what you're doing

#613
post #14
post #6

The thing is, agents aren’t going away. So if Bob can do things with agents, he can do things. I mourn the loss of working on intellectually stimulating programming problems, but that’s a part of my job that’s fading. I need to decide if the remaining work - understanding requirements, managing teams, what have you - is still enjoyable enough to continue. To be honest, I’m looking at leaving software because the job…

> So if Bob can do things with agents, he can do things. The problem arrises when Bob encounters a problem too complex or unique for agents to solve. To me, it seems a bit like the difference between learning how to cook versus buying microwave dinners. Sure, a good microwave dinner can taste really good, and it will be a lot better than what a beginning cook will make. But imagine aspiring cooks just buying premade…

I agree with the observation but want to note that it's specific to the particular (and these days, often too common) setup of AI as agent, doing Bob's work for him. There's a different version where the AI is just present in the conversation. It has opinions, it pushes back, it tells Bob his approach is wrong. In that setup Bob is still the one doing the thinking. The year doesn't happen around him because nothing got delegated. The drift in the article is a design choice. It comes from the agent frame, not from AI itself.

Re: The threat is comfortable drift toward not understanding what you're doing

#614
post #399

Earlier quoted context omitted.

>What he proposes is radical It sounds entirely reasonable and moderate to me.

Also entirely ineffective. Banning individual behavior won't prevent collective dysfunction and will only harm honest actors. The only answer that makes sense is reforming the acceptance of work to be resistant to the inevitable (ab)use of AI.

It's the exact same anti-cheating method used to great effect in top level e-sports. You can't trust competitor-supplied hardware, so the only option is for the institution to ban it and supply all hardware itself. Higher education is primarily about competition between job candidates. Eliminating cheating needs to be top priority or the whole system will collapse.

Re: The threat is comfortable drift toward not understanding what you're doing

#615

Earlier quoted context omitted.

> At the type of place where I taught until recently—a small, selective, private liberal-arts college—administrators can go quite far in limiting AI use, if they have the guts to do so. They should commit to a ruthless de-teching not just of classrooms but of their entire institution. Get rid of Wi-Fi and return to Ethernet, which would allow schools greater control over where and when students use digital technologi…

This isn't aimed at you, but this strikes me as exactly the kind of divorced-from-the-real-world thinking that academia is pilloried for all the time. This kind of proposal will never happen, I'd basically stake my life on it. Students (and their parents) have zero interest in this kind of anti-technology nonsense, so it's DOA. College isn't compulsory, and those students aren't some captive audience you can do whate…

If an elite institution offered a low tech pathway with the right amount of gatekeeping, you could make an aspirational trend out of it.

Re: The threat is comfortable drift toward not understanding what you're doing

#616

Earlier quoted context omitted.

I don’t know about anyone else here, but college was not educating because I was at college. I did all of the reading and studying on my own. The classes weren’t very interesting, most of my TAs didn’t speak the native language well at all, nor did half the professors. I enjoyed my time, I made a lot of lifelong friends, and figured out how to live on my own. My buddies that enrolled in boot camp instead of college l…

Sounds like you just had terrible professors because most of mine were good and we learned quite a bit in classes, at least I did. I distinctly remember one professor who, every class, would meander discussion over many topics then find a way to bring them all together at the very end, crystalizing all of these disparate thoughts into one cohesive theory. And he did that every single class that semester. It was a mar…

I remember my calc teachers, married, last name gulick, university of maryland. The calc book was sold as the same book for calc 1/2/3. The couple, gulick were the authors. Every semester they released a new edition, the only thing that changed was the problem set numbers. So, if you took calc 1/2/3, you spent $200/semester for the same fucking book.

Magical times.

Re: The threat is comfortable drift toward not understanding what you're doing

#617

Earlier quoted context omitted.

> But it's far from clear that we're not moving toward a plateau in what these agents can do. It is a debatable topic, and I agree with you that it's unclear whether we will hit the wall or not at some point. But one point I want to mention is that at the time when the AI agents were only conceived and the most popular type of """AI""" was LLM-based chatbot, it also seemed that we're approaching some kind of plateau…

Are agents something special? We already had LLMs that could call tools. Agents are just that, in a loop, right?

Yes, and this is an incredibly powerful idea. Running fluid flow simulations inside an optimization loop (monte carlo + gradient descent) revolutionized aircraft design, nuclear simulations and geophysics. When the tool being called updates the LLM's training data, or runs experiments that the LLM can learn from, then that potentially becomes a self-improvement loop.

Re: The threat is comfortable drift toward not understanding what you're doing

#618
post #292

If this article was written a year ago, I would have agreed. But knowing what I know today, I highly doubt that the outcomes of LLM/non-LLM users will be anywhere close to similar. LLMs are exceptionally good at building prototypes. If the professor needs a month, Bob will be done with the basic prototype of that paper by lunch on the same day, and try out dozens of hypotheses by the end of the day. He will not be ch…

One issue with this is that you remember and learn when you put actual effort. You can read what the LLM teaches you for years, and not reach the same level of understanding than someone who actually struggled trying to design, implement, debug by hand.

I don’t think we have good answers unfortunately. Im very happy to be able to get the exact tools I want for my specific niche and be able to run experiments in no time. But I also see that intellectually I do not engage at the same level. I can justify my reasoning for high level design decisions I tried to get the agent to follow, but if there is an issue or I need to justify an implementation decision that’s way messier. I had that experience a few times over the past year and every time I have to reverse engineer what the agent might have been doing before I can answer, or realizing I completely misunderstood a specific protocol because I didn’t have to actively engage with it

Re: The threat is comfortable drift toward not understanding what you're doing

#619
I agree with this article, and I think it goes further than astrophysics or even physics. As agentic LLMs are starting to prove long-standing open mathematical conjectures and even invent new ones, I fear we may reach a point in mathematical research (which, as Hogg's article describes astrophysics, has no "right edge") where the machines are just better than us. At that point do we just mostly lose interest? You can see it already in the Go-playing community. Why study for years to be a "pretty good" 9P when machines will always play better? What PhD student will spend years on a hard, interesting, deep math problem? Sure, a few will. Some people become monks too. But not many.

Re: The threat is comfortable drift toward not understanding what you're doing

#620

Earlier quoted context omitted.

Asking all of these questions after (allegedly) reading my entire comment either means you didn't pay attention, in which I'm not going to spend any more effort responding either; or you've completely missed the point, in which case I can probably save myself the effort anyway. In any case, if you're genuinely interested in answers to your questions instead of merely posturing, I suggest you re-read carefully and the…

The problem I have with your logic is that you are hedging your arguments so much that the whole point become meaningless. If you are trying to argue that young aspiring scientists will be able to use LLMs to learn new concepts instead of doing the hard work themselves, then you also need to explain how they will be able to develop the skills to analyze and "run more thorough verification" INDEPENDENTLY of LLMs.

> then you also need to explain how they will be able to develop the skills to analyze and “run more thorough verification” INDEPENDENTLY of LLMs

I’m sure the students will manage. This is the exact same discussion we’ve all been through before, during the rise of Wikipedia, just wearing a new hat. The answer is “vet your sources, don’t trust unsourced claims.” The way they’ll develop the skills is the same way aspiring scientists and students have developed them throughout the entirety of human history’s vast corpus across time: by having good teachers teach them.

Here’s a very simple program I thought of from the top of my head in a minute or two. I’m sure people whose job it is to create educational content will be able to come up with something far better:

Design a small research project with as many LLM-tailored pitfalls as possible. It involves real measurements and real data, and the students may use their LLM to whichever extent they wish. Then, we compare results against the reference data, and find out all the myriads of ways in which LLMs can taint the data and the conclusions to be made from it, and then explore ways how to mitigate it.

Probably not perfect and nitpickable to oblivion, but also not the hardest mental exercise I’ve ever subjected myself to.

Science did fine in a world where information took years or decades to travel the globe, people thought diseases were spread by evil mojo and we had a grand total of four liquids circulating inside our bodies, and scientists saying the wrong things were actively hunted down and silenced. It got there. It’ll do fine in a world where you can semantically search every single written source model trainers could get their hands on _and_ ground the results with references to tangible sources using the same natural language query.

Post reply on HN