Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Automation Without Understanding

arxiv.org

Post reply on HN
Print view View original

21–30 of 64 posts

Re: Automation Without Understanding

#21

Post by taurath »

DaiPlusPlus wrote: post #15
sachaa wrote: post #9

What worries me isn’t AI replacing experts, it’s that we may stop producing people who know enough to notice when AI is confidently wrong.

> it’s that we may stop producing people who know enough to notice when AI is confidently wrong. The running-joke is that a LinkedIn-lunatic AI booster, with a Nano Banana-generated profile-pic, will immediately slide into the chat to tell you that that this is already a solved problem: just spin up another agent to do the work to verify the first agent. Token-cost-be-damned. And we laugh and downvote them to oblivio…

How is the token-cost-be-damned part in the latter example?

I do find that both porting and translation projects have a much higher signal given the ease of mapping to tokens, when there is a proven working source to refer to - the source itself provides the validation. In a new project, you don’t have that validation.

Re: Automation Without Understanding

#22

Post by lioeters »

measurablefunc wrote: post #19
lioeters wrote: post #12

Earlier quoted context omitted.

Curious to see if that can map to what's happening in the software industry/community. > The product of software engineering (or computer science) is clarity and understanding. Not programs, by themselves. Their importance is not just in their specific statements (lines of code in a specific language), but their role in challenging our understanding, presenting challenges that led to computational (?) developments th…

Peter Naur wrote an essay about exactly that https://www.sciencedirect.com/science/article/abs/pii/016560...

Indeed, both Thurston's quote about mathematics, and Naur's programming as theory building, are classics that are relevant now more than ever. A download link for the latter: https://gwern.net/doc/cs/algorithm/1985-naur.pdf (PDF)

Re: Automation Without Understanding

#23

Post by DaiPlusPlus »

taurath wrote: post #21
DaiPlusPlus wrote: post #15

Earlier quoted context omitted.

> it’s that we may stop producing people who know enough to notice when AI is confidently wrong. The running-joke is that a LinkedIn-lunatic AI booster, with a Nano Banana-generated profile-pic, will immediately slide into the chat to tell you that that this is already a solved problem: just spin up another agent to do the work to verify the first agent. Token-cost-be-damned. And we laugh and downvote them to oblivio…

How is the token-cost-be-damned part in the latter example? I do find that both porting and translation projects have a much higher signal given the ease of mapping to tokens, when there is a proven working source to refer to - the source itself provides the validation. In a new project, you don’t have that validation.

> How is the token-cost-be-damned part in the latter example?

It's there, but...

1. The project owner figured out a way to minimize token usage for agent claim verification tasks.

2. Verification agents used older and much cheaper models, including local models for the most trivial things.

3. They could afford it anyway; but I think it's an inevitability that the token-cost for a task will approach some limit for some quality threshold - concurrent with the dollar-cost-per-token shrinking over time as better hardware comes out.

> In a new project, you don’t have that validation.

I'm still trying to understand that part of the project's history, actually. Obviously the HTML5+WebGL+Emscripten+Etc entrypoint was a "new" project; one of the first things they did was build their own means of verification, I just don't know how that part worked-out in practice (besides the agents dogpiling in on TODO.md).

Re: Automation Without Understanding

#24

Post by notpachet »

mondrian wrote: post #20

It sounds ‘the singularity is near’ not so much because AI is reaching escape velocity forward, but because we’re systematically pushing humans back beyond the threshold where computers are legible?

I think about this a lot. From the POV of a medieval peasant, we likely reached the singularity a long time ago.

Re: Automation Without Understanding

#26

Post by ludwik »

notpachet wrote: post #24
mondrian wrote: post #20

It sounds ‘the singularity is near’ not so much because AI is reaching escape velocity forward, but because we’re systematically pushing humans back beyond the threshold where computers are legible?

I think about this a lot. From the POV of a medieval peasant, we likely reached the singularity a long time ago.

You both seem to be using a different definition of "singularity" from the one I'm familiar with. I've always understood it to mean a rapid feedback loop in which AI creates successive, increasingly capable generations of AI outside human control, rather than simply a level of technological advancement that would be incomprehensible to someone

Re: Automation Without Understanding

#27

Post by sandruso »

gedy wrote: post #18
DaiPlusPlus wrote: post #15

Earlier quoted context omitted.

> it’s that we may stop producing people who know enough to notice when AI is confidently wrong. The running-joke is that a LinkedIn-lunatic AI booster, with a Nano Banana-generated profile-pic, will immediately slide into the chat to tell you that that this is already a solved problem: just spin up another agent to do the work to verify the first agent. Token-cost-be-damned. And we laugh and downvote them to oblivio…

It's agents all the way down~!

"Determined" -> "Probabilistic"

Re: Automation Without Understanding

#29

Post by overgard »

I think the differentiation going forward is going to be if you're a person that understands things or if you're a person that delegates your understanding. I don't think there's going to be a lot of economic or social value to prompting an AI to do things you don't understand. First, anyone can do it, so why should someone give you money or credibility for that? Second, as fewer and fewer people bother to learn hard things, the scarcity of that knowledge will increase its value.

I guess even if I end up being wrong about that, I'd rather enrich my life and grow as a person by continuing to learn and do hard things rather than become an annoying cheerleader dependent on unreliable tools.

Re: Automation Without Understanding

#30

Post by bitwize »

Animats wrote: post #28

> "the essay makes the case for treating mathematical capacity as a strategic asset on a par with semiconductor capability." In other words, the mathematicians want more funding.

And the Chinese are still going to kick our ass.

Ancaps are in for a major cope and seethe when they are forced to admit that the communists won the cold war.

Post reply on HN