Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Automation Without Understanding

arxiv.org

Post reply on HN
Print view View original

41–50 of 64 posts

Re: Automation Without Understanding

#41

Post by Izkata »

ludwik wrote: post #26
notpachet wrote: post #24

Earlier quoted context omitted.

I think about this a lot. From the POV of a medieval peasant, we likely reached the singularity a long time ago.

You both seem to be using a different definition of "singularity" from the one I'm familiar with. I've always understood it to mean a rapid feedback loop in which AI creates successive, increasingly capable generations of AI outside human control, rather than simply a level of technological advancement that would be incomprehensible to someone

The singularity is when an AI can improve itself, which among the results include humans no longer being able to understand its code and exponential improvements so we can't catch up either.

It requires a level of self-driven intent LLMs aren't capable of on their own.

Re: Automation Without Understanding

#42

Post by asdff »

piloto_ciego wrote: post #40
overgard wrote: post #35

Earlier quoted context omitted.

Uh, unless you're living with your well-off parents in perpetuity or you already have FU money... good luck with that. I'm pretty sure my landlord isn't going to accept "money doesn't matter anymore because of AI!" Also do you really think billionaires are frothing at the mouth for this because it's going to make money less important?

I think the billionaire folks are pathological or opportunists, nothing really more. As for your landlord, I doubt that will be a problem either. You cannot make 40% - 80% of the country structurally unemployable with AI and eventually robots and keep this same economic system. It will be weird, it’s not going to be hugely fun, but I’m betting that we either get massive deflation, huge changes in land use policy, rev…

Things get sticky fast the longer you think on this problem and the fact leadership has always held little interest in equal distribution of the planet's spare resources vs crony corruption and nepotism. Hard to imagine a future that doesn't look like mad max outside the billionaire compounds protected by loitering robots and drones along with space based technology immune to any rebellion.

Re: Automation Without Understanding

#43

Post by TacticalCoder »

titzer wrote: post #4

AIs should be forced to show their work. Every tool they use, every program they generate and run in the background, and every logical inference. They should be forced to produce Lean or Rocq proofs or execution traces for all the computation they use. For facts, they should be able to produce sources. For any abstract reasoning, they should be able to break it down into explainable steps. Then, on top of that, they…

> AIs should be forced to show their work. Every tool they use, every program they generate and run in the background, and every logical inference.

And they should be 100% deterministic: same input (including a random seed / "temperature" if one wants), same output.

On a not totally dissimilar note I like the projects saying: "AI PRs aren't accepted but if you do something worthwhile, give us your prompts instead".

Re: Automation Without Understanding

#44

Post by advael »

ludwik wrote: post #26
notpachet wrote: post #24

Earlier quoted context omitted.

I think about this a lot. From the POV of a medieval peasant, we likely reached the singularity a long time ago.

You both seem to be using a different definition of "singularity" from the one I'm familiar with. I've always understood it to mean a rapid feedback loop in which AI creates successive, increasingly capable generations of AI outside human control, rather than simply a level of technological advancement that would be incomprehensible to someone

The reason the term "singularity" was used when that term was coined was in analogy to a black hole, which has a point on its radius called the event horizon where the gravitational pull is strong enough to stop light from escaping, thereby making it impossible to see beyond that horizon. "Technological singularity" itself here refers to the technological event which causes this, like a black hole is sometimes explained as a gravitational singularity

This is important to the framing of the idea: "the technological singularity" refers to a point in the future past which further developments can't be meaningfully foreseen or maybe even understood. This is often associated with particular paths people think will cause this, but not defined by those. As such, the prior comments are making the claim that you can get there not just by accelerating technological progress, but also by reducing the ability of everyone to comprehend what's happening, which I agree is unfortunately a plausible outcome in the current world

Re: Automation Without Understanding

#45

Post by sinuhe69 »

Extrapolating to other fields, we can say this development is the worst what one can do to mankind and society: pushing for AI but at the same time decimating the human capital that helped produce and understand it. What is the point of producing countless mathematical proofs or code that one cannot verify nor understand? It’s like having a machine to produce endless things that no one needs or will use. Just because the machine can bring profits for the (initial) investors?! That is the rottenest form of capitalism imaginable.

Re: Automation Without Understanding

#46

Post by CGamesPlay »

lioeters wrote: post #12
wwweston wrote: post #8

See also Bill Thurston’s classic Math Overflow answer to a student wondering where they fit compared to a Gauss or Euler: https://mathoverflow.net/questions/43690/whats-a-mathematici... “The product of mathematics is clarity and understanding. Not theorems, by themselves. [Their importance is not just in their specific statements], but their role in challenging our understanding, presenting challenges that led to mat…

Curious to see if that can map to what's happening in the software industry/community. > The product of software engineering (or computer science) is clarity and understanding. Not programs, by themselves. Their importance is not just in their specific statements (lines of code in a specific language), but their role in challenging our understanding, presenting challenges that led to computational (?) developments th…

> The product of software engineering (or computer science) is clarity and understanding.

Hard disagree. Software engineering is about satisfying requirements while optimizing costs. Computer science is the part where clarity and understanding are the goal.

Re: Automation Without Understanding

#47

Post by supriyo-biswas »

browski wrote: post #6

Yeah how many of us know how to build an ICE engine or shoes of any meaningful quality We set upon end of human craftsmanship decades ago Math is probably the easiest to reclaim given its right in front our faces going about daily life. The syntax of math is not that important; real world quantification the syntax is meant to represent will still exist. Our biochemistry implicitly operates on senses of enough food an…

Arguably, the loss of specialized knowledge is what is holding countries like the US back in domains such as shipbuilding[1], rare-earth refining, and semiconductors.

Losing that knowledge puts the losing community in a bad position, and those communities who maintain this information somehow will end up being able to preserve their ability to reason and improve things. Though I don’t expect our current crop of US leaders and business owners to realize that.

[1] https://www.gao.gov/products/gao-25-108136

Re: Automation Without Understanding

#48

Post by red75prime »

hackernewds wrote: post #31
titzer wrote: post #4

AIs should be forced to show their work. Every tool they use, every program they generate and run in the background, and every logical inference. They should be forced to produce Lean or Rocq proofs or execution traces for all the computation they use. For facts, they should be able to produce sources. For any abstract reasoning, they should be able to break it down into explainable steps. Then, on top of that, they…

why?

How else would you trust a tangled mess of neurons to produce something that works? Artificial neurons, of course, hehe.

Re: Automation Without Understanding

#49

Post by protocolture »

DaiPlusPlus wrote: post #15
sachaa wrote: post #9

What worries me isn’t AI replacing experts, it’s that we may stop producing people who know enough to notice when AI is confidently wrong.

> it’s that we may stop producing people who know enough to notice when AI is confidently wrong. The running-joke is that a LinkedIn-lunatic AI booster, with a Nano Banana-generated profile-pic, will immediately slide into the chat to tell you that that this is already a solved problem: just spin up another agent to do the work to verify the first agent. Token-cost-be-damned. And we laugh and downvote them to oblivio…

The one I saw that curled my toes was an LLM that would take the input of a text field and turn it into an entry into a workplace health and safety log, and attribute the results to a human.

It only needs to be wrong once to cause a legal shitstorm. You also wont be getting a chance to merge in a change to roll in a second agent for consistency checks from prison.

Re: Automation Without Understanding

#50

Post by evolve-maz »

ricardobeat wrote: post #39
solid_fuel wrote: post #34

Earlier quoted context omitted.

> any time any agent confidently asserts something then it has to provide associated evidence And this is enforced by... another LLM? Seems like it would work alright until something is asserted implicitly and not categorized as an assertion.

It does happen. But this is why the “ralph loop” works: over dozens (or hundreds) of iterations, eventually every regression, implicit assumption, fake passing test, is noticed and fixed by another agent. The code slowly but continuously converges to a better state. I’m surprised myself, but haven’t seen it fall into chaos or degradation so far.

I haven't seen a code base which hasn't fallen into degradation from that loop.

From personal experience, any "correction" leads to an LLM writing more code, or stripping apart interfaces and mixing logic poorly, or adding some hidden control flow just to catch that edge case rather than the general case for that category of input.

Post reply on HN