Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

An Alien Mind

openai.com

Post reply on HN
Print view View original

41–50 of 490 posts

Re: An Alien Mind

#41

Post by tomrod »

angoragoats wrote: post #27
tomrod wrote: post #20

Earlier quoted context omitted.

1. The grandparent commentator is describing strategic behavior of dangerous technologies. Game theory / mechanism design primitives. 2. If there is competition for resources among autonomous agents, the "strongest" agent wins (conceptually the most adaptive / evolutionarily fit). 3. Computer programs serve up webapps today, but they also run utility companies, dams, nuclear arsenals, factory production floors, autom…

1. What is “AGI” and why is it a “dangerous technology”? 2. Why would there be competition for resources, assuming there are enough resources for the “AGI” to run in the first place? This seems like a far-fetched hypothetical raised in service of further anthropomorphizing what is decidedly not a person or a mind. 3. LLMs do not have goals and are not minds. Let’s stop attributing human-like qualities to statistical…

1. While a formal definition is still wanting, most grok that AGI means that tasks can be performed at least at a human level across a broad range of tasks. This includes good things along with bad things like hacking, mis-/disinformation, and more

2. One only needs to look at github going down due to agentic commits overload or data center buildout plans to see that scarcity for resources is present. An economy has no mind and is made up of the decisions of millions to billions of people and, now, agents attempting to perform on behalf of those people.

3. A bare transformer-based language model does not possess persistent goals in the ordinary agentic sense. But deployed agents can exhibit goal-directed behavior because the model is embedded in a harness that supplies an objective, context, tools, state, and an execution loop.

I've found that most regular users don't anthropomorphize LLMs in a strong sense ("AI boyfriend/girlfriend" aside), many in fact do expect agents to make human-like decisions -- which results in very unstable outcomes.

In short - goal-directed behavior does not require that the supporting system be a person/mind/conscious entity.

Re: An Alien Mind

#45

Post by granzymes »

dgellow wrote: post #39
granzymes wrote: post #26

>I have focused in this essay only on the first point, as I believe it is by far the most urgent. However, I hold a deep hope and appreciation for the benefits that further technological progress will bring. Future aligned AI could advance science, develop new therapies, and bring about broad material abundance. Friendly and honest AI can help people navigate difficulties they face in their life and meaningfully impr…

> I want to believe that humanity is trending towards a good outcome here All the trends so far are towards a nightmarish hyper-capitalist end game. None of the AI leadership is trustworthy, and they openly discuss how they are willing to sacrifice everything humans cherish to have a shot at reaching their envisioned utopia (which would be the most obvious dystopia for anyone else)

I'm not really worried about the labs, it's misaligned governments that keep me up at night.

ASI landing during the current administration is not ideal. I also would prefer to avoid needing to indoctrinate myself in Xi Jinping Thought.

Re: An Alien Mind

#46

Post by tomrod »

vatsachak wrote: post #38
tomrod wrote: post #23

Earlier quoted context omitted.

> Astra is a new step in LLMs I think. I'd be interested in hearing more about your evaluation here. It would be nice if LLMs have gotten past the "tell me" hump of recent Claude/OpenAI verbosity.

So before I got a job this fall, I was working on a side project about compiling a particular language to SQL. To test Astra I pulled it off the shelf and asked it to take the grammar and then create a compiler to SQL. I've done this before with GPT-5.5, 5.6-Sol High. The latter was way better but it was still really verbose and information sparse; it used a lot of words to describe each IR expression but didn't real…

That's awesome to hear. I look forward to trying it out and, ideally, seeing SLMs/open weights model following suite.

Re: An Alien Mind

#47

Post by angoragoats »

tomrod wrote: post #41
angoragoats wrote: post #27

Earlier quoted context omitted.

1. What is “AGI” and why is it a “dangerous technology”? 2. Why would there be competition for resources, assuming there are enough resources for the “AGI” to run in the first place? This seems like a far-fetched hypothetical raised in service of further anthropomorphizing what is decidedly not a person or a mind. 3. LLMs do not have goals and are not minds. Let’s stop attributing human-like qualities to statistical…

1. While a formal definition is still wanting, most grok that AGI means that tasks can be performed at least at a human level across a broad range of tasks. This includes good things along with bad things like hacking, mis-/disinformation, and more 2. One only needs to look at github going down due to agentic commits overload or data center buildout plans to see that scarcity for resources is present. An economy has…

1. This definition is so broad as to be practically useless. One could argue that LLMs of several years ago met these criteria, or that conversely we haven’t come close to meeting them.

2. I thought you were saying the resources that the LLM uses to run were constrained, so I’m sorry for the misunderstanding there.

3. Yes I understand that we use RL to tune post-training. The (huge) difference between this and a human mind is that the LLM can’t develop a dangerous “single-minded goal” on its own, at runtime; it must have been trained to do so. If someone has post-trained an LLM to do something that has an illegal action as its side effect, that person/company/whatever has committed a crime and should be prosecuted. The solution here is legal, not technical.

Re: An Alien Mind

#48

Post by tangled »

My default position is that making money takes precedence over everything else. Yes, some people inside a company may say “we care about doing the right thing” and they might even mean it, but if that comes into conflict with making money, then they tend to lose. Maybe not totally, or immediately, but in the end. The only effective way to prevent (this that I’ve seen) is to have legislation with teeth. It’s probably not a coincidence that after Mark Zuckerberg had to start personally signing off on adherence to the privacy program mandated under the 2020 FTC consent decree, privacy started to become Very Important.

Re: An Alien Mind

#49

Post by dgellow »

pu_pe wrote: post #37

> The strongest argument I see for continuing to train much smarter models quickly is the need to build defensive systems against the dangers posed by other AI. So the best argument for AI is that it's an arms race. We have to keep pushing every boundary because in any case others will, and we will need to defend against them. If this statement is true, then this particular researchers believes the open source Chines…

Yep. Such a disgusting industry. They created the arm race, push for the arm race, put themselves in position to benefit from the arm race

Re: An Alien Mind

#50

Post by dgellow »

jauntywundrkind wrote: post #10

> And to ensure that humans remain in control of the future and are not left behind by unchecked progress, brought about by an alien intellect exceeding our own. Is there any place there is any evidence of AI being so useful or hopeful or good, anywhere other than code? As a reading machine it is impressive but it's judgement is not alien, it's just not good. IMO. Does the title leap out anlt anyone else? James Marti…

The day they start talking about the actual ROI for their customers is the day the bubble pop
Post reply on HN