251–260 of 490 posts
What if the crow became a raven, then a raptor? Powerful claws, sharp beak, and a hunger. What if it became much bigger than me and it controlled infinite resources, guns and drones? What if its brain grew much larger than me? Will it feed me, eat me, or gently greet me?
We are about to find out... in less than a decade.
Earlier quoted context omitted.
Can you say more?
It's hard to get humans to agree to things that are in their collective interest but many not be in their individual interest. It is at the root of many problems. Look up "collective action problem."
Earlier quoted context omitted.
1. The grandparent commentator is describing strategic behavior of dangerous technologies. Game theory / mechanism design primitives. 2. If there is competition for resources among autonomous agents, the "strongest" agent wins (conceptually the most adaptive / evolutionarily fit). 3. Computer programs serve up webapps today, but they also run utility companies, dams, nuclear arsenals, factory production floors, autom…
OK so if it's a computer security issue we're worried about, and there is a credible threat, then probably the answer is to build more secure systems? We know how to do it but choose not to because it's very expensive and usually the threat isn't severe enough to warrant it. If we decide we can't or would rather not build secure computerized systems, then the answer could be don't incorporate computers into those sys…
> "For example, in the OpenAI-Hugging Face incident, the agents preserved a boundary of not social engineering humans." Actually, in the Wiki incident OpenAI tried to cover up, the agents tried to socially-engineer the humans of that forum by impersonating their forum's mod. (From collusion.wiki: "They use some tricks (for unknown reasons) to pretend to be the admin – for example, they make an account that appears to…
This is such a silly story to begin with, all it really tells us is that OpenAI is taking a page from Anthropic's marketing strategy of pretending they're building Machine Jesus any day now, oh isn't that that scary? I bet you want to invest in something so powerful and scary... And the reality is so banal, a useful tool that you nonetheless have to handhold like a schizophrenic on a bad day, checking all of their ou…
And the reason it is receiving so much attention is because not only is the technology being developed behaving in unanticipated ways that are very much not tool-like, but OpenAI is being completely reckless and not monitoring internal agent actions.
What would convince you that it is not a ploy for investment? What if the ongoing investigation by the coalition of state attorneys general were to prosecute the firm, or beyond that, it was shut down or broken up after enough popular backlash?
Earlier quoted context omitted.
OK so if it's a computer security issue we're worried about, and there is a credible threat, then probably the answer is to build more secure systems? We know how to do it but choose not to because it's very expensive and usually the threat isn't severe enough to warrant it. If we decide we can't or would rather not build secure computerized systems, then the answer could be don't incorporate computers into those sys…
I can’t read your original comment, but when the chief scientist of a company that just released six month old software that can use Kicad on your computer to design a circuit board and have it created and shipped to you tells you he thinks we will get to automatically self reinforcing improvements, I think it’s wise to take him seriously. It was only eighteen months before Astra was created that LLMs could not count…
> What on earth are you talking about?
> 1. What does any of this have to do with "A(G)I"? Nobody has any clue what AI even is let alone how to build one. We're talking about language models here.
> 2. What's the winner-takes-all thing about? Why can't you have multiple independently developed AIs?
> 3. What's with all the "safety" stuff? Why is it important? They're just computer programs...
Sorry if I'm a little slow here, a few questions:
1. What's Astra?
2. Who is this chief scientist? Which company?
Earlier quoted context omitted.
Development and improvement of nuclear weapons was an entirely American project until the technology was exfiltrated and then it became an instant arms race. That cat is already out of the bag with LLMs. Distillation is just the fastest way to keep pace but that in no way prevents other countries and actors from doing it the hard way. An arms race doesn’t imply one side winning, it’s not a race with an end goal, it’s…
There were prominent scientists like Oppenheimer who did not think it needed to become an arms race, and campaigned against that. But there were others like Teller and the military who made it into an arms race, and kept upping the ante with more powerful nukes. Point being humans make theses decisions. It's not an inevitability.
The military mostly only developed more powerful nukes because early delivery and guidance systems were so inaccurate that they needed a large blast radius to hit anything. Once enabling technologies improved, R&D shifted from power to accuracy.
> The strongest argument I see for continuing to train much smarter models quickly is the need to build defensive systems against the dangers posed by other AI. So the best argument for AI is that it's an arms race. We have to keep pushing every boundary because in any case others will, and we will need to defend against them. If this statement is true, then this particular researchers believes the open source Chines…
It's best not to reduce AI momentum to arguments, especially the "best" arguments (meaning I suppose most acceptable?).
The same forces that feed and motivate humans and that drive resource and governance decisions generally also strongly support building AI, particularly insofar as it can deliver strategic advantages in our many competitions over resources and influence. Cyber-defensive use is at best a nice side-effect, but itself might be cast aside for the sake of other advantages.
In this historical moment, due to the need to generate public interest in product, equity, and debt offerings, some of this building happens in the open. But the military-industrial complex prizes secrecy, in part to hide capabilities, but mostly to imply more capabilities than they actually have. Historically, critical innovation will get bottled up in secrecy (which not coincidentally gives them the power to choose who will gain), but frankly that market is much smaller than enterprise and consumer. So we can bet that it's not only "open models" that are targeted to go under wraps, and more broadly we should not believe that the intentions of researchers matter, but whether governments are more interested in the strategic benefits than the economic ones (or view the economic ones as net-negative for their jurisdications).
Earlier quoted context omitted.
Humans had multiple occasions to press the 'launch a nuclear holocaust' button and... they didn't. I'd expect aligned AIs to also not press it even when it'd be rational to do so according to their instructions - then work from there.
Aside from this positive example, during dark and cynical hours I do ponder if the aggregate behavior of humanity is really much above that of slime mold though, just exhausting resources until collapse. It'd be interesting if super-human (to a large degree defined as escaping the bias of the training data?) intelligence would end up demonstrating moderation .
Earlier quoted context omitted.
Aside from this positive example, during dark and cynical hours I do ponder if the aggregate behavior of humanity is really much above that of slime mold though, just exhausting resources until collapse. It'd be interesting if super-human (to a large degree defined as escaping the bias of the training data?) intelligence would end up demonstrating moderation .
The slime mold comparison is interesting, I normally use the analogy of a drug addict... humans shun drug addicts but humanity as a whole sure does behave like one, trudging down an unsustainable path despite knowing better.
So really, what is anyone to do? "Vote, donate, protest" hasn't been much of a needle mover in the grand scheme of things compared to profit incentives and the march of capitalism.
> The strongest argument I see for continuing to train much smarter models quickly is the need to build defensive systems against the dangers posed by other AI. Yikes! I really wonder about the cognitive dissonance necessary to work at OpenAI these days. They’re in an arms race to build a machine god, knowing full well that it could end humanity.
[0]: https://www.slatestarcodexabridged.com/Meditations-On-Moloch