Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

The Cognitive Dark Forest

ryelang.org

Post reply on HN
Print view View original

191–200 of 301 posts

Re: The Cognitive Dark Forest

#191

Post by __MatrixMan__ »

pillefitz wrote: post #181
__MatrixMan__ wrote: post #69

I think this only applies to a rather narrow set of ideas. I'm not really interested in pursuing ideas that stop being good if somebody gets there first. If I bothered to design it its because I wanted it to exist and if somebody makes it exist then I'm happy because then I get to use it. So what kind of things does this apply to? Likely, it's zero sum games, schemes to control other people, ways to be the first to c…

Do you think there's enough such opportunities that we'll all be able to pay our rents?

If we dispense with the waste that goes into fighting slices of the pie and focus instead on making a bigger pie? Absolutely.

If we try to double down on the zero sum games that we learned from our parents, maybe not.

Re: The Cognitive Dark Forest

#192

Post by JoachimS »

I've for a long time visioned AGI as something emergent from advertising agents competing about trying to extract as much "money" from the resource called "humans" as possible. Luring, coercing the resource by feeding it info, forcing it to follow instructions, threatening it, stealing info etc. The agent doesn't need to understand what money is, what a human is or that there really is a physical world.

The Dark Forest idea and the original post resonates well with this.

I few days ago I created a new repo for a new block cipher explicitly not to be used. And directly got several mails from bots promising that they (claiming to be humans) had looked at my repo and they could include it into their portfolio of especially good projects they also had vetted. Being part of this portfolio would almost guarantee that my repo and project would be used. If I only paid them some money first.

Creating the public repo meant sending a signal out into the digital world where agents are hunting for the human prey/resorce to extract value from.

The repo in question: https://github.com/secworks/tau256

Re: The Cognitive Dark Forest

#193

Post by itsgrimetime »

“These AI tools are garbage and can’t create anything worth creating”

“These AI tools are so powerful they can steal your ideas with nothing but a sentence”

I know that’s not exactly what OP is saying but the pretentiousness of the “we knew better” got to me a little bit. I think it’s a cool and unique analogy but I’m not as pessimistic.

Ideas have become so cheap to try/experiment with, more people are able to try 10x more or whatever, and that may keep increasing, I think there are way less hunters than hunted

Re: The Cognitive Dark Forest

#194

Post by chairmansteve »

mekoka wrote: post #182
jrowen wrote: post #155

It is also asymmetric. If you announce your presence, even if 4 out of 5 civs that notice you don’t annihilate you immediately (but they probably should), the fifth might. It’s just a probability game, with permadeath. So hiding is the most rational - the only - strategy of survival. This is a paranoid and cynical strategy that doesn't win out in the known history of life. What works is grow, expand, mingle, maintain…

A typical outlook from 21st century human thinking. We love to draw from our still rather actual history of fear and addictions to zero-sum games, to extrapolate the far advancement of other civilizations. As millennia go by, species can obviously only evolve technologically, while remaining psychologically, philosophically, and spiritually stuck.

"As millennia go by, species can obviously only evolve technologically, while remaining psychologically, philosophically, and spiritually stuck".

Interrsting take, and possibly (probably?) true of humans. But is it true of other (alien) sentient species?

Re: The Cognitive Dark Forest

#195

Post by chairmansteve »

stego-tech wrote: post #87

I’m still optimistic that this is cyclical in nature, and not an inevitable - or indefinite - outcome. Humanity has endured regular cycles of shared enlightenment (usually accompanying profound technological or societal revolutions) and dark forests of protectionism, and we always find a way to the other side. Sometimes these cycles last a century; sometimes, but a few years. Still, we always make it to the other sid…

You are right. In the dark forest, the predators must eventually die out, because they can't find prey.

Re: The Cognitive Dark Forest

#196

Post by bostik »

jrowen wrote: post #155

It is also asymmetric. If you announce your presence, even if 4 out of 5 civs that notice you don’t annihilate you immediately (but they probably should), the fifth might. It’s just a probability game, with permadeath. So hiding is the most rational - the only - strategy of survival. This is a paranoid and cynical strategy that doesn't win out in the known history of life. What works is grow, expand, mingle, maintain…

I always read the dark forest differently. Solution to the problem is not a game-theoretic "hide from the apex predators", but an even more nihilistic "remain hidden, expand and evolve into the apex predator".

Or in a more biblical sense: do unto others before they do unto you.

Re: The Cognitive Dark Forest

#197

Post by jandrewrogers »

bitmasher9 wrote: post #139
jandrewrogers wrote: post #68

Earlier quoted context omitted.

It isn't just about AI. Some R&D domains started disappearing from literature and the public internet a decade before the first LLMs. The incentives to go dark emerged even when the adversary was other humans. AI is just accelerating a trend that was already there. Some areas of frontier computer science research have largely been dark for decades. The strategy is to quietly do several years of iterated hardcore R&D.…

You get a time advantage for doing this strategy, but your talent will be pouched and your competitors will be able to catch up fairly quickly.

I used to think this but it only seems to be true for a shallow tech advantage, which isn’t this scenario. A sufficiently deep stack of compounded tech is robust against even aggressive talent poaching. The knowledge is embedded in the network, not the random individual.

We see this in jet engines, silicon fab, et al.

Re: The Cognitive Dark Forest

#199

Post by auggierose »

> And when we play for survival, we already lost, the result is known, we are just playing to postpone it.

Isn't that just life?

But in general, the ideas of the post are sound. IMO, the consequence is simple: we will become the forest. That is frightening, but not necessarily worse than unchecked capitalism in the dark forest.

Re: The Cognitive Dark Forest

#200

Post by asdff »

jandrewrogers wrote: post #197
bitmasher9 wrote: post #139

Earlier quoted context omitted.

You get a time advantage for doing this strategy, but your talent will be pouched and your competitors will be able to catch up fairly quickly.

I used to think this but it only seems to be true for a shallow tech advantage, which isn’t this scenario. A sufficiently deep stack of compounded tech is robust against even aggressive talent poaching. The knowledge is embedded in the network, not the random individual. We see this in jet engines, silicon fab, et al.

I mean, even north korea has figured out the nuclear bomb, the original greatest secret deep stack of compounded tech. Seems like anyone can figure out anything if they are hell bent on it on this earth. Engineers seem to be more fungible than people anticipate I guess, and no one really comes up with unprecedented unique ideas. The whole research process incentivizes incremental work on known concepts to justify receiving funding at all, since it is in high demand and short supply.
Post reply on HN