Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

The Cognitive Dark Forest

ryelang.org

Post reply on HN
Print view View original

41–50 of 301 posts

Re: The Cognitive Dark Forest

#41

Post by [deleted] »

cbau wrote: post #18
king_phil wrote: post #3

Dark forest makes no sense to me. Why would a civilization eradicate another, spending huge amounts of resources (time, energy, material) when the universe has such an enormous scale that you cannot even get to each other in a timescale that makes much sense...

To quote from the book: > “First: Survival is the primary need of civilization. Second: Civilization continuously grows and expands, but the total matter in the universe remains constant. One more thing: To derive a basic picture of cosmic sociology from these two axioms, you need two other important concepts: chains of suspicion and the technological explosion.” 1. you can never know the intentions of other entities…

[deleted]

Re: The Cognitive Dark Forest

#42

Post by lstodd »

0x3f wrote: post #22
king_phil wrote: post #3

Dark forest makes no sense to me. Why would a civilization eradicate another, spending huge amounts of resources (time, energy, material) when the universe has such an enormous scale that you cannot even get to each other in a timescale that makes much sense...

Competition kills margins (profits, security, QoL), so the budget for eradication should be quite high, but generally speaking the idea is to destroy even fledgling upstarts, back when the cost is low.

And the idea does not make sense once you include intel being incomplete into the equation: what if the preemptive strike will not attain complete eradication?

You might or might not fatally cripple the opponent, but retaliation can do that too and you cannot be sure that it won't. It's MAD all over again.

Re: The Cognitive Dark Forest

#44

Post by lifeformed »

king_phil wrote: post #3

Dark forest makes no sense to me. Why would a civilization eradicate another, spending huge amounts of resources (time, energy, material) when the universe has such an enormous scale that you cannot even get to each other in a timescale that makes much sense...

"Timescales that makes sense" may be a human reasoning but not necessarily the reasoning of inconceivably advanced timeless civilizations. Sure, that planet of fish may be harmless now, but what about in a quick three billion years when they have FTL and AGI and Von Neuman probes and Dyson spheres and antimatter bombs? Easier to click the delete button now to save the trouble later.

Re: The Cognitive Dark Forest

#45

Post by 05 »

thomashop wrote: post #32
nate wrote: post #11

Earlier quoted context omitted.

Are you asking about the 3 body problem version of this? Spoiler alert: The folks doing the eradicating aren't spending much time/energy/anything on eradicating. It's one large missile through space. I think the gist is: sure, we humans can't conceive of getting to anyone else in the universe in any timescale, but if we can keep ourselves from destroying ourselves, we'll eventually figure it out. And we'll spread. An…

I don't think it's correct that we destroyed everything that isn't us. If we take all living beings, we have destroyed only a small percentage.

Not counting by total terrestrial vertebrate biomass.

Re: The Cognitive Dark Forest

#46

Post by alembic_fumes »

> This is the true horror of the cognitive dark forest: it doesn’t kill you. It lets you live and feeds on you. Your innovation becomes its capabilities. Your differentiation becomes its median.

Oh no, the terrible dystopia where anyone can benefit from anyone else's good ideas without restrictions! And without any gatekeepers, licensing agreements, copyright, and not even a lawyer in sight!

If this is the dark future that AI use brings for us, I say bring it. Even if it means that somebody gets filthy rich in the process, while making the rest of the humanity better off.

Re: The Cognitive Dark Forest

#47

Post by jmull »

cbau wrote: post #18
king_phil wrote: post #3

Dark forest makes no sense to me. Why would a civilization eradicate another, spending huge amounts of resources (time, energy, material) when the universe has such an enormous scale that you cannot even get to each other in a timescale that makes much sense...

To quote from the book: > “First: Survival is the primary need of civilization. Second: Civilization continuously grows and expands, but the total matter in the universe remains constant. One more thing: To derive a basic picture of cosmic sociology from these two axioms, you need two other important concepts: chains of suspicion and the technological explosion.” 1. you can never know the intentions of other entities…

I really liked those books, for all the creative ideas... it's fine that they don't all work, but the Dark Forest has to be among the worst of them. It was unfortunate it was highlighted.

Some rebuttals, going point by point...

1. you can know the intentions of other entities by observing and communicating with them.

2. technology explosions, like pretty much exponential phenomena, are self limiting. They necessarily consume the medium that makes them possible.

3. and 4. civilizations aren't necessarily sentient (ours certainly isn't) and don't have an agency, much less goals. Individuals have goals, and some may work for the survival of the civilization they belong to. But others may decide they can profit if they work with the aliens.

4. Multiple civilizations may well come into competition over resources, but that's more of an argument about why the forest would not be dark.

Practically speaking, a civilizations that opts to focus on massive, vastly expensive efforts to find and exterminate far flung civilizations because they may become a rival in the future may be easily outcompeted by civilizations that learn to communicate with and work with other civilizations they encounter.

Re: The Cognitive Dark Forest

#48

Post by [deleted] »

EA-3167 wrote: post #33
MattDamonSpace wrote: post #12

Earlier quoted context omitted.

Sure but the Forest point stands, whatever you can hide from the Forest becomes something that slows it down and allows you some, even if only brief, moat?

There’s a deeply flawed hidden assumption here, which is that the individual in question is the only possible source for the relevant information that the AI can harvest. In the real world that absurdly rare, original thought is rare because we’re in the mix with billions of others. Scientists who hold back publishing breakthroughs have not guaranteed that they will be the sole discoverer, just that someone else will…

[deleted]

Re: The Cognitive Dark Forest

#49

Post by [deleted] »

EA-3167 wrote: post #33
MattDamonSpace wrote: post #12

Earlier quoted context omitted.

Sure but the Forest point stands, whatever you can hide from the Forest becomes something that slows it down and allows you some, even if only brief, moat?

There’s a deeply flawed hidden assumption here, which is that the individual in question is the only possible source for the relevant information that the AI can harvest. In the real world that absurdly rare, original thought is rare because we’re in the mix with billions of others. Scientists who hold back publishing breakthroughs have not guaranteed that they will be the sole discoverer, just that someone else will…

[deleted]

Re: The Cognitive Dark Forest

#50

Post by chongli »

gobdovan wrote: post #37
pugio wrote: post #21

Thanks, this helped crystallize something for me: the play the AI labs are making is anti-fragile (in the Nassim Taleb sense): > The very act of resisting feeds what you resist and makes it less fragile to future resistance. At least along certain dimensions. I don't think the labs themselves are antifragile. Obviously we all know the labs are training on everything (so write/act the way you want future AIs to percei…

Instead of anti-fragility, I'd point you to the law of requisite variety instead. You'll notice that all AI improvements are insanely good for a week or two after launch. Then you'll see people stating that 'models got worse'. What happened in fact is that people adapted to the tool, but the tool didn't adapt anymore. We're using AI as variety resistant and adaptable tools, but we miss the fact that most deployments…

New models literally do get worse after launch, due to optimization. If you charted performance over time, it'd look like a sawtooth, with a regular performance drop during each optimization period.

That's the dirty secret with all of this stuff: "state of the art" models are unprofitable due to high cost of inference before optimization. After optimization they still perform okay, but way below SOTA. It's like a knife that's been sharpened until razor sharp, then dulled shortly after.

Post reply on HN