Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

The Cognitive Dark Forest

ryelang.org

Post reply on HN
Print view View original

61–70 of 301 posts

Re: The Cognitive Dark Forest

#61

Post by gobdovan »

chongli wrote: post #50
gobdovan wrote: post #37

Earlier quoted context omitted.

Instead of anti-fragility, I'd point you to the law of requisite variety instead. You'll notice that all AI improvements are insanely good for a week or two after launch. Then you'll see people stating that 'models got worse'. What happened in fact is that people adapted to the tool, but the tool didn't adapt anymore. We're using AI as variety resistant and adaptable tools, but we miss the fact that most deployments…

New models literally do get worse after launch, due to optimization. If you charted performance over time, it'd look like a sawtooth, with a regular performance drop during each optimization period. That's the dirty secret with all of this stuff: "state of the art" models are unprofitable due to high cost of inference before optimization. After optimization they still perform okay, but way below SOTA. It's like a kni…

Is this insider info? The 'charted performance' caught my eye instantly. Couple things I find odd tho: why sawtooth? it would likely be square waves, as I'd imagine they roll down the cost-saving version quite fast per cohort. Also, aren't they unprofitable either way? Why would they do it for 'profitability'?

Re: The Cognitive Dark Forest

#62

Post by nextos »

pugio wrote: post #21

Thanks, this helped crystallize something for me: the play the AI labs are making is anti-fragile (in the Nassim Taleb sense): > The very act of resisting feeds what you resist and makes it less fragile to future resistance. At least along certain dimensions. I don't think the labs themselves are antifragile. Obviously we all know the labs are training on everything (so write/act the way you want future AIs to percei…

You have a point but current LLM architectures in particular are very fragile to data poisoning [1,2].

[1] https://www.anthropic.com/research/small-samples-poison

[2] https://arxiv.org/abs/2510.07192

Re: The Cognitive Dark Forest

#63

Post by lstodd »

0x3f wrote: post #60
lstodd wrote: post #55

Earlier quoted context omitted.

Point is you cannot know if they are an upstart (whatever upstart means). It can be misinterpretation, it can be camoflage, it can be anything. But once you rain death you're better be prepared to be grateful for what you are about to receive back.

Depends on the context. We certainly knew nobody else had nukes.

That.. was the case for all of four years. And forgive me if I doubt certainity.

Re: The Cognitive Dark Forest

#64

Post by orbital-decay »

Some of that is rose-tinted glasses.

1. Sharing was never really safe, open source by default only became possible because of SaaS and rent-seeking behavior.

2. Early web (not internet) wasn't hyperconnected. With the advent of global-scale social media it was immediately obvious to many this will lead to monoculture and reduced diversity. What thought to be the information superhighway became the information superconductor with zero resistance, carrying infinite current. Also known as short circuit.

Re: The Cognitive Dark Forest

#67

Post by griffzhowl »

> The platform doesn’t need to bother with individual prompts - it just needs to see where the questions cluster. A map of where the world is moving.

This was insightful, but is it much different to the kind of data google and other search engines have had access to for a long time?

And while LLMs might have sped up the rate of code generation, the tech giants have always been able to set a team on reverse engineering whatever they feel like, though they also often just bought up the startup that was producing what they wanted. I guess I'm not seeing exactly where LLMs specifically are creating the dark forest, rather than the consolidated, centralized tech landscape itself

Re: The Cognitive Dark Forest

#68

Post by jandrewrogers »

scottlawson wrote: post #4

The thesis that in the past it was safe to share ideas and projects because the execution was hard, and that now things have changed because of AI is an interesting AI, but I wonder if it is really true. It certainly seems true that for small projects and relatively narrow scoped things that AI can replicate them easily. I'm thinking specifically about blog posts where people share their first steps and simple progra…

It isn't just about AI. Some R&D domains started disappearing from literature and the public internet a decade before the first LLMs. The incentives to go dark emerged even when the adversary was other humans. AI is just accelerating a trend that was already there. Some areas of frontier computer science research have largely been dark for decades.

The strategy is to quietly do several years of iterated hardcore R&D. The cumulative advances are such a step change when seen by would-be fast-followers that it obscures the insights that allowed individual advances to occur. As an exaggerated case, imagine if the public history of powered flight skipped from the Wright Brothers to the Boeing 737.

In practice, this strategy has a major failure mode that people overlook. The sharp discontinuity in capability means that almost nothing that exists in the market is prepared to integrate with it. This is a large impediment to adoption even if the technology is objectively incredible and the market will inevitably get on board.

In short, it looks a lot like being too early to market. This is surmountable with clever execution but with this strategy you've traded one problem for a different one.

Re: The Cognitive Dark Forest

#69

Post by __MatrixMan__ »

I think this only applies to a rather narrow set of ideas.

I'm not really interested in pursuing ideas that stop being good if somebody gets there first. If I bothered to design it its because I wanted it to exist and if somebody makes it exist then I'm happy because then I get to use it.

So what kind of things does this apply to? Likely, it's zero sum games, schemes to control other people, ways to be the first to create a new kind of artificial scarcity, opportunities to make a buck by ruining something that has been so far overlooked by other grifters. In other words: bad ideas.

If AI becomes a threat to those who habitually dwell in such spaces, great, screw em.

In the meantime, the rest of us can build things that we would be happy to be users of, safe in the knowledge that if somebody beats us to it, we'll happily be users of that thing too.

Re: The Cognitive Dark Forest

#70

Post by 0x3f »

lstodd wrote: post #63
0x3f wrote: post #60

Earlier quoted context omitted.

Depends on the context. We certainly knew nobody else had nukes.

That.. was the case for all of four years. And forgive me if I doubt certainity.

Four years is plenty of time to start launching. Also, MAD incentivizes disclosure. What would be the point of having secret nukes? Openly having them is the only way to stop the US using its nukes to stop your nuke program, in this scenario.
Post reply on HN