Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

The Cognitive Dark Forest

ryelang.org

Post reply on HN
Print view View original

131–140 of 301 posts

Re: The Cognitive Dark Forest

#131

Post by DaiPlusPlus »

middayc wrote: post #119
bsza wrote: post #111

The best analogy I can think of (quite similar to this one) is that the internet is low Earth orbit and AI is the Kessler syndrome. We abandon the place not to hide ourselves, but because it is saturated with garbage, and anything you try to put up there will only result in even more garbage being generated, without any positive effect. The ideal solution would be to remove the garbage, but right now we can't even de…

This is interesting. When I read if for the second time, trying to understand it - maybe even better match for the low orbit flying garbage would be "enshitification"? As the time goes on, more and more garbage is produced, and we have no clear way or specific motivated entity to start removing it so it just grows.

Enshittification specifically is when a product/service/platform gets worse from the user’s perspective because the platform vendor can directly profit from user-hostile design; for example, Google intentionally serves up bad results on the first search results page so the user clicks-through to the second page of results, resulting in more advert revenue to Google[1].

…whereas I feel what you’re describing is another Tragedy-of-the-Commons.

[1]: https://jackyan.com/blog/2023/09/google-search-is-worse-by-d...

Re: The Cognitive Dark Forest

#132

Post by rtrgrd »

jandrewrogers wrote: post #68
scottlawson wrote: post #4

The thesis that in the past it was safe to share ideas and projects because the execution was hard, and that now things have changed because of AI is an interesting AI, but I wonder if it is really true. It certainly seems true that for small projects and relatively narrow scoped things that AI can replicate them easily. I'm thinking specifically about blog posts where people share their first steps and simple progra…

It isn't just about AI. Some R&D domains started disappearing from literature and the public internet a decade before the first LLMs. The incentives to go dark emerged even when the adversary was other humans. AI is just accelerating a trend that was already there. Some areas of frontier computer science research have largely been dark for decades. The strategy is to quietly do several years of iterated hardcore R&D.…

Also similar to block withholding [1]

[1] https://bitcoin.stackexchange.com/questions/4943/what-is-a-b...

Re: The Cognitive Dark Forest

#133

Post by dwd »

As a separate analogy, and one related to physical products. I built a website for a guy many years ago who had patented a clamp for frameless glass panels that didn't require drilling the glass; primarily used for pool fencing.

The problem was as soon as he got the patent, it was available to view in countries where the cost to enforce his patent wasn't viable, and the market very quickly filled with cheap imitations. He straight out said at the time he regretted getting the patent.

Re: The Cognitive Dark Forest

#136

Post by chongli »

bonoboTP wrote: post #78
gobdovan wrote: post #61

Earlier quoted context omitted.

Is this insider info? The 'charted performance' caught my eye instantly. Couple things I find odd tho: why sawtooth? it would likely be square waves, as I'd imagine they roll down the cost-saving version quite fast per cohort. Also, aren't they unprofitable either way? Why would they do it for 'profitability'?

It's rumors based on vibes. There are attempts to track and quantify this with repeated model evaluations multiple times per day, this but no sawtooth pattern has emerged as far as I know.

I don't want to go too far down the conspiracy rabbit hole, but the vendors know everyone's prompts so it would be trivial for them to track the trackers and spoof the results. We already know that they substitute different models as a cost-saving measure, so substituting models to fool the repeated evaluations would be trivial.

We also already know that they actively seek out viral examples of poor performance on certain prompts (e.g. counting Rs in strawberry) and then monkey-patch them out with targeted training. How can we be sure they're not trying to spoof researchers who are tracking model performance? Heck, they might as well just call it "regression testing."

If their whole gig is an "emperor's new clothes" bubble situation, then we can expect them to try to uphold the masquerade as long as possible.

Re: The Cognitive Dark Forest

#137

Post by tayo42 »

Skyy93 wrote: post #71

This article makes no real sense to me. >You think of something new and express it - through a prompt, through code, through a product - it enters the system. Your novel idea becomes training data. The sheer act of thinking outside the box makes the box bigger. This was the same before, if you had a novel idea and make a product out of it others follow. Especially for LLMs, they are not (till now) learning on the fly…

>Especially for LLMs, they are not (till now) learning on the fly.

Was this just awkward phrasing or did something change and they learn after training?

Re: The Cognitive Dark Forest

#138

Post by chongli »

Barrin92 wrote: post #130
bsza wrote: post #111

The best analogy I can think of (quite similar to this one) is that the internet is low Earth orbit and AI is the Kessler syndrome. We abandon the place not to hide ourselves, but because it is saturated with garbage, and anything you try to put up there will only result in even more garbage being generated, without any positive effect. The ideal solution would be to remove the garbage, but right now we can't even de…

I don't buy the analogy. The problem with Kessler syndrome is that low earth orbit is physically crowded, you run into collisions. I don't care about the garbage. I don't care about the AI era. I've been writing code in Emacs for 20 years, I'll be writing code in Emacs in 20 years, every open source project I contribute to still looks the same because all these AI people, like the blockchain people do is just make ne…

I think by "internet" they mean search engine results pages. If you restrict yourself to short, common queries and only look at the top 10 results on the page, then the space really is very limited. If all those top 10s for common queries start to get crowded out with AI slop, then people are going to start abandoning search.

Re: The Cognitive Dark Forest

#139

Post by bitmasher9 »

jandrewrogers wrote: post #68
scottlawson wrote: post #4

The thesis that in the past it was safe to share ideas and projects because the execution was hard, and that now things have changed because of AI is an interesting AI, but I wonder if it is really true. It certainly seems true that for small projects and relatively narrow scoped things that AI can replicate them easily. I'm thinking specifically about blog posts where people share their first steps and simple progra…

It isn't just about AI. Some R&D domains started disappearing from literature and the public internet a decade before the first LLMs. The incentives to go dark emerged even when the adversary was other humans. AI is just accelerating a trend that was already there. Some areas of frontier computer science research have largely been dark for decades. The strategy is to quietly do several years of iterated hardcore R&D.…

You get a time advantage for doing this strategy, but your talent will be pouched and your competitors will be able to catch up fairly quickly.

Re: The Cognitive Dark Forest

#140

Post by mekoka »

Nevermark wrote: post #127
munificent wrote: post #126

Earlier quoted context omitted.

> This was the same before, if you had a novel idea and make a product out of it others follow. The article says: "Ideas are cheap - execution is hard" "Announcing, signaling your ideas offered much greater benefit than risk, because your value multiplied by connections, and execution was the moat you could stand behind." That's the key difference. It used to be much harder for a competitor to catch up to the state o…

Sharing any novel idea has never been so costly. I am not arguing against sharing. Sharing can be for the greater good. But as you note, things have changed. We could reasonably assume a genuinely significant good idea, set free, might go in the direction we shoved it for a minute. Or fade into inaccessibility. Not any more.

You seem to be agreeing, not arguing, with the person you're replying to.
Post reply on HN