This paper makes me wonder the long lasting effects of the current media consumption patterns by the alpha-gen kids.
why just kids?
LLMs can get "brain rot"
31–40 of 310 posts
Re: LLMs can get "brain rot"
#32Earlier quoted context omitted.
Yes, I am concerned about the Computer Science profession >"“Brain Rot” for LLMs isn’t just a catchy metaphor—it reframes data curation as cognitive hygiene for AI" A metaphor is exactly what it is because not only do LLMs not possess human cognition, there's certainly no established science of thinking they're literally valid subjects for clinical psychological assessment. How does this stuff get published, this is…
It is a blog post, it was published as a Github page and on arXiv. I think it's intended as a catchy warning to people who are dumping every piece of the internet (and synthetic data based on it!) that there are repercussions.
arXiv is intended to host research papers, not a blog for researchers.
Letting researchers pollute it with blog-gunk is an abuse of the referral/vetting system for submitters.
Re: LLMs can get "brain rot"
#33Earlier quoted context omitted.
I don't think anyone is throwing raw datasets into LLMs and hoping for high quality weights anymore. Nowadays most of the datasets are filtered one way or another, and some of them highly curated even.
I doubt they are highly created you would need experts in every field to do so. Which gives me more performance anxiety for LLMs because one of the most curated fields should be code...
The major labs are hiring experts. They carefully build & curate synthetic data. The market for labelled non-synthetic data is currently ~$3B/year.
The idea that LLMs are just trained on a pile of raw Internet is severely outdated. (Not sure it was ever fully true, but it's far away from that by now).
Coding's one of the easier datasets to curate, because we have a number of ways to actually (somewhat) assess code quality. (Does it work? Does it come with a set of tests and pass it? Does it have stylistic integrity? How many issues get flagged by various analysis tools? Etc, etc)
Re: LLMs can get "brain rot"
#34Re: LLMs can get "brain rot"
#35Can someone explain this in laymen terms?
The study introduces the "LLM Brain Rot Hypothesis," asserting that large language models (LLMs) experience cognitive decline when continuously exposed to low-quality, engaging content, such as sensationalized social media posts. This decline, evident in diminished reasoning, long-context understanding, and ethical norms, highlights the critical need for careful data curation and quality control in LLM training. The findings suggest that standard mitigation strategies are insufficient, urging stakeholders to implement routine cognitive health assessments to maintain LLM effectiveness over time.
TL;DR from https://unrav.io/#view/8f20da5f8205c54b5802c2b623702569
Re: LLMs can get "brain rot"
#36Earlier quoted context omitted.
Attention is all you need.
In today's hyper saturated world, attention is everything: - consumer marketing - politics - venture fundraising When any system has a few power law winners, it makes sense to grab attention. Look at Trump and Musk and now Altman. They figured it out. MrBeast... Attention, even if negative, wedges you into the system and everyone's awareness. Your mousey quiet competitors aren't even seen or acknowledged. The attenti…
You’re absolutely right!
Re: LLMs can get "brain rot"
#37[dead]
Re: LLMs can get "brain rot"
#38Re: LLMs can get "brain rot"
#39Isn't this just garbage in garbage out with an attention grabbing title?
Attention is all you need.
In case anyone missed the reference: https://arxiv.org/abs/1706.03762
> (...) We propose a new simple network architecture, the Transformer, based solely on attention mechanisms, dispensing with recurrence and convolutions entirely.
Re: LLMs can get "brain rot"
#40If most of the content produced by younger generations is about skibidi toilet[1] and 67[2], isn't that what LLMs are going to be trained on? [1] https://en.wikipedia.org/wiki/Skibidi_Toilet [2] https://en.wikipedia.org/wiki/6-7_(meme)
only if the trends last long enough (which they rarely do!), skibidi is already old news according to some kids I know