Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Creativity fundamentally comes from memorization?

shwin.co

Post reply on HN
Print view View original

51–60 of 372 posts

Re: Creativity fundamentally comes from memorization?

#51

Post by somenameforme »

ArcaneMoose wrote: post #40
ArcaneMoose wrote: post #38

What's more is that memories are just a replaying of neuron connections activating in the brain - and when we are prompted by the world around us those connections will fire in response to the stimulus. Quite similar to how AI neural networks function - which is why I believe that AI can indeed be creative and create "new" ideas

I actually made a video diving deeper into this and comparing responses from people and ChatGPT for a creative thinking problem - https://youtu.be/l-9EUBbktqw

I think your hypothesis here (and probably the entire article as well) is strongly challenged by the 'progenitor argument.' Take humans at the dawn of humanity. Language did not even exist beyond what may have been crude sounds or gesturing and collective knowledge did not fall that far beyond 'poke him with the pointy side.' Somehow we went from that to putting a man on the Moon in what was essentially the blink of an eye.

Training an LLM on the entirety of knowledge at this dawn of humanity and, even if you give it literally infinite training time, it's never going to go anywhere. It's going to just continue making relatively simple recombinations of its training set until somebody gives it a new training set to remix. This remix-only nature is no different with modern knowledge, but simply extremely obfuscated because there's such a massive base of information, and nobody is aware of anything more than a minuscule fraction of it all.

---

As for the 'secret' of LLMs, I think it's largely that most language is extremely redundant. One thought or point naturally flows.... why do I complete the rest of this statement? You already know exactly what I'm going to say, right? And from that statement the rest of my argument will also mostly write itself. Yet we do write out the rest, which is kind of weird if you think about it. Anyhow the point is that by looking at language 'flow correlations' over huge samples, LLMs can reconstruct and remix arbitrarily long dialogue from even the shortest of initial inputs. And it usually sounds at least reasonable, except when it doesn't and we call it a hallucination, but it's quite a misnomer because the entire process is a hallucination.

Re: Creativity fundamentally comes from memorization?

#52

Post by ahazred8ta »

shw1n wrote: post #26
singleshot_ wrote: post #25

This claim makes little sense because it fails to distinguish between memory and memorization. I memorize almost nothing, but I remember the broad strokes of a lot of things. This allows me to be creative. In a way, memorization is a severe risk: if you memorized something before it changed, for example, your creativity may not mean much.

Hey thanks for reading! What I was trying to convey is that fundamentally learning is memorization, whether conscious "rote memorization" or more less-intentional committing to memory from doing an activity And that recognizing this allows us to speed up the process of learning fundamentals Which in turn enables creativity as most people see it

repetitio mater studiorum est -- repetition is the mother of learning

"Repetition is the mother of learning, the father of action, which makes it the architect of accomplishment."

Re: Creativity fundamentally comes from memorization?

#53

Post by keiferski »

dzink wrote: post #50

One would argue it’s the opposite as people with memory problems have less imprint of old and more creative new ideas.

I don’t think that’s true. People with memory problems more seem to retread the same foundational ideas repeatedly, whereas those with good memories recognize this repetition and then seek something novel.

Re: Creativity fundamentally comes from memorization?

#54

Post by CuriouslyC »

relaxing wrote: post #46

All very nice and handwavey, but then you see the user’s current venture is scammy deepfakes as a service, which is about as creatively bankrupt as it gets. Shame about the national stereotypes as well. There is plenty of creativity in Asian countries. Just bizarre assertions all around.

Historically eastern Asian cultures have placed duty to a whole host of things before oneself, and in many cases the old aphorism "the nail that sticks up gets hammered down" applies as well. Plus Japan and Korea have well established cultures of adherence to tradition and mastering simple, time tested things to a ridiculous degree rather than trying to innovate.

Not true of everyone, but if you compare a culture that values conformity and tradition to a country that values the freedom for the individual and trying new things, of course it's not going to measure up by western standards of creativity.

Re: Creativity fundamentally comes from memorization?

#56

Post by heenrik »

ahazred8ta wrote: post #49
tikhonj wrote: post #31

Earlier quoted context omitted.

That's true if you broaden the definition of "memorization" to cover all learning, but "learning is necessary for creativity" would not be a particularly interesting thesis. Expertise is the result of learning from past experience, both in developing an internal intuition for what you're doing and in having past patterns to draw upon. To the extent that experts have simple easily verbalizable heuristics, these are la…

This happened about 20 years ago when they were trying to automate recognizing cancer cells. They showed photos to experienced diagnosticians and asked 'What features do you look for?' They couldn't articulate what they were seeing.

The concept is called 'tacit knowledge'.

Re: Creativity fundamentally comes from memorization?

#58

Post by suzzer99 »

ambyra wrote: post #13

It’s not memorizing, it’s actually knowing and understanding the utility of different concepts. When you learn of a problem in a new field, you go through your bucket of tools and modify one to fit the new problem.

For me anyway, it's getting so familiar with something that your mind wanders and considers other possibilities.

Re: Creativity fundamentally comes from memorization?

#59

Post by ArcaneMoose »

somenameforme wrote: post #51
ArcaneMoose wrote: post #40

Earlier quoted context omitted.

I actually made a video diving deeper into this and comparing responses from people and ChatGPT for a creative thinking problem - https://youtu.be/l-9EUBbktqw

I think your hypothesis here (and probably the entire article as well) is strongly challenged by the 'progenitor argument.' Take humans at the dawn of humanity. Language did not even exist beyond what may have been crude sounds or gesturing and collective knowledge did not fall that far beyond 'poke him with the pointy side.' Somehow we went from that to putting a man on the Moon in what was essentially the blink of…

Interesting point - thanks for sharing! I think one big missing piece we have with AIs today is the ability for them to learn on the fly and reconfigure the weights. We are constantly bombarded with input and our neurons adjust accordingly. Current LLMs just use a snapshot. I would be really curious to see how online-first AI models could work, focusing on a constant input stream and iterating on weights. Also I wonder how much knowledge is baked into our DNA through evolution. I have a hunch that this is somewhat analogous to model architectures.

Btw - although I see evidence of LLMs creating "new ideas" through combinations of ideas, I am a bit mystified by their apparent reasoning issues. I wonder how that is different in nature from the memory-based approach. ARC-AGI benchmark has had me thinking about this for sure.

Re: Creativity fundamentally comes from memorization?

#60

Post by kqr »

I was the sort of person who did not believe in memorisation as a solution for anything. Then I tried getting really good at spaced repetition for a year (yes, it is a skill that needs to be trained for good results) and I've completely changed my mind.

Spaced repetition allows me to become proficient even in things I don't get the natural opportunity to practise daily, so that when the day comes and I need them, I have some level of knowledge already. This has happened to Kubernetes troubleshooting, statistics, PowerShell windows programming, and traffic engineering just in recent history.

I have yet to publish some of these, but I have examples from statistics:

https://two-wrongs.com/intuition-and-spaced-repetition.html

https://two-wrongs.com/inventing-fishers-exact-test.html

The latter is certainly creative in my book, although it does imply creativity within strict bounds.

Post reply on HN