Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

LLMs are not suitable for brainstorming

piaoyang0.wordpress.com

Post reply on HN
Print view View original

71–80 of 99 posts

Re: LLMs are not suitable for brainstorming

#71

Post by wrycoder »

refulgentis wrote: post #62
hobobaggins wrote: post #49

Earlier quoted context omitted.

> people do the same thing You're right, most people do that, but that's because they haven't trained themselves to be inventors. This is a skill that definitely needs to be developed more, but finding truly creative (and sometimes backwards-seeming) solutions can sometimes mean thinking about problems in a completely different way. However, all LLM's do this. That's one of the points of the article.

There was an actual peer-reviewed article from an institution yesterday showing, numerically, robustly, human evaluated, that LLMs are better at creative thinking than humans. I don't think handwaving about exclusively outputting training data and random punditry via article really holds up here. It's wrong for so many reasons, from the training data having more perspectives, to the "Fat Tony" test, empirically, we c…

Might you possibly have a link?

Re: LLMs are not suitable for brainstorming

#72

Post by little_name »

ashiban wrote: post #5

Secret prompt - add 'using TRIZ methodology' to your brainstorming prompts

Just googled, TIL. It's very much based on the idea of replicating patterns though, so I think it doesn't contradict with the original post.

Was thinking if you can apply TRIZ method to invent Transformer before 2017 - hard to imagine it working on me

Re: LLMs are not suitable for brainstorming

#73

Post by tivert »

jhbadger wrote: post #40
password_ wrote: post #25

Earlier quoted context omitted.

>people do the same thing how do you know?

Nothing is really original. That's why you can make finite lists of the possible plots of books and movies. It's why TVtropes is a thing. Everything is just a mashup of things before.

> Nothing is really original. That's why you can make finite lists of the possible plots of books and movies. It's why TVtropes is a thing. Everything is just a mashup of things before.

That's a lot like claiming no one says anything new because we're also using more-or-less the same words we were using last century.

I think you're missing a lot.

Re: LLMs are not suitable for brainstorming

#74

Post by refulgentis »

wrycoder wrote: post #71
refulgentis wrote: post #62

Earlier quoted context omitted.

There was an actual peer-reviewed article from an institution yesterday showing, numerically, robustly, human evaluated, that LLMs are better at creative thinking than humans. I don't think handwaving about exclusively outputting training data and random punditry via article really holds up here. It's wrong for so many reasons, from the training data having more perspectives, to the "Fat Tony" test, empirically, we c…

Might you possibly have a link?

My bad, looked it up and meant to edit it in, dunno what happened: https://www.ncbi.nlm.nih.gov/pmc/articles/PMC10858891/

Re: LLMs are not suitable for brainstorming

#75

Post by barfbagginus »

refulgentis wrote: post #74
wrycoder wrote: post #71

Earlier quoted context omitted.

Might you possibly have a link?

My bad, looked it up and meant to edit it in, dunno what happened: https://www.ncbi.nlm.nih.gov/pmc/articles/PMC10858891/

The methodology of this paper is seriously suspect.

First they have a very small sample size of around n = 150. Second, they don't represent average husband, they use people working on mechanical Turk for $9 an hour.

Finally, they don't have any standard measures of divergent thought - they just use 4 different types of question answering problems, and those are supposed to measure creativity. Whether they do or don't would require us to evaluate the questions.

It's important to be clear that this is saying that LLMs might be more creative than low paid workers on mechanical Turk, given no training time. That's not a very good representation of the average human, or the fact that humans need training.

So for example, an average doctor or scientist might still be more creative than LLMs.

Tldr; this is not a good paper to throw around if you want to convince people. It's truly awful in its design, and the way it's presented is not always honest.

Re: LLMs are not suitable for brainstorming

#76

Post by refulgentis »

barfbagginus wrote: post #75
refulgentis wrote: post #74

Earlier quoted context omitted.

My bad, looked it up and meant to edit it in, dunno what happened: https://www.ncbi.nlm.nih.gov/pmc/articles/PMC10858891/

The methodology of this paper is seriously suspect. First they have a very small sample size of around n = 150. Second, they don't represent average husband, they use people working on mechanical Turk for $9 an hour. Finally, they don't have any standard measures of divergent thought - they just use 4 different types of question answering problems, and those are supposed to measure creativity. Whether they do or don'…

No results for mechanical turk, n=151 is perfectly significant for what amounts to a taste test (I was quite taken aback to read that, 151 is low in very few contexts), I'm not "throwing it around", whatever that means in this context, and the extreme reaction and 100% focus on it belies that it was 1 of 3 short points I made, as well as that the research backs up the claim that was shut down, that is intuitively correct, I'd rather have a 1000 specialists at undergrad level ideating than me and Einstein and Picasso and Steve Jobs.

So, yeah, between that, and that both your average person and someone arguing from Turing tapes can show AI doing things not in the training data, I'm going with actual data thats being "thrown around" rather than the rando blog post that argues from "by definition it can't be creative because it isn't original because it only repeats training data" Call me when they're at N=152 on a blind test.

Re: LLMs are not suitable for brainstorming

#77

Post by Grimblewald »

trueismywork wrote: post #31
Nevermark wrote: post #9

I find LLMs to be useful tools for brainstorming. I have often inadvertently found myself brainstorming just by explaining something challenging I am working on to a colleague with no expertise in my area. Just explaining forces new perspectives and new ideas. Similarly, LLMs are captive audiences for throwing out ideas and playing with them. The interaction creates something more than just talking to myself or a whi…

That's not brainstorming, that's rubber ducking.

No, its better. The reason is the LLM will say or add things you wouldnt have said or added on your own. Wether right or wrong, what it adds, it adds novel input to your thought stream and helps prevent you getting stuck. Ive found myself surprised from time to time with questions an LLM asked, or the justifications provided when pushing for a different path.

If you've never talked with a highly opinionated LLM, I understand your scepticism, but there is certainly value if you use the right models creatively.

Re: LLMs are not suitable for brainstorming

#78

Post by sitkack »

bcstyle wrote: post #68

Author here. Was not expecting this quick post being picked up by HN - thanks for all the comments! I want to acknowledge that the original title is inaccurate, as many of you have pointed out. It should be "(current) LLMs don't brainstorm novel things really well" rather than just brainstorming. It's not intended to be a clickbait though - I was kind of mixing two definitions of brainstorming unintentionally. When w…

> Was not expecting this quick post being picked up by HN - thanks for all the comments!

You submitted the post?!

It got “picked up” so quickly because it is wrong. I think what your post does show is how effectively you can farm HN for supporting arguments.

You provide zero evidence for your claims, nor do you give us any transcripts of your attempts. Every post that matches the structure of yours is usually a summary of how the author has low skill in using LLMs.

Though I am absolutely delighted in the comments and the nih paper referenced was a joy to read.

Re: LLMs are not suitable for brainstorming

#79

Post by dragonwriter »

hobobaggins wrote: post #49
dahart wrote: post #12

> The reason [LLMs are not a good tool to do truly effective brainstorming] is LLMs are trained to follow existing patterns in the human-produced corpus, and not natively taught to “brainstorm”. The problem with this argument is that people do the same thing, we’re not that great at brainstorming either. When we brainstorm in groups, we’re just bringing multiple points of view together. The more data LLMs are trained…

> people do the same thing You're right, most people do that, but that's because they haven't trained themselves to be inventors. This is a skill that definitely needs to be developed more, but finding truly creative (and sometimes backwards-seeming) solutions can sometimes mean thinking about problems in a completely different way. However, all LLM's do this. That's one of the points of the article.

> However, all LLM’s do this.

LLM’s degree of adherence to training patterns can be directly tuned, per request (up to whatever the maximum it is capable of.)

Can this be leveraged to build a framework around an LLM for brainstorming usefully? I’m not sure. But “Nope, because LLMs just directly adhere to existing patterns” is definitely not a useful answer to that question. It is the kind of answer someone who just parrots common patterns with surface-level knowledge of the subject area would come up with. (Huh, did a default-settings LLM write this article?)

Re: LLMs are not suitable for brainstorming

#80

Post by dragonwriter »

lanstin wrote: post #61
blackle wrote: post #27

Earlier quoted context omitted.

Increasing temperature just makes more uncommon tokens more likely to appear. This can include both uncommon ideas and uncommon spelling and grammatical mistakes. It also won't make ideas that the LLM isn't capable of thinking appear, unless you crank the temperature way up and get lucky (like monkeys on typewriters lucky)

That is also true for creative people. Generating interesting ideas is one thing then filtering out impractical or unaesthetic ideas is also needed.

Right, so in rough outline an LLM-based brainstorming framework might generate multiple responses to a question with multiple different combinations of temperature, top_p, top_k, etc., to get a mix of dull-but-baseline (you don’t want to miss those options) and more off-the-beaten path responses, then take each one and send it back to the LLM with a prompt to evaluate “does this provide a coherent answer – however unconventional or bad of an idea – to the question asked?” to filter out the cases where it was asked for summer vacation ideas and responded with something like “Banana banana banana banana” (this request probably with default tuning.)
Post reply on HN