Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

LLMs are not suitable for brainstorming

piaoyang0.wordpress.com

Post reply on HN
Print view View original

21–30 of 99 posts

Re: LLMs are not suitable for brainstorming

#21

Post by bawolff »

> What’s worse is when we ask topics that don’t have consensus currently, the LLMs won’t behave in a more creative and independent way (as we hoped), but more susceptible to issues like hallucination.

I mean, what is the difference between creativity and hallucination (honest question)?

---

Maybe this means AI would be better at the opposite - after you brainstorm creative out there ideas, AI can tell you if they have been tried in the past and what the consensus view is on why they failed, allowing you to adjust course.

Re: LLMs are not suitable for brainstorming

#22

Post by nequo »

dahart wrote: post #12

> The reason [LLMs are not a good tool to do truly effective brainstorming] is LLMs are trained to follow existing patterns in the human-produced corpus, and not natively taught to “brainstorm”. The problem with this argument is that people do the same thing, we’re not that great at brainstorming either. When we brainstorm in groups, we’re just bringing multiple points of view together. The more data LLMs are trained…

Setting the temperature to a higher value would emulate this, wouldn't it? Then the model would be more willing to deviate from the most likely next token.

Re: LLMs are not suitable for brainstorming

#23

Post by XorNot »

dahart wrote: post #12

> The reason [LLMs are not a good tool to do truly effective brainstorming] is LLMs are trained to follow existing patterns in the human-produced corpus, and not natively taught to “brainstorm”. The problem with this argument is that people do the same thing, we’re not that great at brainstorming either. When we brainstorm in groups, we’re just bringing multiple points of view together. The more data LLMs are trained…

> and have unbounded error when extrapolating outside their training examples

I would say, try reading a conspiracy theory forum - or simply wait for one to get posted on HN, and then revisit this particular conclusion.

I don't think this is a unique failure of LLMs, except insofar as LLMs have less of a physical grounding in reality then humans do: but we have people out there who have convinced themselves they can project forcefields, and take that all the way through to getting punched in the face by a martial artist when it turns out that, no, they can't.

Re: LLMs are not suitable for brainstorming

#25

Post by password_ »

dahart wrote: post #12

> The reason [LLMs are not a good tool to do truly effective brainstorming] is LLMs are trained to follow existing patterns in the human-produced corpus, and not natively taught to “brainstorm”. The problem with this argument is that people do the same thing, we’re not that great at brainstorming either. When we brainstorm in groups, we’re just bringing multiple points of view together. The more data LLMs are trained…

>people do the same thing

how do you know?

Re: LLMs are not suitable for brainstorming

#26

Post by firewolf34 »

resource_waste wrote: post #6

>However, here I would like to argue that (especially in cutting edge scenarios) LLMs are not a good tool to do truly effective brainstorming. Great title, they baited and switched. 95% of the time I don't need effective brainstorming, I need a bunch of ideas, let me pick the best, and move on. If its a real engineering problem, then I need >truly effective brainstorming.

Seriously, it's the same argument that people give for "ChatGPT can't give me good code, I don't know why", just rephrased. The deluge of "GPT is not useful for X" articles meant to bait the average critic, despite it being used en masse for "subproblems in X-space" already...

They're asking the wrong type of work from it. If you need some boilerplate or a transformation, it's going to give you a fantastic template to work with. If you need it, on the other hand, to engineer out a highly-specific and nuanced solution with an esoteric codebase to a complex problem, maybe not so much. The former is wide, the latter is narrow. It's going to take maybe a bit more breakdown of the scope into proper subproblems before you'll get a good answer; and that's something you can do yourself, or have an agent perform across multiple queries maybe (though I'll admit, more work needs to be done for the whole multi-agent workflows to be truly useful).

Re: LLMs are not suitable for brainstorming

#27

Post by blackle »

nequo wrote: post #22
dahart wrote: post #12

> The reason [LLMs are not a good tool to do truly effective brainstorming] is LLMs are trained to follow existing patterns in the human-produced corpus, and not natively taught to “brainstorm”. The problem with this argument is that people do the same thing, we’re not that great at brainstorming either. When we brainstorm in groups, we’re just bringing multiple points of view together. The more data LLMs are trained…

Setting the temperature to a higher value would emulate this, wouldn't it? Then the model would be more willing to deviate from the most likely next token.

Increasing temperature just makes more uncommon tokens more likely to appear. This can include both uncommon ideas and uncommon spelling and grammatical mistakes. It also won't make ideas that the LLM isn't capable of thinking appear, unless you crank the temperature way up and get lucky (like monkeys on typewriters lucky)

Re: LLMs are not suitable for brainstorming

#29

Post by trueismywork »

dahart wrote: post #12

> The reason [LLMs are not a good tool to do truly effective brainstorming] is LLMs are trained to follow existing patterns in the human-produced corpus, and not natively taught to “brainstorm”. The problem with this argument is that people do the same thing, we’re not that great at brainstorming either. When we brainstorm in groups, we’re just bringing multiple points of view together. The more data LLMs are trained…

We don't need more data, we need independent LLMs with slight randomness.

Re: LLMs are not suitable for brainstorming

#30

Post by little_name »

WhitneyLand wrote: post #24

Brainstorming startup ideas? How often is that a successful approach at all? I can’t think of a good case study off the top of my head.

I don't know a case study but I was personally told by 2 different founders who had 100MM+ exits that their startup ideas were formed (at least partially) from brainstorm sessions. One guy even pointed to me the library where they had the whiteboard session which led to the idea that they exited in the end.

The SPC also have a blog that proposes brainstorming from -1 to 1: https://blog.southparkcommons.com/how-to-go-from-minus-1-to-...

Post reply on HN