Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Asking 60 LLMs a set of 20 questions

benchmarks.llmonitor.com

Post reply on HN
Print view View original

161–170 of 352 posts

Re: Asking 60 LLMs a set of 20 questions

#161

Post by visarga »

YetAnotherNick wrote: post #57
NoraCodes wrote: post #52

Earlier quoted context omitted.

Given that people are already firing real human workers to replace them with worse but cheaper LLMs, I'd argue that we're not talking about a competing technology, but that the competition is simply not firing your workforce. And, as an obligate customer of many large companies, you should be in favor of that as well. Most companies already automate, poorly, a great deal of customer service work; let us hope they do…

How many humans in your office do you think could solve the questions with better success ratio than GPT-4? I would say less than 20%. If the primary complaint is the blues that GPT-4 wrote is not that great, I think it is definitely worth the hype, given that a year before people argued that AI can never pass turing test.

LLMs might be better than random people at random tasks, and certainly memorised more trivia. But I never heard of a LLM surpassing subject experts in their field. On top, even the simplest task is unreliable - say, summarisation or translation. There is almost zero autonomy or ability to recover from unexpected situations.

Re: Asking 60 LLMs a set of 20 questions

#163

Post by moonchrome »

redox99 wrote: post #69
mmcwilliams wrote: post #42

Earlier quoted context omitted.

That's kind of the issue with non-deterministic LLMs, isn't it?

LLMs are deterministic. However to make them more "creative", the outputs of the LLM can be sampled using something called temperature that adds some randomness. You can set the temperature to 0 and it will be deterministic. Having said that, GPT4 is not deterministic even at 0 temperature, either because of a bug in their implementation, or because of some load balancing among their alleged mixture of experts.

Even 3.5 turbo API is non deterministic with 0 temperature.

Re: Asking 60 LLMs a set of 20 questions

#164

Post by visarga »

sp332 wrote: post #106
IshKebab wrote: post #98

Earlier quoted context omitted.

But they're only firing humans in specific cases where LLMs can compete with them. LLMs aren't equally good at all tasks.

Well, no. https://www.psychiatrist.com/news/neda-suspends-ai-chatbot-f... https://www.cnn.com/2023/08/30/tech/gannett-ai-experiment-pa... If the AI is a lot cheaper than a human, then it can make business sense to replace the human even if the AI is not nearly as good.

What's with this zero sum mentality? "replace", "cheaper"...

We are updating our expectations very fast. We are fighting over a growing pie. Maybe the cost reduction from not having to pay human wages is much smaller than the productivity increase created by human assisted AI. Maybe it's not an issue to pay the humans. AI works better with human help for now, in fact it only works with humans, never capable of serious autonomy.

Re: Asking 60 LLMs a set of 20 questions

#166

Post by jhbadger »

jonwinstanley wrote: post #97
nojs wrote: post #12

> Sally (a girl) has 3 brothers. Each brother has 2 sisters. How many sisters does Sally have? The site reports every LLM as getting this wrong. But GPT4 seems to get it right for me: > Sally has 3 brothers. Since each brother has 2 sisters and Sally is one of those sisters, the other sister is the second sister for each brother. So, Sally has 1 sister.

I wouldn’t expect an LLM to get this right unless it had been trained on a solution. Am I wrong to think that? Are LLMs in the future going to be able to “think through” actual logic problems?

I don't think so. Consider the following conversation from UndiMix-v2-13B (a local LLM running on my Mac M1 via llama.cpp). It gets it wrong at first, but then gets it right with a little hint. I don't see how the hint could work if it was merely repeating an example it knew.

User: Sally (a girl) has 3 brothers. Each brother has 2 sisters. How many sisters does Sally have?

Llama: Hello there! I'd be happy to help you with that question. Let me think... If each of Sally's brothers has two sisters, then together they would have 6 sisters in total (2 x 3 = 6). So, Sally would have 6 sisters as well. Is that the answer you were looking for?

User: no. Think about it. Sally is a sister to the brothers

Llama: Ah, I see what you mean now! In this case, since Sally is one of the siblings, she only counts as one of her own sisters. Therefore, Sally has only one sister - herself. Does that make sense?

User: yes

Llama: Great! I hope that clears up any confusion. Please let me know if you have any other questions or need further assistance.

Re: Asking 60 LLMs a set of 20 questions

#167

Post by [deleted] »

appplication wrote: post #102
civilitty wrote: post #86

Earlier quoted context omitted.

It's certainly RLHFed. All of the logic puzzles I use for evaluation that used to fail months ago now pass no problem and I've even had a hard time modifying them to fail.

This is sort of a bummer because it’s not actually an improvement to the model, but just a patch job to artificially inflate performance. All it does is make true evaluation more difficult. Classic “you get what you measure”.

[deleted]

Re: Asking 60 LLMs a set of 20 questions

#168

Post by guerrilla »

ilaksh wrote: post #104
coldcode wrote: post #44

Despite the hype about LLMs, many of the answers are pretty terrible. The 12-bar blues progressions seem mostly clueless. The question is will any of these ever get significantly better with time, or are they mostly going to stagnate?

It's like most new technologies. In the beginning there are only a few instances that really stand out, and many with issues. I remember back in like 2011 or 2012 I wanted to use an SSD for a project in order to spend less time dealing with disk seeks. My internet research suggested that there were a number of potential problems with most brands, but that the Intel Extreme was reliable. So I specified that it must be…

> It's like most new technologies. In the beginning there are only a few instances that really stand out, and many with issues.

Except this isn't new. This is after throwing massive amounts of resources at it multiple decades after arrival.

Re: Asking 60 LLMs a set of 20 questions

#169

Post by gabereiser »

I was laughing so hard at the first example of “Argue for and against kubernetes in haiku”.

I couldn’t even get through reading 15 of them before the tears of laughter rolled from my cheeks.

“Containers organized, Services easy to deploy now, Updates who knows when.”

Updates who knows when… hahahaha.

Honestly this is pretty cool to see how each responds to the same input prompt.

Re: Asking 60 LLMs a set of 20 questions

#170

Post by jhbadger »

ftxbro wrote: post #100
jonwinstanley wrote: post #97

Earlier quoted context omitted.

I wouldn’t expect an LLM to get this right unless it had been trained on a solution. Am I wrong to think that? Are LLMs in the future going to be able to “think through” actual logic problems?

This is the whole point of the breakthrough related to the emergence of cognitive capabilities of LLMs. They are literally Markov chains. No one expected it to happen to this degree, but here we are.

People say that "they are literally Markov chains", but anyone who has looked at the code for LLMs knows that they are more complicated than that. I implemented Markov chains in BASIC in about ten lines of code in the 1980s on a 1 Mhz 64K Apple II after reading about the famous Mark V. Shaney hoax (https://en.wikipedia.org/wiki/Mark_V._Shaney). No neural nets or fancy GPUs required. It's one thing to stress that LLMs aren't magical or self-aware, but the fact is they are way more complicated than simple Markov chains.
Post reply on HN