Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Asking 60 LLMs a set of 20 questions

benchmarks.llmonitor.com

Post reply on HN
Print view View original

261–270 of 352 posts

Re: Asking 60 LLMs a set of 20 questions

#261

Post by seabass-labrax »

DylanDmitri wrote: post #96
maweaver wrote: post #58

Earlier quoted context omitted.

By "Humans", I assume you mean something like "adult humans, well-educated in the relevant fields". Otherwise, most of these responses look like they would easily beat most humans.

I think most high-school educated adults, with the ability to make a couple web searches, would do fine on all these questions. It would take the humans minutes instead of seconds because they don't have the internet memorized. Me, Kubernetes Haikus, time taken 84 seconds: ---------- Kubernetes rules With its smooth orchestration You can reach web scale ---------- Kubernetes sucks Lost in endless YAML hell Why is it…

I think you're spot on here. Yes, if one's trying to compare human and GPT intelligence, then you have to define what counts as memorisation and what counts as reasoning. But what most people outside of academia are trying to do is work out whether a GPT can effectively replace a human in some time-consuming task, and to be able to do so without access to the internet is rarely an important factor.

Re: Asking 60 LLMs a set of 20 questions

#262

Post by seabass-labrax »

insulanus wrote: post #121
antman wrote: post #115

I have seen numerous posts of llm q&a and by the time people try to replicate them gpt4 is fixed. It either means that OpenAI is actively monitoring the Internet and fixes them or the Internet is actively conspiring to present falsified results for gpt4 to discredit OpenAI

It would be nice if the organizations would publish a hash of the code and the trained dataset.

You aren't able to get access to the 'Open'AI dataset though, are you? Agreed, it would be an excellent addition for comparing source-available models, but that doesn't help with the accusations of OpenAI's foul play nor of the existence of an anti-OpenAI conspiracy.

Re: Asking 60 LLMs a set of 20 questions

#263

Post by ComputerGuru »

dr_kiszonka wrote: post #116
ComputerGuru wrote: post #88

Earlier quoted context omitted.

Clever use of the dev tools console. I think you’re being a little uncharitable, the prompt said “smiley” and not “smiling emoji” and “smiley” was once upon a time used for the whole slew of emoji. With that in mind, several others also pass.

Good point, but you may be too charitable. At least according to Wikipedia, a smiley is supposed to be smiling. https://en.wikipedia.org/wiki/Smiley

I don’t know about Wikipedia but I’m speaking from lived historical experience.

Here’s a throwback for you: https://www.phpbb.com/customise/db/styles/smilies-13

(All “emoji” images were just called smilies in a lot of software.)

Re: Asking 60 LLMs a set of 20 questions

#264

Post by MichaelMoser123 »

nojs wrote: post #12

> Sally (a girl) has 3 brothers. Each brother has 2 sisters. How many sisters does Sally have? The site reports every LLM as getting this wrong. But GPT4 seems to get it right for me: > Sally has 3 brothers. Since each brother has 2 sisters and Sally is one of those sisters, the other sister is the second sister for each brother. So, Sally has 1 sister.

google bard also gave the correct answer, even without adding 'lets think step by step'.

Re: Asking 60 LLMs a set of 20 questions

#266

Post by NoraCodes »

visarga wrote: post #160
NoraCodes wrote: post #52

Earlier quoted context omitted.

Given that people are already firing real human workers to replace them with worse but cheaper LLMs, I'd argue that we're not talking about a competing technology, but that the competition is simply not firing your workforce. And, as an obligate customer of many large companies, you should be in favor of that as well. Most companies already automate, poorly, a great deal of customer service work; let us hope they do…

> Given that people are already firing real human workers to replace them with worse but cheaper LLMs That's impossible, LLMs are not that good. They might be firing people and crashing service quality.

Yes, correct.

Re: Asking 60 LLMs a set of 20 questions

#267

Post by 1024core »

jhbadger wrote: post #166
jonwinstanley wrote: post #97

Earlier quoted context omitted.

I wouldn’t expect an LLM to get this right unless it had been trained on a solution. Am I wrong to think that? Are LLMs in the future going to be able to “think through” actual logic problems?

I don't think so. Consider the following conversation from UndiMix-v2-13B (a local LLM running on my Mac M1 via llama.cpp). It gets it wrong at first, but then gets it right with a little hint. I don't see how the hint could work if it was merely repeating an example it knew. User: Sally (a girl) has 3 brothers. Each brother has 2 sisters. How many sisters does Sally have? Llama: Hello there! I'd be happy to help you…

Llama: ... Therefore, Sally has only one sister - herself. Does that make sense?

User: yes

----

Um... no!

Re: Asking 60 LLMs a set of 20 questions

#268

Post by 1024core »

LAC-Tech wrote: post #171

Only tried chatGPT 3.5, but my god does it waffle on. Everything I ask ends with a paragraph saying "It's important to remember that..." like an after-school special from a 90s show. It can never just give you code, it has to say "Sure!, to {paraphase your question}, open a terminal...". It's interesting to see 20th century sci-fi depictions of this kind of AI/Search is being short and to the point. I guess they can'…

Try instructions like this: https://twitter.com/nisten/status/1696229059183730833

(link posted above, not mine).

Re: Asking 60 LLMs a set of 20 questions

#269

Post by 1024core »

caturopath wrote: post #224
LAC-Tech wrote: post #171

Only tried chatGPT 3.5, but my god does it waffle on. Everything I ask ends with a paragraph saying "It's important to remember that..." like an after-school special from a 90s show. It can never just give you code, it has to say "Sure!, to {paraphase your question}, open a terminal...". It's interesting to see 20th century sci-fi depictions of this kind of AI/Search is being short and to the point. I guess they can'…

Yeah, I have tried a number of instructions to try to keep ChatGPT from blabbering and from sounding like a PR person. I haven't found the perfect incantation yet. > It's interesting to see 20th century sci-fi depictions of this kind of AI/Search is being short and to the point. Sci-fi told us that the AI would be so logical that you could just say a paradox aloud and it would blow up. What we got is something that c…

https://twitter.com/nisten/status/1696229059183730833 ?

Re: Asking 60 LLMs a set of 20 questions

#270

Post by westurner »

typpo wrote: post #131

In case anyone's interested in running their own benchmark across many LLMs, I've built a generic harness for this at https://github.com/promptfoo/promptfoo . I encourage people considering LLM applications to test the models on their _own data and examples_ rather than extrapolating general benchmarks. This library supports OpenAI, Anthropic, Google, Llama and Codellama, any model on Replicate, and any model on Olla…

ChainForge has similar functionality for comparing : https://github.com/ianarawjo/ChainForge

LocalAI creates a GPT-compatible HTTP API for local LLMs: https://github.com/go-skynet/LocalAI

Is it necessary to have an HTTP API for each model in a comparative study?

Post reply on HN