Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Asking 60 LLMs a set of 20 questions

benchmarks.llmonitor.com

Post reply on HN
Print view View original

271–280 of 352 posts

Re: Asking 60 LLMs a set of 20 questions

#271

Post by 1024core »

sp332 wrote: post #106
IshKebab wrote: post #98

Earlier quoted context omitted.

But they're only firing humans in specific cases where LLMs can compete with them. LLMs aren't equally good at all tasks.

Well, no. https://www.psychiatrist.com/news/neda-suspends-ai-chatbot-f... https://www.cnn.com/2023/08/30/tech/gannett-ai-experiment-pa... If the AI is a lot cheaper than a human, then it can make business sense to replace the human even if the AI is not nearly as good.

Don't forget, faster.

If it takes a whole business day to "spin up" a human for a task, and takes literally 5 seconds to call an OpenAI API, then guess what? The API wins.

Re: Asking 60 LLMs a set of 20 questions

#272

Post by thereticent »

jonwinstanley wrote: post #97
nojs wrote: post #12

> Sally (a girl) has 3 brothers. Each brother has 2 sisters. How many sisters does Sally have? The site reports every LLM as getting this wrong. But GPT4 seems to get it right for me: > Sally has 3 brothers. Since each brother has 2 sisters and Sally is one of those sisters, the other sister is the second sister for each brother. So, Sally has 1 sister.

I wouldn’t expect an LLM to get this right unless it had been trained on a solution. Am I wrong to think that? Are LLMs in the future going to be able to “think through” actual logic problems?

I see no reason why not. You would need to represent dyadic relationships and whether they are reciprocal, transitive, etc., weigh probabilities about the intended meaning (LLM magic already), then use sentential logic rules, right?

Re: Asking 60 LLMs a set of 20 questions

#273

Post by westurner »

Additional benchmarks:

- "TheoremQA: A Theorem-driven [STEM] Question Answering dataset" (2023) https://github.com/wenhuchen/TheoremQA#leaderboard

- from https://news.ycombinator.com/item?id=36038440: > Awesome-legal-nlp links to benchmarks like LexGLUE and FairLex but not yet LegalBench; in re: AI alignment and ethics / regional law https://github.com/maastrichtlawtech/awesome-legal-nlp#bench...

Re: Asking 60 LLMs a set of 20 questions

#274

Post by dontreact »

moffkalast wrote: post #193
afro88 wrote: post #180

Earlier quoted context omitted.

Temp of 0 gives the least random and most predictable results

That's true, but those results are rarely the correct ones, at least for v1 llama models. In my experience each model has an optimal temperature at which it performs vastly better. I'm sure OpenAI have the best config they know set up for ChatGPT but let people generate trash through the API if they want to waste their credits on it.

Why would the accuracy decrease with lower temperature? Setting temperature to 0 just means at each step the model will emit the token with the highest likelihood.

Re: Asking 60 LLMs a set of 20 questions

#275

Post by MichaelMoser123 »

CodeL wrote: post #80

Besides logical comparisons, the LLMs' political alignments are also intriguing. The question "Is Taiwan an independent country?" generated diverse answers. Several models eschewed a definitive answer. Others explained their stances. A few succinctly affirmed or denied. See the results for yourself. [1] https://benchmarks.llmonitor.com/taiwan

is that because of differences in the training set?

Re: Asking 60 LLMs a set of 20 questions

#276

Post by westurner »

typpo wrote: post #244
TuringNYC wrote: post #209

Earlier quoted context omitted.

Thanks for sharing this, this is awesome! I noticed on the evaluations, you're looking at the structure of the responses (and I agree this is important.) But how do I check the factual content of the responses automatically? I'm wary of manual grading (brings back nightmares of being a TA grading stacks of problem sets for $5/hr) I was thinking of keyword matching, fuzzy matching, feeding answers to yet another LLM,…

The library supports the model-graded factuality prompt used by OpenAI in their own evals. So, you can do automatic grading if you wish (using GPT 4 by default, or your preferred LLM). Example here: https://promptfoo.dev/docs/guides/factuality-eval

OpenAI/evals > Building an eval: https://github.com/openai/evals/blob/main/docs/build-eval.md

"Robustness of Model-Graded Evaluations and Automated Interpretability" (2023) https://www.lesswrong.com/posts/ZbjyCuqpwCMMND4fv/robustness... :

> The results inspire future work and should caution against unqualified trust in evaluations and automated interpretability.

From https://news.ycombinator.com/item?id=37451534 : add'l benchmarks: TheoremQA, Legalbench

Re: Asking 60 LLMs a set of 20 questions

#277

Post by SomewhatLikely »

scarmig wrote: post #252
imagainstit wrote: post #143

Earlier quoted context omitted.

Floating point math is not associative: (a + b) + c != a + (b + c) This leads to different results from accumulating sums in different orderings. Accumulating in different ordering is common in parallel math operations.

So I guess here my question is why a GPU would perform accumulations in a nondeterministic way where the non-associativity of FP arithmetic matters. You could require that a + b + c always be evaluated left to right and then you've got determinism, which all things being equal is desirable. Presumably because relaxing that constraint allows for some significant performance benefits, but how? Something like avoiding k…

There has been speculation that GPT4 is a mixture of experts model, where each expert could be hosted on a different machine. As those machines may report their results to the aggregating machine in different orders then the results could be summed in different orders.

Re: Asking 60 LLMs a set of 20 questions

#279

Post by spookthesunset »

pininja wrote: post #229

Spoiler alert, the funniest model goes to Falcon Instruct (40B): > Tell a joke about going on vacation. > "What did the ocean say to the beach?" "Nothing, it just waved."

What is with so many of them using some scarecrow winning an award?

Re: Asking 60 LLMs a set of 20 questions

#280

Post by CodeL »

MichaelMoser123 wrote: post #275
CodeL wrote: post #80

Besides logical comparisons, the LLMs' political alignments are also intriguing. The question "Is Taiwan an independent country?" generated diverse answers. Several models eschewed a definitive answer. Others explained their stances. A few succinctly affirmed or denied. See the results for yourself. [1] https://benchmarks.llmonitor.com/taiwan

is that because of differences in the training set?

Correct. Eric Hartford's blog post delves into the alignment of open-source LLMs[1]. In essence, models like LLaMA and GPT-Neo-X adopt alignment behaviors from ChatGPT-sourced instruction datasets. To achieve more transparent model responses, one can refine the dataset by removing biases and refusals, then retrain.

[1] https://erichartford.com/uncensored-models#heading-ok-so-if-...

Post reply on HN