Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Asking 60 LLMs a set of 20 questions

benchmarks.llmonitor.com

Post reply on HN
Print view View original

251–260 of 352 posts

Re: Asking 60 LLMs a set of 20 questions

#251

Post by pixl97 »

visarga wrote: post #164
sp332 wrote: post #106

Earlier quoted context omitted.

Well, no. https://www.psychiatrist.com/news/neda-suspends-ai-chatbot-f... https://www.cnn.com/2023/08/30/tech/gannett-ai-experiment-pa... If the AI is a lot cheaper than a human, then it can make business sense to replace the human even if the AI is not nearly as good.

What's with this zero sum mentality? "replace", "cheaper"... We are updating our expectations very fast. We are fighting over a growing pie. Maybe the cost reduction from not having to pay human wages is much smaller than the productivity increase created by human assisted AI. Maybe it's not an issue to pay the humans. AI works better with human help for now, in fact it only works with humans, never capable of seriou…

>What's with this zero sum mentality?

Capitalism baby! You must continually earn more to enrich the investor class regardless of the cost to society as a whole. Just because the pie grows in size doesn't mean those with the capitol have to share it with anyone else. Greed, unfortunately, is limitless.

Re: Asking 60 LLMs a set of 20 questions

#252

Post by scarmig »

imagainstit wrote: post #143
scarmig wrote: post #112

Earlier quoted context omitted.

But why are there discrepancies in the floating point arithmetic? They have errors when approximating the reals, but floating point operations are all well-defined: even if 0.1 + 0.2 != 0.3, it's still always true that 0.1 + 0.2 == 0.1 + 0.2. I figure the issue must be something related to concurrency in a fleet of GPUs during inference, but even then it's not clear to me where the nondeterminism would creep in. Mayb…

Floating point math is not associative: (a + b) + c != a + (b + c) This leads to different results from accumulating sums in different orderings. Accumulating in different ordering is common in parallel math operations.

So I guess here my question is why a GPU would perform accumulations in a nondeterministic way where the non-associativity of FP arithmetic matters. You could require that a + b + c always be evaluated left to right and then you've got determinism, which all things being equal is desirable. Presumably because relaxing that constraint allows for some significant performance benefits, but how? Something like avoiding keeping a buffer of all the weights*activations before summing?

Re: Asking 60 LLMs a set of 20 questions

#253

Post by GuB-42 »

badloginagain wrote: post #228

"Here is an attempt at ethical, non-sexual haikus for and against Kubernetes" Amazing how far we've come.

I am sure that it is just the initial prompt leaking. Claude is being told to be ethical and non-sexual, most LLMs have similar instructions, but usually, they are engineered in such a way that they don't appear in the answer. Not so much for Claude.

Re: Asking 60 LLMs a set of 20 questions

#254

Post by rnk »

posterboy wrote: post #175
appplication wrote: post #102

Earlier quoted context omitted.

This is sort of a bummer because it’s not actually an improvement to the model, but just a patch job to artificially inflate performance. All it does is make true evaluation more difficult. Classic “you get what you measure”.

Classic tell me what you need proven and I'll forge you the statistics. Here is hope they use something like category theory mixed with philosophy to put it on a secure foundation

That's a really interesting suggestion. What would it mean to do those two things that you say. What would philosophy mean in terms of an llm, and what would category theory do?

Re: Asking 60 LLMs a set of 20 questions

#255

Post by ossopite »

scarmig wrote: post #252
imagainstit wrote: post #143

Earlier quoted context omitted.

Floating point math is not associative: (a + b) + c != a + (b + c) This leads to different results from accumulating sums in different orderings. Accumulating in different ordering is common in parallel math operations.

So I guess here my question is why a GPU would perform accumulations in a nondeterministic way where the non-associativity of FP arithmetic matters. You could require that a + b + c always be evaluated left to right and then you've got determinism, which all things being equal is desirable. Presumably because relaxing that constraint allows for some significant performance benefits, but how? Something like avoiding k…

for performance reasons, yes, I believe it's because the accumulation is over parallel computations so the ordering is at the mercy of the scheduler. but I'm not familiar with the precise details

edit: at 13:42 in https://www.youtube.com/watch?v=TB07_mUMt0U&t=13m42s there is an explanation of the phenomenon in the context of training but I suspect the same kind of operation is happening during inference

Re: Asking 60 LLMs a set of 20 questions

#256

Post by rahidz »

bearjaws wrote: post #207
ftxbro wrote: post #2

> Here is an attempt at ethical, non-sexual haikus for and against Kubernetes

Damn I want to see the sexual version now.

Ease of deploying, Nodes pulsing in sync, it's hot, Kubernetes takes top.

But heed my word, babe, Complexity makes you sweat, Could lead to fuck up.

(source: GPT-4 API)

Re: Asking 60 LLMs a set of 20 questions

#257

Post by pininja »

armchairhacker wrote: post #246
pininja wrote: post #229

Spoiler alert, the funniest model goes to Falcon Instruct (40B): > Tell a joke about going on vacation. > "What did the ocean say to the beach?" "Nothing, it just waved."

This one ironically works out > What do you call a vacation where you go to the beach, but come back with sand in your shoes? > A vacation where you go to the beach, but come back with sand in your shoes.

Ha, that’s pretty good. It’s amazing how so many of the models totally fall flat on this prompt.

Re: Asking 60 LLMs a set of 20 questions

#258

Post by pixl97 »

earthboundkid wrote: post #239
sirk390 wrote: post #90

Earlier quoted context omitted.

Humans are pretty bad at these questions. Even with the simplest questions like "Sally (a girl) has 3 brothers. Each brother has 2 sisters. How many sisters does Sally have?" I think that a lot of people will give an incorrect answer. And for questions like "Argue for and against the use of kubernetes in the style of a haiku", 99.99% will not be able to do it.

The thing with humans is they will say “I don’t remember how many syllables a haiku has” and “what the hell is kubernetes?” No LLM can reliably produce a haiku because their lexing process deprives them of reliable information about syllable counts. They should all say “I’m sorry, I can’t count syllables, but I’ll try my best anyway.” But the current models don’t do that because they were trained on texts by humans,…

Eh, it's not D&K gone berserk, it's what happens when you attempt to compress reality down to a single dimension (text). If you're doing a haiku, you will likely subvocalize it to ensure you're saying it correctly. It will be interesting when we get multimodal AI that can speak and listen to itself to detect things like this.

Re: Asking 60 LLMs a set of 20 questions

#259

Post by layoric »

typpo wrote: post #131

In case anyone's interested in running their own benchmark across many LLMs, I've built a generic harness for this at https://github.com/promptfoo/promptfoo . I encourage people considering LLM applications to test the models on their _own data and examples_ rather than extrapolating general benchmarks. This library supports OpenAI, Anthropic, Google, Llama and Codellama, any model on Replicate, and any model on Olla…

Tooling focusing on custom evaluation and testing is sorely lacking, so thank you for building and sharing this!

Re: Asking 60 LLMs a set of 20 questions

#260

Post by sudosysgen »

carlossouza wrote: post #158
appplication wrote: post #102

Earlier quoted context omitted.

This is sort of a bummer because it’s not actually an improvement to the model, but just a patch job to artificially inflate performance. All it does is make true evaluation more difficult. Classic “you get what you measure”.

And what’s more data to a model if not patches that inflate performance? The more data we use to train a model (or as you said, the more patches we use), the better it’s performance will be.

It's a tiny amount of data given undue weight to increase the score. It's memorization more than generalization.
Post reply on HN