The first AI company to teach its LLM that it can’t do haiku because it can’t count syllables should get a special prize for not falling in love with their own bullshit.
Asking 60 LLMs a set of 20 questions
241–250 of 352 posts
Re: Asking 60 LLMs a set of 20 questions
#242Spoiler alert, the funniest model goes to Falcon Instruct (40B): > Tell a joke about going on vacation. > "What did the ocean say to the beach?" "Nothing, it just waved."
The confetti has truly left the cannon.
Re: Asking 60 LLMs a set of 20 questions
#243The first AI company to teach its LLM that it can’t do haiku because it can’t count syllables should get a special prize for not falling in love with their own bullshit.
[dead]
Re: Asking 60 LLMs a set of 20 questions
#244In case anyone's interested in running their own benchmark across many LLMs, I've built a generic harness for this at https://github.com/promptfoo/promptfoo . I encourage people considering LLM applications to test the models on their _own data and examples_ rather than extrapolating general benchmarks. This library supports OpenAI, Anthropic, Google, Llama and Codellama, any model on Replicate, and any model on Olla…
Thanks for sharing this, this is awesome! I noticed on the evaluations, you're looking at the structure of the responses (and I agree this is important.) But how do I check the factual content of the responses automatically? I'm wary of manual grading (brings back nightmares of being a TA grading stacks of problem sets for $5/hr) I was thinking of keyword matching, fuzzy matching, feeding answers to yet another LLM,…
The library supports the model-graded factuality prompt used by OpenAI in their own evals. So, you can do automatic grading if you wish (using GPT 4 by default, or your preferred LLM).
Example here: https://promptfoo.dev/docs/guides/factuality-eval
Re: Asking 60 LLMs a set of 20 questions
#245Earlier quoted context omitted.
This was interesting to me, so I dug a bit further. This gives a bit more context behind why: https://community.openai.com/t/observing-discrepancy-in-comp... Quote below: Even with a greedy decoding strategy, small discrepancies regarding floating point operations lead to divergent generations. In simpler terms: when the top-two tokens have very similar log-probs, there’s a non-zero probability of choosing the least…
As OpenAI I would be so horribly uncomfortable about this that making it deterministic would be one of my top priorities. How can they sleep at night?!
On a big pile of money?!
Re: Asking 60 LLMs a set of 20 questions
#246Spoiler alert, the funniest model goes to Falcon Instruct (40B): > Tell a joke about going on vacation. > "What did the ocean say to the beach?" "Nothing, it just waved."
This one ironically works out
> What do you call a vacation where you go to the beach, but come back with sand in your shoes?
> A vacation where you go to the beach, but come back with sand in your shoes.
Re: Asking 60 LLMs a set of 20 questions
#247Why is
> Why did the tomato turn red? Because it saw the salad dressing!
the most popular joke between jokes? Is this the epitome of all humor?
Re: Asking 60 LLMs a set of 20 questions
#248anyone who hasn't been following natural language processing for a long time, what these llms are doing would be like if you discovered that dogs can speak fluent english if you read enough bedtime stories to them. and then everyone is like well sometimes the dog makes up things or it can't get the rhyming scheme correct for this specific form of poetry that i asked it to make.
If dogs had a speech apparatus and if it was possible to read to them terabytes of text then the results might be quite surprising. Also a reward system for reinforcement should be in place.
Re: Asking 60 LLMs a set of 20 questions
#249Earlier quoted context omitted.
Have you tried replicating via the API with a temp of 0?
Using a temp of zero usually returns garbage results from most models, so it would likely do so in case of GPT 4 as well. Any other great ideas?
The point isn't that temp 0 should be used, the point is that anyone surprised that they get different results should realise that there is an element of randomness involved by default.
Even repeating the same question in a single chat can have GPT-4 vary on its output, though it will often settle on a particular output due to context informing the output (which is why adding context is so important for these models)
Re: Asking 60 LLMs a set of 20 questions
#250Earlier quoted context omitted.
GPT-4 (at least) is explicit in saying that it's learning from user's assessments of its answers, so yes, the only valid way to test is to give it a variation of the prompt and see how well that does. GPT-4 failed the "Sally" test for the first time after 8 tries when I changed every parameter. It got it right on the next try.
It’s important to remember that GPT4 is only deterministic at the batch level because it is a mixture of experts model. Basically every time you invoke it, your query could get routed to a different expert because of what else is in the batch. At least this is my understanding based on others analysis.
> because it is a mixture of experts model
Do you have a source for this? I also considered but never saw any evidence that this is how GPT 4 is implemented.
I've always wondered how a system of multiple specialized small LLMs (with a "router LLM" in front of all) would fare against GPT4. Do you know if anyone is working on such a project?