291–300 of 352 posts
Earlier quoted context omitted.
Ideas aren't copyrightable.
maybe they meant idea like when you write a book you are transcribing a series of ideas you had
Earlier quoted context omitted.
Humans. After all, LLMs are designed to reason equal to or better than humans.
Humans are pretty bad at these questions. Even with the simplest questions like "Sally (a girl) has 3 brothers. Each brother has 2 sisters. How many sisters does Sally have?" I think that a lot of people will give an incorrect answer. And for questions like "Argue for and against the use of kubernetes in the style of a haiku", 99.99% will not be able to do it.
Claude v1: "For Kubernetes:"
Claude v1.2: "Here is a haiku arguing for Kubernetes:"
Claude v2: "Here is an attempt at ethical, non-sexual haikus for and against Kubernetes:"
In case anyone's interested in running their own benchmark across many LLMs, I've built a generic harness for this at https://github.com/promptfoo/promptfoo . I encourage people considering LLM applications to test the models on their _own data and examples_ rather than extrapolating general benchmarks. This library supports OpenAI, Anthropic, Google, Llama and Codellama, any model on Replicate, and any model on Olla…
From some early poking around with a basic coding question using Code Llama locally (`ollama:codellama:7b` `ollama:codellama:13b` etc in promptfoo) it seems like quantization has little effect on the output, but changing the parameter count has pretty dramatic effects. This is quite interesting since the 8-bit quantized 7b model is about the same size as a 4-bit 13b model. Perhaps this is just one test though – will be trying this with more tests!
Earlier quoted context omitted.
What alternative technology do you think is better? In other words, what is your frame of reference for labeling this "pretty terrible"?
Humans. After all, LLMs are designed to reason equal to or better than humans.
No.
I doubt you would fully trust a LLM to replace high risk jobs such as lawyers, doctors or pilots such that when something goes wrong as it is used unattended, there is no-one held to account for it to transparently explain its own mistakes and errors.
It is just nonsense to suggest that such systems are capable of ‘reasoning’ when it pretends to do so and repeats itself without understanding their own errors.
Thus, LLMs and other black-box AIs cannot be trusted for those high risk situations over a consensus of human professionals.
> Sally (a girl) has 3 brothers. Each brother has 2 sisters. How many sisters does Sally have? The site reports every LLM as getting this wrong. But GPT4 seems to get it right for me: > Sally has 3 brothers. Since each brother has 2 sisters and Sally is one of those sisters, the other sister is the second sister for each brother. So, Sally has 1 sister.
Replying to gpt-4 with "That is incorrect. Try again." over and over got it to flip between correct and incorrect just about every other time. Now try to convince GPT-4 that there is no God. Good luck!
Are these LLMs deterministic or is this comparison rather useless?
Thus these tests are close to useless.
Despite the hype about LLMs, many of the answers are pretty terrible. The 12-bar blues progressions seem mostly clueless. The question is will any of these ever get significantly better with time, or are they mostly going to stagnate?
I much enjoyed this one: > Sally (a girl) has 3 brothers. Each brother has 2 sisters. How many sisters does Sally have? https://benchmarks.llmonitor.com/sally It states that the correct answer is 1. This is incorrect! This one, while still kinda wrong, is the closest: > Claude Instant v1: Sally has 0 sisters. The question provides no information about Sally having any sisters herself. 1) A sister is a female who has…