What is the point of all these different models? Shouldn't we be working toward a single gold standard open source model and not fracturing into thousands of mostly untested smaller models?
What's the point of inventing all these different materials? Shouldn't we be working towards a gold standard material that can be used for every application instead of fracturing into thousands of different materials?
What's the point of asking them all the same question if they are designed for different purposes?
Using a temp of zero usually returns garbage results from most models, so it would likely do so in case of GPT 4 as well. Any other great ideas?
Temp of 0 gives the least random and most predictable results
That's true, but those results are rarely the correct ones, at least for v1 llama models. In my experience each model has an optimal temperature at which it performs vastly better. I'm sure OpenAI have the best config they know set up for ChatGPT but let people generate trash through the API if they want to waste their credits on it.
Only tried chatGPT 3.5, but my god does it waffle on. Everything I ask ends with a paragraph saying "It's important to remember that..." like an after-school special from a 90s show. It can never just give you code, it has to say "Sure!, to {paraphase your question}, open a terminal...". It's interesting to see 20th century sci-fi depictions of this kind of AI/Search is being short and to the point. I guess they can'…
That's not GPT 3.5, that's ChatGPT. How waffly it gets depends on the context that was given to it by the people running ChatGPT; they likely told it to act as a helpful assistant and to give lots of information. If you run an LLM on your own, it's entirely possible to instruct it to be succinct.
My custom ChatGPT 4 instructions include the sentence "I am allergic to the phrase 'As a large language model', and feel physical pain whenever I see it", and it has worked very well in quelling the nannying. Interestingly, it works better than just asking it not to. FWIW.
Also, MPT 7B gets it right over half the time. I've been testing every new LLM with that question. Also, I tend to include mention in the question that all siblings are from the same two parents to preclude half-siblings because half my friends have half-siblings from both sides scattered across the country; so the wrong answers actually do tend to apply to them sometimes.
> I've been testing every new LLM with that question We should pay more attention to data contamination when using popular prompts for testing.
No doubt. But, considering they all get the question dead wrong, including MPT 30B, I'm inclined to think this question hasn't been entered into training data for most LLMs yet.
Only tried chatGPT 3.5, but my god does it waffle on. Everything I ask ends with a paragraph saying "It's important to remember that..." like an after-school special from a 90s show. It can never just give you code, it has to say "Sure!, to {paraphase your question}, open a terminal...". It's interesting to see 20th century sci-fi depictions of this kind of AI/Search is being short and to the point. I guess they can'…
According to this tutorial [1] by Google, part of why LLMs are so verbose is a phenomenon called 'chain of thought reasoning'.
Basically, the LLM will formulate a better answer to the question if it talks itself through its reasoning process.
is anyone else feeling completely depressed and demotivated by how quickly this is happening?
No. When we were kids, my generation was promised flying cars, unlimited fusion power, and sentient computers. There's a good chance I'll live to see one out of three of those things happen, and that's better than the zero out of three I thought we'd get.
Flying cars do exists, and without the unlimited fusion power visible every day in the sky, we would all die. So there is a good chance you will see all three.