I find it notable that it's not GPT-2. It's GPT2.
I’m thinking of this too. This could be it.
GPT-4.5 or GPT-5 being tested on LMSYS?
81–90 of 380 posts
Re: GPT-4.5 or GPT-5 being tested on LMSYS?
#82It could be lmsys's model, trained off GPT4 data and others in the arena.
Re: GPT-4.5 or GPT-5 being tested on LMSYS?
#83My very first response from gpt2-chatbot included a fictional source :( > A study by Lucon-Xiccato et al. (2020) tested African clawed frogs (Xenopus laevis) and found that they could discriminate between two groups of objects differing in number (1 vs. 2, 2 vs. 3, and 3 vs. 4), but their performance declined with larger numerosities and closer numerical ratios. It appears to be referring to this[1] 2018 study from t…
Its been known that most of these models hallucinate research articles frequently, perplexity.ai seems to do quite well in that regard. Not sure why that is your specific metric though when LLMs seem to be improving across a large class of other metrics.
This wasn't a "metric." It was a test to see whether or not this LLM might actually be useful to me. Just like every other LLM, the answer is a hard no: using this chatbot for real work is at best a huge waste of time, and at worst unconscionably reckless. For my specific question, I would have been much better off with a plain Google Scholar search.
Re: GPT-4.5 or GPT-5 being tested on LMSYS?
#84Man, its knowledge is insane. I run a dying forum. I first prompted with "Who is at ?" and it gave me a very endearing, weirdly knowledgeable bio of myself and my contributions to the forum including various innovations I made in the space back in the day. It summarized my role on my own forum better than I could have ever written it. And then I asked "who are other notable users at " and it gave me a list of some mo…
I wonder if it's RAGing? Dying Internet forums are way, way into the tail of the data.
Re: GPT-4.5 or GPT-5 being tested on LMSYS?
#85Man, its knowledge is insane. I run a dying forum. I first prompted with "Who is at ?" and it gave me a very endearing, weirdly knowledgeable bio of myself and my contributions to the forum including various innovations I made in the space back in the day. It summarized my role on my own forum better than I could have ever written it. And then I asked "who are other notable users at " and it gave me a list of some mo…
I wonder if it's RAGing? Dying Internet forums are way, way into the tail of the data.
I asked about gpderetta and he did hallucinate quite a bit (after some flattery).
Re: GPT-4.5 or GPT-5 being tested on LMSYS?
#86Earlier quoted context omitted.
> but in Spanish it might tell you "gatos tienen tres patas" Have you actually had a State of the art LLM do something like this? Because this >But it does not ever figure out that cats and gatos are the same thing. And in particular, a true English-language fact about cats is still true if you translate it into Spanish is just untrue. You can definitely query knowledge only learnt in one language in other languages.
I wasn't talking about "state of the art LLMs," I am aware that commercial offerings are much better trained in Spanish. This was a thought experiment based on comments from people testing GPT-3.5 with Swahili. > You can definitely query knowledge only learnt in one language in other languages. Do you have a source on that? I believe this is simply not true, unless maybe the pretraining data has enough context-specif…
>I wasn't talking about "state of the art LLMs," I am aware that commercial offerings are much better trained in Spanish. This was a thought experiment based on comments from people testing GPT-3.5 with Swahili.
A thought experiment from other people comments on another language. So...No. Fabricating failure modes from their personally constructed ideas about how LLMs work seems to be a frustratingly common occurrence in these kinds of discussions.
>Frustratingly, just few months ago I read a paper describing how LLMs excessively rely on English-language representations of ideas, but now I can't find it.
Most LLMs are trained on English overwhelmingly. GPT-3 had a 92.6% English dataset. https://github.com/openai/gpt-3/blob/master/dataset_statisti...
That the models are as proficient as they are is evidence enough of knowledge transfer clearly happening. https://arxiv.org/abs/2108.13349. If you trained a model on the Catalan tokens GPT-3 was trained on alone, you'd just get a GPT-2 level gibberish model at best. I don't doubt you, i just don't think it means what you think it means.
As for papers, these are some interesting ones.
How do languages influence each other? Studying cross-lingual data sharing during LLM fine-tuning - https://arxiv.org/pdf/2305.13286
Teaching Llama a New Language Through Cross-Lingual Knowledge Transfer - https://arxiv.org/abs/2404.04042
Zero-shot cross-lingual transfer in instruction tuning of large language models - https://arxiv.org/abs/2402.14778
Multilingual LLMs are Better Cross-lingual In-context Learners with Alignment - https://arxiv.org/abs/2305.05940
It's not like there is perfect transfer but the idea that there's none at all seemed so ridiculous to me (and why i asked the first question). Models would be utterly useless in multilingual settings if that were really the case.
Like the 3rd paper, GPT's instruction tuning is done basically only in English - https://mobile.twitter.com/janleike/status/16252072516309606...
Re: GPT-4.5 or GPT-5 being tested on LMSYS?
#87Man, its knowledge is insane. I run a dying forum. I first prompted with "Who is at ?" and it gave me a very endearing, weirdly knowledgeable bio of myself and my contributions to the forum including various innovations I made in the space back in the day. It summarized my role on my own forum better than I could have ever written it. And then I asked "who are other notable users at " and it gave me a list of some mo…
I wonder if it's RAGing? Dying Internet forums are way, way into the tail of the data.
No, I tested it, this is a pure model without RAG (at least no internet search)
Re: GPT-4.5 or GPT-5 being tested on LMSYS?
#88I FEEL THE AGI
I
LOVE
YOU
ALL.
- gpt-2 chatbot
Re: GPT-4.5 or GPT-5 being tested on LMSYS?
#89It failed on this math problem: how much alcohol volume is there in 16 grams of a 40% ABV drink, with the rest being water? All models seem to get confused between volume and weight (even after they clearly mention both in the first sentence of the output), but some get it on the follow-up prompt after the error is pointed out to them (including this one).
sorry can you explain what the correct solution is. Do you need to know how much the density of alcohol is rel. water? Also water and alcohol volumes don't add as far as I remember, so you need to account that that too?
Yes, you need to know the densities, but the models outputted the correct densities even as part of the wrong answer.
Not sure what you mean by volumes "not adding". One way to calculate it is like:
density_alcohol = 0.789g/ml
density_water = 1g/ml
weight_total = 16g
(density = weight / volume)
volume_alcohol = 0.4 * volume_total
volume_water = 0.6 * volume_total
weight_total = weight_alcohol + weight_water
weight_alcohol =
density_alcohol * volume_alcohol =
0.789 * volume_alcohol =
0.789 * (0.4 * volume_total) =
0.3156 * volume_total
weight_water =
density_water * volume_water =
volume_water =
0.6 * volume_total
weight_total =
0.3156 * volume_total + 0.6 * volume_total =
0.9156 * volume_total
volume_total =
weight_total / 0.9156 =
16 / 0.9156 =
17.4749 (rounded)
volume_alcohol =
0.4 * volume_total =
6.99 ml (rounded)
Re: GPT-4.5 or GPT-5 being tested on LMSYS?
#90Earlier quoted context omitted.
This is platform-specific, right? Chromium corrects for drift ( https://source.chromium.org/chromium/chromium/src/+/main:thi... )
I wish that was true but you can easily see it drifting in Chrome let lastms function tick() { if (lastms === undefined) lastms = new Date().getMilliseconds() else if (lastms !== new Date().getMilliseconds()) throw new Error('Drifted') } setInterval(tick, 1000)
[deleted]