Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

GPT-4.5 or GPT-5 being tested on LMSYS?

rentry.co

Post reply on HN
Print view View original

371–380 of 380 posts

Re: GPT-4.5 or GPT-5 being tested on LMSYS?

#371

Post by astrange »

ayewo wrote: post #367
astrange wrote: post #257

Earlier quoted context omitted.

Apparently much of ChatGPT's purple prose and occasional rare word usage is because it's speaking African-accented English because they used Kenyan/Nigerian workers for training.

> Apparently much of ChatGPT's purple prose and occasional rare word usage is because it's speaking African-accented English because they used Kenyan/Nigerian workers for training. Do you have a source for the 2nd part of your claim that they use Nigerians for RLHF? I’m know OpenAI use Kenyans via Sama / Samasource [0] and it has been discussed here [1] before. 0: https://www.sama.com/ 1: OpenAI used Kenyan workers o…

Mostly that some of the words it uses like "delve" are common in Nigerian English, according to them anyway, and they consider themselves especially big fans of writing purple prose the same way it does.

https://www.nairaland.com/3716282/nigerians-use-english-lang...

Re: GPT-4.5 or GPT-5 being tested on LMSYS?

#372

Post by Kerbonut »

HarHarVeryFunny wrote: post #292
zarzavat wrote: post #243

Earlier quoted context omitted.

There are different shades of AGI, but we don’t know if they will happen all at once or not. For example, if an AI can replace the average white collar worker and therefore cause massive economic disruption, that would be a shade of AGI. Another shade of AGI would be an AI that can effectively do research level mathematics and theoretical physics and is therefore capable of very high-level logical reasoning. We don’t…

The term "AGI" has been loosely used for so many years that it doesn't mean anything very specific. The meaning of words derives from their usage. To me Shane Legg's (DeepMind) definition of AGI meaning human level across full spectrum of abilities makes sense. Being human or super-human level at a small number of specialized things like math is the definition of narrow AI - the opposite of general/broad AI. As long…

What if it sleeps for 8 hours every 16 hours and during that sleep period, it updates its weights with whatever knowledge it learned that day? Then it doesn't need $1B of compute every 3 months, it would use the $1B of compute for 8 hours every day. Now extrapolate the compute required for this into the future and the costs will come down. I don't know where I was going with that...

Re: GPT-4.5 or GPT-5 being tested on LMSYS?

#373

Post by swyx »

david_allison wrote: post #291
Kuinox wrote: post #279

Earlier quoted context omitted.

You can ask it's knowledge cutoff and it will respond November 2023. It have no idea of the big events of the beginning of 2024, like the earthquake in Japan.

It's close to live-ingesting GitHub. I uploaded a text + translation into English on Sunday, and it's almost perfectly replicated the translation. 0 Google results.

how is this proof of live ingesting github?

Re: GPT-4.5 or GPT-5 being tested on LMSYS?

#374

Post by versteegen »

jacobsimon wrote: post #370
versteegen wrote: post #359

Earlier quoted context omitted.

The internal state at layer M of token N is available at every following token > N and layer > M via attention heads. Transformed by a matrix but a very direct lookup mechanism. The state after the final attention layer is not addressable in this way, but it immediately becomes the output token which is of course accessible. Note also that sequential computations such as loops translate nicely to parallel ones, e.g.…

But is this lookup mechanism available from one token prediction to the next? I’ve heard conflicting things, with others saying that transformers are stateless and therefore don’t share this information across prediction steps. I might be misunderstanding something fundamental.

Yes, attention (in transformer decoders) looks backwards to internal state at previous tokens. (In transform encoders like in BERT it can also look forwards.) When they said "stateless" I think they meant that you can recompute the state from the tokens, so the state can be discarded at any time: the internal state is entirely deterministic, it's only the selection of output tokens that involves random sampling. What's also a critical feature of transformers is that you can compute the state at layer N for all tokens in parallel, because it depends only on layer N-1 for the current and all previous tokens, not on layer N for the previous token as in LSTMs or typical RNNs. The whole point of the transformer architecture is to allow that parallel compute, at the cost of directly depending on every previous token rather than just the last.

So if you wished you could implement a transformer by recomputing everything on every token. That would be incredibly inefficient. However, if you're continuing a conversation with an LLM you likely would recompute all the state for all tokens on each new user input, because the alternative is to store all that state in memory until the user gets back to you again a minute later. If you have too many simultaneous users you won't have enough VRAM for that. (In some cases moving it out of VRAM temporarily might be practical.)

Re: GPT-4.5 or GPT-5 being tested on LMSYS?

#375

Post by brianjking »

swyx wrote: post #373
david_allison wrote: post #291

Earlier quoted context omitted.

It's close to live-ingesting GitHub. I uploaded a text + translation into English on Sunday, and it's almost perfectly replicated the translation. 0 Google results.

how is this proof of live ingesting github?

lol, clearly doesn't seem to be.

Re: GPT-4.5 or GPT-5 being tested on LMSYS?

#376

Post by HarHarVeryFunny »

Kerbonut wrote: post #372
HarHarVeryFunny wrote: post #292

Earlier quoted context omitted.

The term "AGI" has been loosely used for so many years that it doesn't mean anything very specific. The meaning of words derives from their usage. To me Shane Legg's (DeepMind) definition of AGI meaning human level across full spectrum of abilities makes sense. Being human or super-human level at a small number of specialized things like math is the definition of narrow AI - the opposite of general/broad AI. As long…

What if it sleeps for 8 hours every 16 hours and during that sleep period, it updates its weights with whatever knowledge it learned that day? Then it doesn't need $1B of compute every 3 months, it would use the $1B of compute for 8 hours every day. Now extrapolate the compute required for this into the future and the costs will come down. I don't know where I was going with that...

These current LLMs are purely pre-trained - there is no way to do incremental learning (other than a small amount of fine-tuning) without disrupting what they were pre-trained on. In any case, even if someone solves incremental learning, this is just a way of growing the dataset, which is happening anyway, and under the much more controlled/curated way needed to see much benefit.

There is very much a recipe (10% if this, 20% of that, curriculum learning, mix of modalities, etc) for the type of curated dataset creation and training schedule needed to advance model capabilities. There have even been some recent signs of "inverse scaling" where a smaller model performs better in some areas than a larger one due to getting this mix wrong. Throwing more random data at them isn't what is needed.

I assume we will eventually move beyond pre-trained transformers to better architectures where maybe architectural advances and learning algorithms do have more potential for AI-designed improvement, but it seems the best role for AI currently is synthetic data generation, and developer tools.

Re: GPT-4.5 or GPT-5 being tested on LMSYS?

#377

Post by peheje »

zimmund wrote: post #333
peheje wrote: post #330

Opening this page caused my RAM usage of Chrome to skyrocket. Amount of processes doubled. When looking at Chrome task manager (Windows shortcut Shift+Esc) list, sub-frames popped up I didn't recognize such as: https://adnxs.com/ https://adsrvr.org/ https://btloader.com/ https://doubleclick.net/ https://gumgum.com/ https://openx.net/ https://smartadserver.com/ And many more. Each taking up RAM. In incognito mode / wi…

The site seems to use gradio (which from what I've experienced in the past is quite slow). All those domains can be avoided just using an ad blocker, I suggest ublock origin.

I tried ublock, seems to work, only btloader is still shown. I think the previous adblocker I had still loaded them in, just hiding them visually.

Re: GPT-4.5 or GPT-5 being tested on LMSYS?

#378

Post by ayewo »

astrange wrote: post #371
ayewo wrote: post #367

Earlier quoted context omitted.

> Apparently much of ChatGPT's purple prose and occasional rare word usage is because it's speaking African-accented English because they used Kenyan/Nigerian workers for training. Do you have a source for the 2nd part of your claim that they use Nigerians for RLHF? I’m know OpenAI use Kenyans via Sama / Samasource [0] and it has been discussed here [1] before. 0: https://www.sama.com/ 1: OpenAI used Kenyan workers o…

Mostly that some of the words it uses like "delve" are common in Nigerian English, according to them anyway, and they consider themselves especially big fans of writing purple prose the same way it does. https://www.nairaland.com/3716282/nigerians-use-english-lang...

Delve is not commonly used Nigerian English nor is it common in the corrupt form of English that is widely spoken in Nigeria called pidgin.

The link you shared doesn’t really back up your claim though.

It merely talks about government communications and reporting by the press that tend to use unfamiliar words when simple words would have done the same job. That’s a phenomenon that’s not unique to Nigerians though.

Re: GPT-4.5 or GPT-5 being tested on LMSYS?

#379

Post by b33j0r »

ShamelessC wrote: post #186
b33j0r wrote: post #173

No criticism at this particular thing, but it is a good opportunity to mention that GPT-5 only exists if you have non-public information. Sure it does. I mention because it is not a good sign that “people are getting this,” when youtubers are using headlines like “What GPT-7 means for your sales leads!” The fallacy is kind of allowed by us who understand it better, when we accept semver from companies as actually bei…

> No criticism at this particular thing, but it is a good opportunity to mention that GPT-5 only exists if you have non-public information. What? > Sure it does. What? Contradicting yourself immediately? > I mention because it is not a good sign that “people are getting this,” when youtubers are using headlines like “What GPT-7 means for your sales leads!” …what? > The fallacy is kind of allowed by us who understand…

My only point was that people were reacting to future versions of a confidential product as if we know. I didn’t contradict myself by saying “obviously they are working on new versions.”

My point was that we have no idea what the new versions are.

Re: GPT-4.5 or GPT-5 being tested on LMSYS?

#380

Post by lelandfe »

MyFirstSass wrote: post #178
hombre_fatal wrote: post #37

Man, its knowledge is insane. I run a dying forum. I first prompted with "Who is at ?" and it gave me a very endearing, weirdly knowledgeable bio of myself and my contributions to the forum including various innovations I made in the space back in the day. It summarized my role on my own forum better than I could have ever written it. And then I asked "who are other notable users at " and it gave me a list of some mo…

Weird, it doesn't seem to have any info on reddit users or their writings. I tried asking about a bunch, also just about general "legendary users" from various subreddits and it seemingly just hallucinated.

They may have put restrictions in place. I know they did after the NYT sued them over being able to reproduce paywalled articles.

Which is different from not actually having that info.

Post reply on HN