Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Grok 4 Fast now has 2M context window

docs.x.ai

Post reply on HN
Print view View original

231–240 of 328 posts

Re: Grok 4 Fast now has 2M context window

#231

Post by ErikBjare »

Ey7NFZ3P0nzAe wrote: post #56
vlovich123 wrote: post #43

Earlier quoted context omitted.

That’s a very basic way to keep the LLM inferring past the context window size (there’s better, smarter ways) but that’s not at all what the question was which is how they train a 2M token length window. My understanding at a basic level is that you need corpuses that are >2M in length for training data which is where the problem comes in for - there’s only so much long form content and it’s swamped by all the smalle…

AFAIK nobody does that. They train on much much shorter text but with use tricks in the position encoding steps that can be extrapolated by the LLMs. Lile ROPE and YARN etc.

AFAIK (not much) it definitely helps to train on longer sequences even with rope/yarn and is needed if you care about long context performance (and not just the long context capability).

Re: Grok 4 Fast now has 2M context window

#232

Post by giancarlostoro »

jorvi wrote: post #193
mehdibl wrote: post #8

What matter is not context or the recod token/s you get. But the quality for the model. And it seem Grok pushing the wrong metrics again, after launching fast.

Grok's biggest feature is that unlike all the other premier models (yes I know about ChatGPT's new adult mode), it hasn't been lobotomized by censoring.

I would argue over censorship is the better word. Ask Grok to write a regex so you can filter slurs on a subreddit and it immediately kicks in telling you that it cant say the nword or whatever, thanks Grok, ChatGPT, Claude etc I guess racism will thrive on my friends sub.

Re: Grok 4 Fast now has 2M context window

#233

Post by solatic »

throwuxiytayq wrote: post #226
mg wrote: post #175

Earlier quoted context omitted.

That makes me wonder if we could simply test this by letting the LLM add or multiply a long list of numbers? Here is an experiment: https://www.gnod.com/search/#q=%23%20Calcuate%20the%20below%... The correct answer: Correct: 20,192,642.460942328 Here is what I got from different models on the first try: ChatGPT: 20,384,918.24 Perplexity: 20,000,000 Google: 25,167,098.4 Mistral: 200,000,000 Grok: Timed out after 300s…

I’m starting to find it unreasonably funny how people always want language models to multiply numbers for some reason. Every god damn time. In every single HN thread. I think my sanity might be giving out.

A model, no, but an agent with a calculator tool?

Then there's the question of why not just build the calculator tool into the model?

Re: Grok 4 Fast now has 2M context window

#234

Post by solumunus »

giancarlostoro wrote: post #232
jorvi wrote: post #193

Earlier quoted context omitted.

Grok's biggest feature is that unlike all the other premier models (yes I know about ChatGPT's new adult mode), it hasn't been lobotomized by censoring.

I would argue over censorship is the better word. Ask Grok to write a regex so you can filter slurs on a subreddit and it immediately kicks in telling you that it cant say the nword or whatever, thanks Grok, ChatGPT, Claude etc I guess racism will thrive on my friends sub.

I can’t tell if this is serious or not. Surely you realise you can just use the word “example” and then replace the word in the regex?!

Re: Grok 4 Fast now has 2M context window

#236

Post by jknutson »

solumunus wrote: post #234
giancarlostoro wrote: post #232

Earlier quoted context omitted.

I would argue over censorship is the better word. Ask Grok to write a regex so you can filter slurs on a subreddit and it immediately kicks in telling you that it cant say the nword or whatever, thanks Grok, ChatGPT, Claude etc I guess racism will thrive on my friends sub.

I can’t tell if this is serious or not. Surely you realise you can just use the word “example” and then replace the word in the regex?!

I think they would want a more optimized regex. Like a long list of swears, merged down into one pattern separated by tunnel characters, and with all common prefixes / suffixes combined for each group. That takes more than just replacing one word. Something like the output of the list-to-tree rust crate.

Re: Grok 4 Fast now has 2M context window

#238

Post by jgalt212 »

sd9 wrote: post #229
jorvi wrote: post #193

Earlier quoted context omitted.

Grok's biggest feature is that unlike all the other premier models (yes I know about ChatGPT's new adult mode), it hasn't been lobotomized by censoring.

I am amazed people actually believe this Grok is the most biased of the lot, and they’re not even trying to hide it particularly well

According to a recent Economist article, even Grok is left-biased.

Re: Grok 4 Fast now has 2M context window

#240

Post by ahtihn »

jknutson wrote: post #236
solumunus wrote: post #234

Earlier quoted context omitted.

I can’t tell if this is serious or not. Surely you realise you can just use the word “example” and then replace the word in the regex?!

I think they would want a more optimized regex. Like a long list of swears, merged down into one pattern separated by tunnel characters, and with all common prefixes / suffixes combined for each group. That takes more than just replacing one word. Something like the output of the list-to-tree rust crate.

Wouldn't the best approach for that be to write a program that takes a list of words and output an optimized regex?

I'm sure an LLM can help write such a program. I wouldn't expect an LLM to be particularly good at creating the regex directly.

Post reply on HN