Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Exploring Claude/GPT Knowledge Cutoffs and Pre-Training Timelines

blog.sshh.io

Post reply on HN
Print view View original

11–20 of 22 posts

Re: Exploring Claude/GPT Knowledge Cutoffs and Pre-Training Timelines

#11

Post by tobwen »

sshh12 wrote: post #10
tobwen wrote: post #9

From my own experiments with various models, I suspect that LLMs have distinct/partitioned cutoff dates; for example, historical literature doesn't change (Greek history, Shakespeare, Goethe), general knowledge (updated only in certain areas), technologies (updated regularly), software also remains surprisingly stable - for example, with GIT, a basic command set is sufficient to do 99% of the jobs - new features are…

I would have thought that as well but at least for coding vs world event facts I didn't see obvious differences in cutoffs

In my tests, I compiled the changelogs from various projects and checked how much functionality the model could recall over several test runs. In some cases, even with the latest models, the cutoff was around June 2024. Beyond that, it didn't recognize the features. However, it did know the cause of the last pope's death and even which hospital he was in (April 2025). However, the model did not know when Trump was sworn for the second time (Jan 2025) - but it did know THAT he was elected a second time (Nov 2024).

Re: Exploring Claude/GPT Knowledge Cutoffs and Pre-Training Timelines

#12

Post by tedsanders »

ddxv wrote: post #2

This was great! One thing that wasn't addressed that I always assume, is that a marketing name like "Opus 5" is not a single model, but many models, versions and gets minor updates over time. I also assumed many questions get routed to simpler models or programs to answer correctly, but it almost surprisingly didn't seem that way from the post. Anyways, great post.

I can explain how we do it at OpenAI.

In the API, we keep the models fixed. There are tiny caveats like rare bug fixes or models like `chat-latest`, but this is spiritually true. Suspicions of models changing over time are either human hallucinations or bugs on our end.

However, in ChatGPT, we sometimes update models without changing their names. For example, we recently launched an update to GPT-5.6 Sol in ChatGPT (https://openai.com/index/improving-gpt-5-6-sol-in-chatgpt/). Our goal isn't to be opaque or sneaky, but just to not exhaust people trying to keep track of little changes. When the changes are big, we give models a new name so that people know to expect something different.

Re: Exploring Claude/GPT Knowledge Cutoffs and Pre-Training Timelines

#14

Post by derefr »

tedsanders wrote: post #12
ddxv wrote: post #2

This was great! One thing that wasn't addressed that I always assume, is that a marketing name like "Opus 5" is not a single model, but many models, versions and gets minor updates over time. I also assumed many questions get routed to simpler models or programs to answer correctly, but it almost surprisingly didn't seem that way from the post. Anyways, great post.

I can explain how we do it at OpenAI. In the API, we keep the models fixed. There are tiny caveats like rare bug fixes or models like `chat-latest`, but this is spiritually true. Suspicions of models changing over time are either human hallucinations or bugs on our end. However, in ChatGPT, we sometimes update models without changing their names. For example, we recently launched an update to GPT-5.6 Sol in ChatGPT (…

That's an good answer to the question when taken in strict terms of a model version = weights.

But, insofar as:

1. "a model" as presented to the user, isn't just its weights, but also anything else happening on the "business layer" (though this maybe applies more to ChatGPT than "direct" model access via the API); and

2. said business layer has any "knowledge base"-type stuff going on in it (i.e. automatic or tool-call-triggered embedding of results from search of some vector-DB into which has been embedded distilled pre-validated trustworthy info — like per-user memory mechanisms, but searching + injecting from global shared data sources); and

3. said "knowledge base" mechanisms are where most of the up-to-date, fast-changing info a model "knows" (without having to do a web search) is actually coming from;

...then do y'all ever update the pinned knowledge-base data snapshot associated with the model version, without updating the weights themselves?

Re: Exploring Claude/GPT Knowledge Cutoffs and Pre-Training Timelines

#15

Post by discodave »

tedsanders wrote: post #12
ddxv wrote: post #2

This was great! One thing that wasn't addressed that I always assume, is that a marketing name like "Opus 5" is not a single model, but many models, versions and gets minor updates over time. I also assumed many questions get routed to simpler models or programs to answer correctly, but it almost surprisingly didn't seem that way from the post. Anyways, great post.

I can explain how we do it at OpenAI. In the API, we keep the models fixed. There are tiny caveats like rare bug fixes or models like `chat-latest`, but this is spiritually true. Suspicions of models changing over time are either human hallucinations or bugs on our end. However, in ChatGPT, we sometimes update models without changing their names. For example, we recently launched an update to GPT-5.6 Sol in ChatGPT (…

Is there any scaffolding or harness runnning around the model on the cloud/server side? Seems like there's a lot of opportunity to make changes/improvements without making your customers call a different API or change the model parameter.

As an example, S3 team was able to migrate from eventually-consistent, to consistent without making any API changes, a complete re-architecture on the backend with 0 API changes.

Re: Exploring Claude/GPT Knowledge Cutoffs and Pre-Training Timelines

#16

Post by tedsanders »

derefr wrote: post #14
tedsanders wrote: post #12

Earlier quoted context omitted.

I can explain how we do it at OpenAI. In the API, we keep the models fixed. There are tiny caveats like rare bug fixes or models like `chat-latest`, but this is spiritually true. Suspicions of models changing over time are either human hallucinations or bugs on our end. However, in ChatGPT, we sometimes update models without changing their names. For example, we recently launched an update to GPT-5.6 Sol in ChatGPT (…

That's an good answer to the question when taken in strict terms of a model version = weights. But, insofar as: 1. "a model" as presented to the user, isn't just its weights, but also anything else happening on the "business layer" (though this maybe applies more to ChatGPT than "direct" model access via the API); and 2. said business layer has any "knowledge base"-type stuff going on in it (i.e. automatic or tool-ca…

Same spirit as above. API is fixed (though of course things like web search results can change from day to day). ChatGPT and Codex harnesses do change over time, though they never result in name changes. We document the big changes here: https://help.openai.com/en/articles/6825453-chatgpt-release-...

Re: Exploring Claude/GPT Knowledge Cutoffs and Pre-Training Timelines

#19

Post by cma »

rad-b wrote: post #4

Great read and interesting analysis! I’m less charitable toward Anthropic supposedly not distilling ChatGPT for training purposes. Maybe not today, but during the GPT-4 era when Anthropic was the underdog - I can see it happen. Packaged along with some of Amodei’s clever jumping through hoops to prove how that is, in fact, virtuous.

Distilling is only prohibited from chats you prompt. But there are lots in the open like the LMSYS Chatbot Arena data. It's a essentially one of the largest public sources of preference data, maybe still useful for boostraping RLHF and successor techniques (possibly a more highly educated contributor base than the low paid contractors), and was full of GPT-4-era inputs.

Re: Exploring Claude/GPT Knowledge Cutoffs and Pre-Training Timelines

#20

Post by PeterStuer »

myworkaccount2 wrote: post #5

I wonder if this kind of analysis will give us a way to check if the frontier labs are waiting for the right moment to release their models. To me it feels obvious that these companies are not releasing models as soon as they are done doing their post training / testing with any new model. But there is no real way to know how much of this "waiting" any lab is doing, if we can get better estimates this way maybe we ca…

They are releasing as little as they can get away with. The Open Weights models are one of the big reasons we have seen the progress we have had.
Post reply on HN