Qwen-3.8 for me automatically set copyright footer on a website to 2025 and thought Astro 5.x is the latest version which first came out in December 2024
Show HN: How Stale Is Your AI? Release age and training cutoff for 20 models
41–50 of 51 posts
Re: Show HN: How Stale Is Your AI? Release age and training cutoff for 20 models
#42For general purpose use this is interesting, but if I'm just using an LLM for coding, does this matter at all? I would hope something like a new java version after a model's publish date can be handled and understood by the model through tool calls and context even if it's not explicitly in the training data, the same way the LLM doesn't have my existing code or the plan to change it baked in from training.
My experience with Claude is that it doesn’t handle its own training cutoff properly. It responds as though its cutoff is today. Gemini is even worse about this. I’m going to add this site to my agents files so that they’re explicitly aware of their own limitations.
[flagged]
Re: Show HN: How Stale Is Your AI? Release age and training cutoff for 20 models
#43It still matters, but in the age of good reasoning, tool use, and web search, this is much less of a problem than it used to be.
It can be quite hard to determine what needs a tool call or not. LLMs are not well calibrated to what they know and don't know, and tool calls can add latency and extra costs. There are lots of things that are "obvious" right until they aren't - especially political events and disasters.
[flagged]
Re: Show HN: How Stale Is Your AI? Release age and training cutoff for 20 models
#44[dead]
Re: Show HN: How Stale Is Your AI? Release age and training cutoff for 20 models
#45Earlier quoted context omitted.
It's not strictly compute, because this has noticeably improved in open-weight models too, such as Gemma and Qwen. I suspect they noticed this issue and adjusted their training to be better about it over time.
I built a toy news-summarizing agent with Gemma 4, and it was so frustrating, actually, because of the cut-off date. The model wasted over half the token budget, each time, on internal debates over the current date. When generating a World Cup summary, for example, it refused to believe qualification rounds were over and refused to even call the web searching tool to collect the data. I injected the current datetime…
Did you try other models (e.g. Qwen3.6 or others)? I'm curious how others fare.
I've noticed Qwen3.6 struggles a bit with today/date based logic.
Re: Show HN: How Stale Is Your AI? Release age and training cutoff for 20 models
#46Earlier quoted context omitted.
It's not strictly compute, because this has noticeably improved in open-weight models too, such as Gemma and Qwen. I suspect they noticed this issue and adjusted their training to be better about it over time.
I built a toy news-summarizing agent with Gemma 4, and it was so frustrating, actually, because of the cut-off date. The model wasted over half the token budget, each time, on internal debates over the current date. When generating a World Cup summary, for example, it refused to believe qualification rounds were over and refused to even call the web searching tool to collect the data. I injected the current datetime…
Qwen 3.6 did the same thing for me.
Only after some cajolling it finally went to check the history I asked it to (I had been testing KoboldCPP's web search).
Re: Show HN: How Stale Is Your AI? Release age and training cutoff for 20 models
#47Depending on the use case certain models
Re: Show HN: How Stale Is Your AI? Release age and training cutoff for 20 models
#48An aside - isn't it nice that have been fewer model releases the past week or so? A brief respite!
Re: Show HN: How Stale Is Your AI? Release age and training cutoff for 20 models
#49Earlier quoted context omitted.
I was not expecting model weights to be updated on current events. It’s clearly warranted because a model that trusts its weights on current events will give an outdated answer. Extremely improbable events happen all the time.
I think the models are trying to optimistically avoid doing web searches, because they're surprisingly a lot harder to do well than you'd think.
How so?
Re: Show HN: How Stale Is Your AI? Release age and training cutoff for 20 models
#50After Trump's last inauguration, ChatGPT would still tell me that Biden was President of the US. I understand that the training cutoff was before Biden dropped out. But it knew, or should have known, the current date and that there had been an election since its last update, but it didn't qualify the answer. When I asked it to search the web, it got it right. The moral I took away was to always ask for the search whe…
I forgot which was it, ChatGPT or Gemini, but one of them insisted on calling Trump "former president" even when discussing decisions he just announced as president. Lol
The Trump and former president terms were likely firmly stuck together in the embedding space. The model doesn’t validate every single token it produces because validation itself requires tokens. A bloom filter of outdated embeddings will help, when the labs get around to adding it.