Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Show HN: How Stale Is Your AI? Release age and training cutoff for 20 models

stale.jock.pl

Post reply on HN
Print view View original

21–30 of 51 posts

Re: Show HN: How Stale Is Your AI? Release age and training cutoff for 20 models

#21

Post by lukewarm707 »

jasonjmcghee wrote: post #4

It still matters, but in the age of good reasoning, tool use, and web search, this is much less of a problem than it used to be.

in my chat with gemini it could not differentiate between current events and fiction.

if you point it to the web it got the point, but started treating everything like fiction. so it simply started making up possible scenarios and playing them off as real answers when asked for factual information.

i could not tell what the issue was or how to fix it because the reasoning is encrypted. the obfuscation model spat out something like: 'the user is asking for details about a fictional scenario in which the usa has assassinated the leader of iran'

i really don't like the way big ai companies are going. encrypted thinking, guardrails, adversarial personality, moralizing. it is creating something anti-human.

Re: Show HN: How Stale Is Your AI? Release age and training cutoff for 20 models

#22

Post by Isamu »

gjskngnf wrote: post #8

I remember when the US captured Venezuelan president Maduro, and when I posed a prompt related to this, the model said that’s pure fiction. I told it to double check. Still didn’t want to entertain the idea. It only acquiesced when I specifically directed it to check Reuters. I haven’t noticed this problem in months. Model cutoff seems to be less of a problem these days.

>the model said that’s pure fiction.

Were you expecting your model to be updated on current events? Why?

Also the specific event you are referring to is a statistically very improbable event, prior to its actually happening.

>It only acquiesced when I specifically directed it to check Reuters.

Do all models do this? They check in with Reuters? Why would a model think that you asking about an extremely improbable event warranted reaching out to Reuters?

Re: Show HN: How Stale Is Your AI? Release age and training cutoff for 20 models

#23

Post by mywittyname »

Isamu wrote: post #22
gjskngnf wrote: post #8

I remember when the US captured Venezuelan president Maduro, and when I posed a prompt related to this, the model said that’s pure fiction. I told it to double check. Still didn’t want to entertain the idea. It only acquiesced when I specifically directed it to check Reuters. I haven’t noticed this problem in months. Model cutoff seems to be less of a problem these days.

>the model said that’s pure fiction. Were you expecting your model to be updated on current events? Why? Also the specific event you are referring to is a statistically very improbable event, prior to its actually happening. >It only acquiesced when I specifically directed it to check Reuters. Do all models do this? They check in with Reuters? Why would a model think that you asking about an extremely improbable even…

He asked it to double check. It's reasonable to expect the LLM to handle that trivial task.

Re: Show HN: How Stale Is Your AI? Release age and training cutoff for 20 models

#24

Post by gjskngnf »

Isamu wrote: post #22
gjskngnf wrote: post #8

I remember when the US captured Venezuelan president Maduro, and when I posed a prompt related to this, the model said that’s pure fiction. I told it to double check. Still didn’t want to entertain the idea. It only acquiesced when I specifically directed it to check Reuters. I haven’t noticed this problem in months. Model cutoff seems to be less of a problem these days.

>the model said that’s pure fiction. Were you expecting your model to be updated on current events? Why? Also the specific event you are referring to is a statistically very improbable event, prior to its actually happening. >It only acquiesced when I specifically directed it to check Reuters. Do all models do this? They check in with Reuters? Why would a model think that you asking about an extremely improbable even…

I was not expecting model weights to be updated on current events.

It’s clearly warranted because a model that trusts its weights on current events will give an outdated answer. Extremely improbable events happen all the time.

Re: Show HN: How Stale Is Your AI? Release age and training cutoff for 20 models

#25

Post by ahmedfromtunis »

Catloafdev wrote: post #19
super256 wrote: post #11

Earlier quoted context omitted.

It's a "problem" of compute, I think. If you query without an account on ChatGPT you will see the model look up less stuff and research less, than when you have a paid account and choose "medium" or "high" in the effort slider. Which makes sense, because of you have looked into search and crawlers you notice that search is actual quite expensive (which is why e.g. Kagi charges a few bucks for search every month).

It's not strictly compute, because this has noticeably improved in open-weight models too, such as Gemma and Qwen. I suspect they noticed this issue and adjusted their training to be better about it over time.

I built a toy news-summarizing agent with Gemma 4, and it was so frustrating, actually, because of the cut-off date.

The model wasted over half the token budget, each time, on internal debates over the current date.

When generating a World Cup summary, for example, it refused to believe qualification rounds were over and refused to even call the web searching tool to collect the data.

I injected the current datetime at the very beginning of the system prompt, but Gemma refused to believe it!

The m-effer insisted the timestamp was fake and hypothesized it was being evaluated in a synthetic lab test with simulated future dates!

No amount of system prompting could convince it to trust the clock.

That was the most frustrating and bizarre "bug" I ever faced!

Re: Show HN: How Stale Is Your AI? Release age and training cutoff for 20 models

#26

Post by ahmedfromtunis »

delichon wrote: post #5

After Trump's last inauguration, ChatGPT would still tell me that Biden was President of the US. I understand that the training cutoff was before Biden dropped out. But it knew, or should have known, the current date and that there had been an election since its last update, but it didn't qualify the answer. When I asked it to search the web, it got it right. The moral I took away was to always ask for the search whe…

I forgot which was it, ChatGPT or Gemini, but one of them insisted on calling Trump "former president" even when discussing decisions he just announced as president. Lol

Re: Show HN: How Stale Is Your AI? Release age and training cutoff for 20 models

#27

Post by xienze »

gjskngnf wrote: post #24
Isamu wrote: post #22

Earlier quoted context omitted.

>the model said that’s pure fiction. Were you expecting your model to be updated on current events? Why? Also the specific event you are referring to is a statistically very improbable event, prior to its actually happening. >It only acquiesced when I specifically directed it to check Reuters. Do all models do this? They check in with Reuters? Why would a model think that you asking about an extremely improbable even…

I was not expecting model weights to be updated on current events. It’s clearly warranted because a model that trusts its weights on current events will give an outdated answer. Extremely improbable events happen all the time.

I think the models are trying to optimistically avoid doing web searches, because they're surprisingly a lot harder to do well than you'd think.

Re: Show HN: How Stale Is Your AI? Release age and training cutoff for 20 models

#28

Post by spindump8930 »

jasonjmcghee wrote: post #4

It still matters, but in the age of good reasoning, tool use, and web search, this is much less of a problem than it used to be.

It can be quite hard to determine what needs a tool call or not. LLMs are not well calibrated to what they know and don't know, and tool calls can add latency and extra costs. There are lots of things that are "obvious" right until they aren't - especially political events and disasters.

Re: Show HN: How Stale Is Your AI? Release age and training cutoff for 20 models

#29

Post by spindump8930 »

VCFundedGenYer wrote: post #13

I remember running the docker container for ollama and its knowledge cutoff is somewhere in 2023 still. That's unacceptable.

This is one of the jagged mismatches between users and LLM developers. The median user doesn't care or want to know about static models and knowledge cutoffs and whether a model can do tool calls or if tool calls even happened. They just want something that works.

Fortunately increased capabilities seem to make this a basic expectation with new releases.

Re: Show HN: How Stale Is Your AI? Release age and training cutoff for 20 models

#30

Post by NegativeLatency »

gjskngnf wrote: post #8

I remember when the US captured Venezuelan president Maduro, and when I posed a prompt related to this, the model said that’s pure fiction. I told it to double check. Still didn’t want to entertain the idea. It only acquiesced when I specifically directed it to check Reuters. I haven’t noticed this problem in months. Model cutoff seems to be less of a problem these days.

Gets me with AWS stuff on claude all the time, fortunately there's a official amazon MCP for their docs which helps a lot, but I still have to occasionally tell it to check the docs/mcp.
Post reply on HN