Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Show HN: How Stale Is Your AI? Release age and training cutoff for 20 models

stale.jock.pl

Post reply on HN
Print view View original

11–20 of 51 posts

Re: Show HN: How Stale Is Your AI? Release age and training cutoff for 20 models

#11

Post by super256 »

gjskngnf wrote: post #8

I remember when the US captured Venezuelan president Maduro, and when I posed a prompt related to this, the model said that’s pure fiction. I told it to double check. Still didn’t want to entertain the idea. It only acquiesced when I specifically directed it to check Reuters. I haven’t noticed this problem in months. Model cutoff seems to be less of a problem these days.

It's a "problem" of compute, I think. If you query without an account on ChatGPT you will see the model look up less stuff and research less, than when you have a paid account and choose "medium" or "high" in the effort slider.

Which makes sense, because of you have looked into search and crawlers you notice that search is actual quite expensive (which is why e.g. Kagi charges a few bucks for search every month).

Re: Show HN: How Stale Is Your AI? Release age and training cutoff for 20 models

#12

Post by InsideOutSanta »

gjskngnf wrote: post #8

I remember when the US captured Venezuelan president Maduro, and when I posed a prompt related to this, the model said that’s pure fiction. I told it to double check. Still didn’t want to entertain the idea. It only acquiesced when I specifically directed it to check Reuters. I haven’t noticed this problem in months. Model cutoff seems to be less of a problem these days.

Came here to say the same thing. Models used to rely heavily on world knowledge from their training data. They are now much better at tool use and deciding when to research a topic, rather than just answering from memory.

I wonder how much that extends to using LLMs for programming. I assume most knowledge of programming language syntax still comes from training data.

Re: Show HN: How Stale Is Your AI? Release age and training cutoff for 20 models

#14

Post by bigmadshoe »

dominotw wrote: post #7
jasonjmcghee wrote: post #4

It still matters, but in the age of good reasoning, tool use, and web search, this is much less of a problem than it used to be.

all the reasoning still comes from pretraining data

I think they mean reasoning their way to the need for a web search.

Re: Show HN: How Stale Is Your AI? Release age and training cutoff for 20 models

#15

Post by binlog »

dominotw wrote: post #7
jasonjmcghee wrote: post #4

It still matters, but in the age of good reasoning, tool use, and web search, this is much less of a problem than it used to be.

all the reasoning still comes from pretraining data

Says who? Models can also use results from tool calls in their reasoning loops.

Re: Show HN: How Stale Is Your AI? Release age and training cutoff for 20 models

#17

Post by jasonjmcghee »

bigmadshoe wrote: post #14
dominotw wrote: post #7

Earlier quoted context omitted.

all the reasoning still comes from pretraining data

I think they mean reasoning their way to the need for a web search.

Or other kind of search / knowledge acquisition / computer use etc to get the information needed

Re: Show HN: How Stale Is Your AI? Release age and training cutoff for 20 models

#18

Post by jasonjmcghee »

VCFundedGenYer wrote: post #13

I remember running the docker container for ollama and its knowledge cutoff is somewhere in 2023 still. That's unacceptable.

ollama is just an inference engine - it just runs models.

it must ship with some default old model if you didn't need to explicitly download one

Re: Show HN: How Stale Is Your AI? Release age and training cutoff for 20 models

#19

Post by Catloafdev »

super256 wrote: post #11
gjskngnf wrote: post #8

I remember when the US captured Venezuelan president Maduro, and when I posed a prompt related to this, the model said that’s pure fiction. I told it to double check. Still didn’t want to entertain the idea. It only acquiesced when I specifically directed it to check Reuters. I haven’t noticed this problem in months. Model cutoff seems to be less of a problem these days.

It's a "problem" of compute, I think. If you query without an account on ChatGPT you will see the model look up less stuff and research less, than when you have a paid account and choose "medium" or "high" in the effort slider. Which makes sense, because of you have looked into search and crawlers you notice that search is actual quite expensive (which is why e.g. Kagi charges a few bucks for search every month).

It's not strictly compute, because this has noticeably improved in open-weight models too, such as Gemma and Qwen. I suspect they noticed this issue and adjusted their training to be better about it over time.
Post reply on HN