Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Show HN: How Stale Is Your AI? Release age and training cutoff for 20 models

stale.jock.pl

Post reply on HN
Print view View original

31–40 of 51 posts

Re: Show HN: How Stale Is Your AI? Release age and training cutoff for 20 models

#31

Post by NegativeLatency »

InsideOutSanta wrote: post #12
gjskngnf wrote: post #8

I remember when the US captured Venezuelan president Maduro, and when I posed a prompt related to this, the model said that’s pure fiction. I told it to double check. Still didn’t want to entertain the idea. It only acquiesced when I specifically directed it to check Reuters. I haven’t noticed this problem in months. Model cutoff seems to be less of a problem these days.

Came here to say the same thing. Models used to rely heavily on world knowledge from their training data. They are now much better at tool use and deciding when to research a topic, rather than just answering from memory. I wonder how much that extends to using LLMs for programming. I assume most knowledge of programming language syntax still comes from training data.

I find they generally do ok, but a few lines in an AGENTS.md or manual prompting to verify stuff against current docs/source, and check for current version of software helps a lot.

Re: Show HN: How Stale Is Your AI? Release age and training cutoff for 20 models

#32

Post by SirMaster »

This is one of the things that bothers me about AI.

To me, intelligence or an intelligent entity should be able to learn from its mistakes and learn new things on its own. Having to start from scratch to teach an AI new facts or new skills is not very intelligent IMO.

Re: Show HN: How Stale Is Your AI? Release age and training cutoff for 20 models

#33

Post by SirMaster »

Isamu wrote: post #22
gjskngnf wrote: post #8

I remember when the US captured Venezuelan president Maduro, and when I posed a prompt related to this, the model said that’s pure fiction. I told it to double check. Still didn’t want to entertain the idea. It only acquiesced when I specifically directed it to check Reuters. I haven’t noticed this problem in months. Model cutoff seems to be less of a problem these days.

>the model said that’s pure fiction. Were you expecting your model to be updated on current events? Why? Also the specific event you are referring to is a statistically very improbable event, prior to its actually happening. >It only acquiesced when I specifically directed it to check Reuters. Do all models do this? They check in with Reuters? Why would a model think that you asking about an extremely improbable even…

If OpenAI is going to call Astra AGI, then I would expect it to be able to update it's weights to new knowledge, because a generally intelligent being can indeed do this.

I can teach myself to play an instrument, and I'm not just building this huge lookup table that I have to access every time I play the instrument. I am updating the weights in my neurons.

Until AI can do this it's not AGI in my book.

Re: Show HN: How Stale Is Your AI? Release age and training cutoff for 20 models

#34

Post by ck2 »

SirMaster wrote: post #32

This is one of the things that bothers me about AI. To me, intelligence or an intelligent entity should be able to learn from its mistakes and learn new things on its own. Having to start from scratch to teach an AI new facts or new skills is not very intelligent IMO.

few more generations they will invent "patch in place"

(with "AI" developing the technique of course)

Re: Show HN: How Stale Is Your AI? Release age and training cutoff for 20 models

#35

Post by ctkhn »

For general purpose use this is interesting, but if I'm just using an LLM for coding, does this matter at all? I would hope something like a new java version after a model's publish date can be handled and understood by the model through tool calls and context even if it's not explicitly in the training data, the same way the LLM doesn't have my existing code or the plan to change it baked in from training.

Re: Show HN: How Stale Is Your AI? Release age and training cutoff for 20 models

#36

Post by hasteg »

speedping wrote: post #10

Pre-AI internet data is like pre-war steel The slop would multiply if we keep feeding it to new models in a loop

This is likely being solved with stuff like watermarking which Anthropic just added to Claude... I'd imagine they are testing new data and verifying that training data (or at least the data which do NOT want to be AI generated, i.e actual human text) is not generated, at least with claude.

Re: Show HN: How Stale Is Your AI? Release age and training cutoff for 20 models

#37

Post by prng2021 »

ctkhn wrote: post #35

For general purpose use this is interesting, but if I'm just using an LLM for coding, does this matter at all? I would hope something like a new java version after a model's publish date can be handled and understood by the model through tool calls and context even if it's not explicitly in the training data, the same way the LLM doesn't have my existing code or the plan to change it baked in from training.

The inference time results are quite different. These models haven’t just been trained with tons of docs, blog entries and videos about Java 8. They’ve also been trained on tons of code examples from simple to complex real world ones.

You’re not going to get all that just by pointing the LLM to the recently released Java 27 documentation. That information is also potentially adding tons of content to your context, which is already filled with tons of other data (your code, other recently released libraries it has to get documentation for, etc).

Re: Show HN: How Stale Is Your AI? Release age and training cutoff for 20 models

#38

Post by ctkhn »

prng2021 wrote: post #37
ctkhn wrote: post #35

For general purpose use this is interesting, but if I'm just using an LLM for coding, does this matter at all? I would hope something like a new java version after a model's publish date can be handled and understood by the model through tool calls and context even if it's not explicitly in the training data, the same way the LLM doesn't have my existing code or the plan to change it baked in from training.

The inference time results are quite different. These models haven’t just been trained with tons of docs, blog entries and videos about Java 8. They’ve also been trained on tons of code examples from simple to complex real world ones. You’re not going to get all that just by pointing the LLM to the recently released Java 27 documentation. That information is also potentially adding tons of content to your context, wh…

Gotcha, that makes a lot of sense. Thank you

Re: Show HN: How Stale Is Your AI? Release age and training cutoff for 20 models

#39

Post by smt88 »

ctkhn wrote: post #35

For general purpose use this is interesting, but if I'm just using an LLM for coding, does this matter at all? I would hope something like a new java version after a model's publish date can be handled and understood by the model through tool calls and context even if it's not explicitly in the training data, the same way the LLM doesn't have my existing code or the plan to change it baked in from training.

My experience with Claude is that it doesn’t handle its own training cutoff properly. It responds as though its cutoff is today. Gemini is even worse about this.

I’m going to add this site to my agents files so that they’re explicitly aware of their own limitations.

Re: Show HN: How Stale Is Your AI? Release age and training cutoff for 20 models

#40

Post by tannertech »

gjskngnf wrote: post #8

I remember when the US captured Venezuelan president Maduro, and when I posed a prompt related to this, the model said that’s pure fiction. I told it to double check. Still didn’t want to entertain the idea. It only acquiesced when I specifically directed it to check Reuters. I haven’t noticed this problem in months. Model cutoff seems to be less of a problem these days.

ChatGPT once told me I was the target of a sophisticated nation state misinformation campaign when I linked it a Reuters article
Post reply on HN