Viewing profile — sftombu

rateLLMiter is a Python rate limiter that smoothes out requests to LLM APIs to get faster, more consistent performance. It uses a ticket bucket algorithm rather than the usual expo…

The models benchmarked by RULER do worse in needle in a needlestack. It will be interested to see how 4o does with RULER.

Previous answer to this question: https://news.ycombinator.com/item?id=40361419s

It can't answer the questions without the limericks in the prompt. The benchmark is to establish how well it uses the context window. For example, I just asked it "What is sought b…

Previous answer to this question: https://news.ycombinator.com/item?id=40361419

Previous answer to this question: https://news.ycombinator.com/item?id=40361419

Interesting!

I tested the LLMs to make sure they could not answer the questions unless the limerick was given to them. Other than 4o, they do very badly on this benchmark, so I don't think the …

They come from a database of 98k limericks -- https://zenodo.org/records/5722527

That is an interesting idea

The reason I made Needle in a needlestack is the LLMs are getting to good at needle in a haystack. Until GPT-4o, no model was good at the NIAN benchmark.

If you ask the questions without providing the limerick first, it never gets the right answer. When the LLM gets the wrong answer, it is usually because it reverts to its training …

I was thinking about something similar -- to make part of the question be sufficient information that the LLM can find the limerick. Then the 2nd part would ask something that woul…

I had the same experience. With a 16k prompt, Turbo was nearly flawless. But it wasn't very good at 32k and not usable at 100+. You have to repeat information to get good results w…

If I had access to Gemini with a reasonable token rate limit, I would be happy to test Gemini. I have had good results with it in other situations.

GPT-4o's breakthrough memory -- https://nian.llmonpy.ai/