Only a single 46Gb GPU? Wow AI sure is amazing tech.
Show HN: A 6M-token movable window on a single 46GB GPU
11–18 of 18 posts
Re: Show HN: A 6M-token movable window on a single 46GB GPU
#12You couldn't be bothered to write a coherent summary of what this actually is and what it does, you just let the AI write some random noise, eh?
This paper show a new tool called Galahad.Normally, AI has to think and guess the answer every time, which costs time and money. The knowledge of the model grows next to it not the model itself and no it is not the same as cache No fine-tuning needed It gives the exact same right answer every time, costs zero extra tokens, and saves lots of energy
how is it not the same as a cache it its exact description matches the description of a cache?
Re: Show HN: A 6M-token movable window on a single 46GB GPU
#13Earlier quoted context omitted.
So, you don't know what it is, either.
We handeld llm like a human brain we decopelled knowledge from the memory and build a memory layer that makes redoing things free and fast, so the llm can once it learned something solves it for free the next time
this sounds like ur explaining caching
Re: Show HN: A 6M-token movable window on a single 46GB GPU
#14You couldn't be bothered to write a coherent summary of what this actually is and what it does, you just let the AI write some random noise, eh?
This paper show a new tool called Galahad.Normally, AI has to think and guess the answer every time, which costs time and money. The knowledge of the model grows next to it not the model itself and no it is not the same as cache No fine-tuning needed It gives the exact same right answer every time, costs zero extra tokens, and saves lots of energy
Maybe it all works, but the paper is not trivial to decipher and the GitHub repository does not seem to exist. It doesn't seem to define what are the inputs to the system (what is a query? UTF-8 text? tokens?) and what are the outputs. It'd really help if the algorithm was written out step by step with all the expected type information included.
At first I thought it was similar to something I've built before as a long-term slowly degrading cache for augmenting an FFN by caching well-learned answers, answering by performing a beam search in the key space resulting in located key accuracy measure (how well it corresponds to the input query) and answer confidence (has it been a long time since verification?), but that's not quite it? It feels similar in some way, but is it?
Re: Show HN: A 6M-token movable window on a single 46GB GPU
#15Earlier quoted context omitted.
This paper show a new tool called Galahad.Normally, AI has to think and guess the answer every time, which costs time and money. The knowledge of the model grows next to it not the model itself and no it is not the same as cache No fine-tuning needed It gives the exact same right answer every time, costs zero extra tokens, and saves lots of energy
how is it not the same as a cache it its exact description matches the description of a cache?
A cache remembers answers (only useful for the exact same question again). We remember the proven method and redo the work on every new question, so it solves ones it's never seen, which a cache simply can't.
Re: Show HN: A 6M-token movable window on a single 46GB GPU
#16Earlier quoted context omitted.
We handeld llm like a human brain we decopelled knowledge from the memory and build a memory layer that makes redoing things free and fast, so the llm can once it learned something solves it for free the next time
this sounds like ur explaining caching
Sounds like they're explaining magic. Because current models cannot learn and they cannot remember.
Re: Show HN: A 6M-token movable window on a single 46GB GPU
#17Earlier quoted context omitted.
this sounds like ur explaining caching
Sounds like they're explaining magic. Because current models cannot learn and they cannot remember.
there is RL for LLMs which actually changes the weights but its more specialization than learning and wont counter the probabalistic nature of the thing
Re: Show HN: A 6M-token movable window on a single 46GB GPU
#18Earlier quoted context omitted.
Sounds like they're explaining magic. Because current models cannot learn and they cannot remember.
there is RL for LLMs which actually changes the weights but its more specialization than learning and wont counter the probabalistic nature of the thing
Training is not something you can just bolt on, and it generally requires even larger hardware than inference for a given model, and a huge amount of time. If you're aiming for "free", a OP claims, RL aint it.