None of the compaction accuracies look impressive.
I think matching or exceeding the original cache at 20% compacted size is fairly impressive.
The original cache had 70% accuracy, and the alternatives were only worse.
Fast KV Compaction via Attention Matching
11–17 of 17 posts
Re: Fast KV Compaction via Attention Matching
#12Considering the insanity of the AI arms race going on now, and the incredible sums of money be thrown at any slight advantage, is there any reason to believe that any meaningful AI breakthrough would be openly published for anyone to leverage?
These folks are MIT, so citations are valuable to them. Citations convert into prestige, academic career progression, or a favorable exit from academia into industry.
Also, I don't see why you couldn't patent this if you wanted to monetize it.
Re: Fast KV Compaction via Attention Matching
#13Considering the insanity of the AI arms race going on now, and the incredible sums of money be thrown at any slight advantage, is there any reason to believe that any meaningful AI breakthrough would be openly published for anyone to leverage?
The inventor's grace period under first to file changes still gives them/their university a year to file if they publish openly.
Re: Fast KV Compaction via Attention Matching
#14Re: Fast KV Compaction via Attention Matching
#15Considering the insanity of the AI arms race going on now, and the incredible sums of money be thrown at any slight advantage, is there any reason to believe that any meaningful AI breakthrough would be openly published for anyone to leverage?
These folks are MIT, so citations are valuable to them. Citations convert into prestige, academic career progression, or a favorable exit from academia into industry. Also, I don't see why you couldn't patent this if you wanted to monetize it.
> Also, I don't see why you couldn't patent this if you wanted to monetize it.
We all just saw the prior art published for the public. That will preclude patenting this work. Further reduction to practice is required.
(I am not a lawyer).
Re: Fast KV Compaction via Attention Matching
#16Considering the insanity of the AI arms race going on now, and the incredible sums of money be thrown at any slight advantage, is there any reason to believe that any meaningful AI breakthrough would be openly published for anyone to leverage?
I know the frontier “labs” are holding back publications.
I don’t think it will last among researchers who think beyond production LLMs
Re: Fast KV Compaction via Attention Matching
#17Considering the insanity of the AI arms race going on now, and the incredible sums of money be thrown at any slight advantage, is there any reason to believe that any meaningful AI breakthrough would be openly published for anyone to leverage?
Yes there is. Lots of researchers are more interested in making a contribution to societal flourishing than in making incredible sums of money. That’s why there’s still lots of top AI researchers in academia.