Prompt + Lora? Train an adapter?
Prompt Caching
71–75 of 75 posts
Re: Prompt Caching
#72Earlier quoted context omitted.
You have to store the KV cache, not the tokens. For Gemma 27B (probably slightly larger than Flash), this would be: Size of KV cache = 2 * (num_layers) * (num_kv_heads * dim_head) * seq_length * precision 8-bit Gemma 27B KV cache = 2 * (46) * (16 * 144) * 1e6 * 1 byte ≈ 200 GB Note that this doesn't take further optimizations into account that Google might be using. Formula: https://developer.nvidia.com/blog/masterin…
Is there some easy to understand source / paper about how this caching works?
Re: Prompt Caching
#73Earlier quoted context omitted.
The "cost" is the published price that Google charges for Gemini caching
By this logic the Windows or MS office should have the price slightly higher than an USB sticker
What are you trying to get at, the meaning of "cost"?
When I used the word "cost", I am referring to the price I pay, my cost, for using caching. That number is a known, published figure
Re: Prompt Caching
#74Earlier quoted context omitted.
By this logic the Windows or MS office should have the price slightly higher than an USB sticker
What are you trying to get at, the meaning of "cost"? When I used the word "cost", I am referring to the price I pay, my cost, for using caching. That number is a known, published figure
[flagged]