Prompt caching for cheaper LLM tokens
1–10 of 76 posts
Re: Prompt caching for cheaper LLM tokens
#2[under-the-rug stub] [see https://news.ycombinator.com/item?id=45988611 for explanation]
Thanks for sharing; you clearly spent a lot of time making this easy to digest. I especially like the tokens-to-embedding visualisation.
I recently had some trouble converting a HF transformer I trained with PyTorch to Core ML. I just couldn’t get the KV cache to work, which made it unusably slow after 50 tokens…
Re: Prompt caching for cheaper LLM tokens
#3[under-the-rug stub] [see https://news.ycombinator.com/item?id=45988611 for explanation]
Thanks for sharing; you clearly spent a lot of time making this easy to digest. I especially like the tokens-to-embedding visualisation. I recently had some trouble converting a HF transformer I trained with PyTorch to Core ML. I just couldn’t get the KV cache to work, which made it unusably slow after 50 tokens…
Thank you so much Yes, I recently wrote https://github.com/samwho/llmwalk and had a similar experience with cache vs no cache. It’s so impactful.
Re: Prompt caching for cheaper LLM tokens
#4Link seems to be broken: content briefly loads then is replaced with "Something Went Wrong" then "D is not a function". Stays broken with adblock disabled.
Re: Prompt caching for cheaper LLM tokens
#5This is a surprising good read of how LLM works in general.
Re: Prompt caching for cheaper LLM tokens
#6Earlier quoted context omitted.
Thanks for sharing; you clearly spent a lot of time making this easy to digest. I especially like the tokens-to-embedding visualisation. I recently had some trouble converting a HF transformer I trained with PyTorch to Core ML. I just couldn’t get the KV cache to work, which made it unusably slow after 50 tokens…
Thank you so much Yes, I recently wrote https://github.com/samwho/llmwalk and had a similar experience with cache vs no cache. It’s so impactful.
Hopefully you can write the teased next article about how Feedforward and Output layers work. The article was super helpful for me to get better understanding on how LLM GPTs work!
Re: Prompt caching for cheaper LLM tokens
#7[under-the-rug stub] [see https://news.ycombinator.com/item?id=45988611 for explanation]
Really well done article.
I'd note, when I gave the input/output screenshot to ChatGPT 5.2 it failed on it (with lots of colorful chain of thought), though Gemini got it right away.
Re: Prompt caching for cheaper LLM tokens
#8[under-the-rug stub] [see https://news.ycombinator.com/item?id=45988611 for explanation]
Amazing article. I was under the misapprehension that temp and other output parameters actually do affect caching. Turns out I was wrong and this explains why beautifully.
Great work. Learned a lot!
Re: Prompt caching for cheaper LLM tokens
#9Took me a minute to see it is same Ngrok which provided freemium tunnels to localhost. How did they adapt to the AI revolution?
Re: Prompt caching for cheaper LLM tokens
#10[under-the-rug stub] [see https://news.ycombinator.com/item?id=45988611 for explanation]
[deleted]