[under-the-rug stub] [see https://news.ycombinator.com/item?id=45988611 for explanation]
What an excellent write-up. Thank you!
Prompt caching for cheaper LLM tokens
11–20 of 76 posts
Re: Prompt caching for cheaper LLM tokens
#12Took me a minute to see it is same Ngrok which provided freemium tunnels to localhost. How did they adapt to the AI revolution?
It is the same ngrok!
The product has grown a lot since the mid 2010s. Still got free localhost tunnelling, but we also have a whole bunch of production-grade API gateway tooling and, as of recently, AI gateway stuff too.
Re: Prompt caching for cheaper LLM tokens
#13Re: Prompt caching for cheaper LLM tokens
#14[under-the-rug stub] [see https://news.ycombinator.com/item?id=45988611 for explanation]
Really well done article. I'd note, when I gave the input/output screenshot to ChatGPT 5.2 it failed on it (with lots of colorful chain of thought), though Gemini got it right away.
Huh, when I was writing the article it was GPT-5.1 and I remember it got it no problem.
Re: Prompt caching for cheaper LLM tokens
#15Earlier quoted context omitted.
Thank you so much Yes, I recently wrote https://github.com/samwho/llmwalk and had a similar experience with cache vs no cache. It’s so impactful.
Hopefully you can write the teased next article about how Feedforward and Output layers work. The article was super helpful for me to get better understanding on how LLM GPTs work!
Yeah! It’s planned for sure. It won’t be the direct next one, though. I’m taking a detour into another aspect of LLMs first.
I’m really glad you liked it, and seriously the resources I link at the end are fantastic.
Re: Prompt caching for cheaper LLM tokens
#16[under-the-rug stub] [see https://news.ycombinator.com/item?id=45988611 for explanation]
Amazing article. I was under the misapprehension that temp and other output parameters actually do affect caching. Turns out I was wrong and this explains why beautifully. Great work. Learned a lot!
Yay, glad I could help! The sampling process is so interesting on its own that I really want to do a piece on it as well.
Re: Prompt caching for cheaper LLM tokens
#17This is a surprising good read of how LLM works in general.
It’s funny, I didn’t set out for that to be the case. When I pitched the idea internally, I wanted to scratch my own itch (what on earth is a cached token?) and produce a good post. But then I realised I had to go deeper and deeper to get to my answer and accidentally made a very long explainer.
Re: Prompt caching for cheaper LLM tokens
#18Link seems to be broken: content briefly loads then is replaced with "Something Went Wrong" then "D is not a function". Stays broken with adblock disabled.
Another person had this problem as well and we couldn’t figure out what causes it. We suspect something to do with WebGL support. What browser/device are you using? Does it still break if you disable all extensions? I’d love to fix this.
Re: Prompt caching for cheaper LLM tokens
#19[under-the-rug stub] [see https://news.ycombinator.com/item?id=45988611 for explanation]
Amazing article. I was under the misapprehension that temp and other output parameters actually do affect caching. Turns out I was wrong and this explains why beautifully. Great work. Learned a lot!
I had a “somebody is wrong on the internet!!” discussion about exactly this a few weeks ago, and they proclaimed to be a professor in AI.
Where do people get the idea from that temperature affects caching in any way? Temperature is about next token prediction / output, not input.
Re: Prompt caching for cheaper LLM tokens
#20Blog starts loading and then gives "Something Went Wrong. D is not a function" error displayed