LLM Inference Handbook
bentoml.com
LLM Inference Handbook
1–10 of 28 posts
Re: LLM Inference Handbook
#2Hi everyone. I'm one of the maintainers of this project. We're both excited and humbled to see it on Hacker News!
We created this handbook to make LLM inference concepts more accessible, especially for developers building real-world LLM applications. The goal is to pull together scattered knowledge into something clear, practical, and easy to build on.
We’re continuing to improve it, so feedback is very welcome!
GitHub repo: https://github.com/bentoml/llm-inference-in-production
Re: LLM Inference Handbook
#3It's a really beautiful project, and I’d like to ask something purely out of curiosity and with the best intentions. What’s the name of the design trend you used for your website? I really loved the website too.
Re: LLM Inference Handbook
#4Hi everyone. I'm one of the maintainers of this project. We're both excited and humbled to see it on Hacker News! We created this handbook to make LLM inference concepts more accessible, especially for developers building real-world LLM applications. The goal is to pull together scattered knowledge into something clear, practical, and easy to build on. We’re continuing to improve it, so feedback is very welcome! GitH…
Amazing work on this, beautifully put together and very useful!
Re: LLM Inference Handbook
#5Very good reference thanks for collating this!
Re: LLM Inference Handbook
#6Ooh this looks really neat! I'd love to see more content in the future on Structured outputs/Guided generation and sampling. Another great reference on inference-time algorithms for sampling is here: https://rentry.co/samplers
Re: LLM Inference Handbook
#7Thanks for putting this together! From now on I only need one link to point interested ones to learn.
Only one suggestion: On page "OpenAI-compatible API" it would be great to have also a simple example for the pure REST call instead of the need to import the OpenAI package.
Re: LLM Inference Handbook
#8[dead]
Re: LLM Inference Handbook
#9If I remember, BentoML was about MLOps, I remember trying it about a year back. Did the company pivot ?
Re: LLM Inference Handbook
#10If I remember, BentoML was about MLOps, I remember trying it about a year back. Did the company pivot ?
There is a big pie in the market around LLM serving. It make sense for a serving framework to extend into the space