11–20 of 28 posts
Ooh this looks really neat! I'd love to see more content in the future on Structured outputs/Guided generation and sampling. Another great reference on inference-time algorithms for sampling is here: https://rentry.co/samplers
It's a really beautiful project, and I’d like to ask something purely out of curiosity and with the best intentions. What’s the name of the design trend you used for your website? I really loved the website too.
[0] font-family: -apple-system, BlinkMacSystemFont, "Segoe UI", Roboto, Helvetica, Arial, sans-serif;
Hi everyone. I'm one of the maintainers of this project. We're both excited and humbled to see it on Hacker News! We created this handbook to make LLM inference concepts more accessible, especially for developers building real-world LLM applications. The goal is to pull together scattered knowledge into something clear, practical, and easy to build on. We’re continuing to improve it, so feedback is very welcome! GitH…
Hi everyone. I'm one of the maintainers of this project. We're both excited and humbled to see it on Hacker News! We created this handbook to make LLM inference concepts more accessible, especially for developers building real-world LLM applications. The goal is to pull together scattered knowledge into something clear, practical, and easy to build on. We’re continuing to improve it, so feedback is very welcome! GitH…
I have a question. In https://github.com/bentoml/llm-inference-in-production/blob/..., you have a single picture that defines TTFT and ITL. That does not match my understanding (but you guys know probably more than me): In the graphic, it looks like that the model is generating 4 tokens T0 to T3, before outputting a single output token.
I'd have expected that picture for ITL (except that then the labeling of the last box is off), but for TTFT, I'd have expected that there's only a single token T0 from the decode step, that then immediately is handed to detokenization and arrives as first output token (if we assume a streaming setup, otherwise measuring TTFT makes little sense).
Running them well is very important too. As we get to grips with everything models can do and look to deploy them widely knowledge of how to best run them becomes ever more important.
Hi everyone. I'm one of the maintainers of this project. We're both excited and humbled to see it on Hacker News! We created this handbook to make LLM inference concepts more accessible, especially for developers building real-world LLM applications. The goal is to pull together scattered knowledge into something clear, practical, and easy to build on. We’re continuing to improve it, so feedback is very welcome! GitH…
I'm not going to open an issue on this, but you should consider expanding on the self-hosting part of the handbook and explicitly recommend llama.cpp for local self-hosted inference.
Hi everyone. I'm one of the maintainers of this project. We're both excited and humbled to see it on Hacker News! We created this handbook to make LLM inference concepts more accessible, especially for developers building real-world LLM applications. The goal is to pull together scattered knowledge into something clear, practical, and easy to build on. We’re continuing to improve it, so feedback is very welcome! GitH…
At the very least, the sections should be a single page each.
Earlier quoted context omitted.
I'm not going to open an issue on this, but you should consider expanding on the self-hosting part of the handbook and explicitly recommend llama.cpp for local self-hosted inference.
The self hosting section covers corporate use case using vLlm and sglang as well as personal desktop use using Ollama which is a wrapper over llama.cpp.
Earlier quoted context omitted.
The self hosting section covers corporate use case using vLlm and sglang as well as personal desktop use using Ollama which is a wrapper over llama.cpp.
Recommending Ollama isn't useful for end users, its just a trap in a nice looking wrapper.
Earlier quoted context omitted.
Recommending Ollama isn't useful for end users, its just a trap in a nice looking wrapper.
Strong disagree on this. Ollama is great for moderately technical users who aren't really programmers or proficient with the command line.
When you get a model offered by Ollama's service, you have no clue what you're getting, and normal people who have no experience aren't even aware of this.
Ollama is an unrestricted footgun because of this.
