Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

The efficient frontier of LLM inference

baseten.co

Post reply on HN
Print view View original

51–57 of 57 posts

Re: The efficient frontier of LLM inference

#51

Post by naasking »

kgeist wrote: post #36

I'm currently trying to write an inference engine that combines the benefits of llama.cpp (one binary deployment, good support for heterogenous non-datacenter compute, wide quantization support) with the benefits of vLLM/SGlang (things like proper paged attention for better VRAM utilization and high concurrency). Datacenter hardware is expensive and there's shortage of it but llama.cpp is slow/unoptimized for concurr…

Instead of creating your own engine, would it really be that hard to add paged attention to llama.cpp?

Re: The efficient frontier of LLM inference

#52

Post by naasking »

naasking wrote: post #51
kgeist wrote: post #36

I'm currently trying to write an inference engine that combines the benefits of llama.cpp (one binary deployment, good support for heterogenous non-datacenter compute, wide quantization support) with the benefits of vLLM/SGlang (things like proper paged attention for better VRAM utilization and high concurrency). Datacenter hardware is expensive and there's shortage of it but llama.cpp is slow/unoptimized for concurr…

Instead of creating your own engine, would it really be that hard to add paged attention to llama.cpp?

There's actually already a fork that implements the preliminaries:

https://github.com/ggml-org/llama.cpp/discussions/21961

Re: The efficient frontier of LLM inference

#53

Post by fsckboy »

regularfry wrote: post #44
fsckboy wrote: post #19

"the efficient frontier" is an important landmark of (investment) portfolio theory. It proves/explains/illustrates how you can combine selections from a diffuse cloud of individual investments and still land on a frontier that is better than any of your individual choices. It's the entire basis of "diversify your portfolio". The efficient frontier of LLM inference is a line, not a frontier. this is a frontier: https:…

> The efficient frontier of LLM inference is a line, not a frontier. The efficient frontier of portfolio theory is also a line. Not sure I get your point here.

not sure i get why you don't get it. why have the word frontier if it doesn't mean something different than other words?

Re: The efficient frontier of LLM inference

#54

Post by polynomial »

bessbd wrote: post #30
jumploops wrote: post #16

> Speculative decoding is the process of guessing which tokens a model might generate, then validating those guesses. As a computer engineer, it’s always interesting to see optimizations applied at different levels of the stack. Speculative execution became pretty popular in the 90s, eventually used in basically every x86 design. Then in the mid-2000s the Speculator[0] paper brought that concept to distributed system…

> guessing which tokens a model might generate A transformer that generates tokens based on pre-training? We could call it a GPT for short.

Yo dawg I heard you liked transformers

Re: The efficient frontier of LLM inference

#55

Post by renszarv »

am17an wrote: post #45
kgeist wrote: post #36

I'm currently trying to write an inference engine that combines the benefits of llama.cpp (one binary deployment, good support for heterogenous non-datacenter compute, wide quantization support) with the benefits of vLLM/SGlang (things like proper paged attention for better VRAM utilization and high concurrency). Datacenter hardware is expensive and there's shortage of it but llama.cpp is slow/unoptimized for concurr…

I’m trying to do the same!

You are not alone, I want to do the same :D

Re: The efficient frontier of LLM inference

#56

Post by kridsdale1 »

polynomial wrote: post #54
bessbd wrote: post #30

Earlier quoted context omitted.

> guessing which tokens a model might generate A transformer that generates tokens based on pre-training? We could call it a GPT for short.

Yo dawg I heard you liked transformers

RIP Optimus Prime.

Re: The efficient frontier of LLM inference

#57

Post by regularfry »

fsckboy wrote: post #53
regularfry wrote: post #44

Earlier quoted context omitted.

> The efficient frontier of LLM inference is a line, not a frontier. The efficient frontier of portfolio theory is also a line. Not sure I get your point here.

not sure i get why you don't get it. why have the word frontier if it doesn't mean something different than other words?

Because... it's an accurate description of the concept it's being applied to?
Post reply on HN