Viewing profile — smarterclayton

A good rule of thumb is that a prefill token is about 1/6th the compute cost of decode token, and that you can get about 15k prefill tokens a second on Llama3 8B on a single H100. …

That's a good example - I can at least answer about why it's a difference: different target user. As I understand the Dynamo SDK it is about simplifying and helping someone get sta…

llm-d is intended to be three clean layers: 1. Balance / schedule incoming requests to the right backend 2. Model server replicas that can run on multiple hardware topologies 3. Pr…

We inherit any multi-host support from vLLM, so https://docs.vllm.ai/en/latest/serving/distributed_serving.h... would be the expected path. We plan to publish examples of multi-hos…

Inference is the process of evaluating a model ("inferring" a response to the inputs). LLMs are uniquely difficult to serve because they push the limits on the hardware. The models…

llm-d would make sense if you are running a very large production LLM serving setup - say 5+ full H100 hosts. The aim is to be much more focused than kserve is on exactly the needs…

CPU tracking is provided by the metrics API, which either reads kubelet metrics directly (the original, old, but simplest way), or a metrics adapter that reads the metrics from a t…

Pretty well. Anthropic runs some of Claude inference on GKE and TPU v5e - this talk at the last GCP conference has some details: https://youtu.be/b87I1plPeMg?si=T4XSFUzXG8BwpphR Ec…

There is a lot of work to make the actual infrastructure and lower level management of lots and lots of GPUs/TPUs open as well - my team focuses on making the infrastructure bit at…

As an aside: This principle (always shutdown uncleanly) was a significant point of design discussion in Kubernetes, another one of the projects that adapted lessons learned inside …

From the report: "As part of the evaluation process, on a popular benchmark, HellaSwag (Zellers et al., 2019), we find that an additional hundred finetuning steps on specific websi…

True. The pod’s monotonic and atomic lifecycle across containers is a significant difference, but you can broadly accomplish similar behaviors with an alloc for sharing resources.

I will note that a trend I have observed with recent ML - as we increasingly use accelerators and models correspondingly grow in size, we are returning to a "one machine, one workl…

Also, I should point out that a set of machines hosting TPUs is referred to as a "pod", which is not the same thing as a Kubernetes pod (also referenced in this doc). The term "pod…

Disclaimer: work associated with this team, didn't write or review the blog post Article stated that it was throughput scheduling the pods on the clusters (from unrelated benchmark…

Our bottleneck was serialization of objects to bytes to send to etcd. Etcd cpu should be about 0.01-0.001 control plane CPU, and control plane apiserver CPU has been dominated by s…

That’s fair - as the perpetrator of much of that abstraction I believe that highlighting the type system aspect of this code obscures the real problem we were solving for, which Go…

I have a lot of respect for Kris but in this context, as the person who approved the PR adding the “Object” interface to the code base (and participated in most of the subsequent d…

> kubernetes went ahead and implemented an oop system in golang I don’t think this was ever an objective, can you clarify what you mean by “oop system” and where we implemented it?…

Agreed that is a feature and not a bug. But! The one thing that custom orchestrators can’t do is easily get the benefit of kubelet isolation of containers and resource management. …

In general, the intent here is to leave open room for just that. dependsOn was proposed during the kep review but deferred. But because init containers and regular containers share…

The challenge with a separate attribute is that it is not forward compatible with new features we might add to pods around ordering and lifecycle. If we used a simple boolean, even…

We did that to leave open more complex ordering of both init containers and sidecars (regular containers do not have a restart order). For instance, you might have a service mesh t…