1–10 of 379 posts
Sure, they have huge GPU clusters, but there must be more going on - model optimizations, sharding, custom hardware, clever load balancing, etc.
What engineering tricks make this possible at such massive scale while keeping latency low?
Curious to hear insights from people who've built large-scale ML systems.
That's a really, really big "sure."
Almost every trick to run a LLM at OpenAI's scale is a trade secret and may not be easily understood by mere mortals anyways (e.g. bare-metal CUDA optimizations)
Hint: it's a money thing.
However I can share this written by my colleagues! You'll find great explanations about accelerator architectures and the considerations made to make things fast.
https://jax-ml.github.io/scaling-book/
In particular your questions are around inference which is the focus of this chapter https://jax-ml.github.io/scaling-book/inference/
Edit: Another great resource to look at is the unsloth guides. These folks are incredibly good at getting deep into various models and finding optimizations, and they're very good at writing it up. Here's the Gemma 3n guide, and you'll find others as well.
https://docs.unsloth.ai/basics/gemma-3n-how-to-run-and-fine-...
1. prompt caching
2. some RAG to save resources
3. of course lots model optimizations and CUDA optimizations
4. lots of throttling
5. offloading parts of the answer that are better served by other approaches (if asked to add numbers, do a system call to a calculator instead of using LLM)
6. a lot of sharding
One thing you should ask is: What does it mean to handle a request with chatgpt? It might not be what you think it is.
source: random workshops over the past year.
They are also partnering with rivals like Google for additional capacity https://www.reuters.com/business/retail-consumer/openai-taps...