281–290 of 379 posts
I work at Google on these systems everyday (caveat this is my own words not my employers)). So I simultaneously can tell you that its smart people really thinking about every facet of the problem, and I can't tell you much more than that. However I can share this written by my colleagues! You'll find great explanations about accelerator architectures and the considerations made to make things fast. https://jax-ml.git…
Earlier quoted context omitted.
Same explanation but with less mysticism: Inference is (mostly) stateless. So unlike training where you need to have memory coherence over something like 100k machines and somehow avoid the certainty of machine failure, you just need to route mostly small amounts of data to a bunch of big machines. I don't know what the specs of their inference machines are, but where I worked the machines research used were all 8gpu…
> Inference is (mostly) stateless. ... you just need to route mostly small amounts of data to a bunch of big machines. I think this might just be the key insight. The key advantage of doing batched inference at a huge scale is that once you maximize parallelism and sharding, your model parameters and the memory bandwidth associated with them are essentially free (since at any given moment they're being shared among a…
One clever ingredient in OpenAI's secret sauce is billions of dollars of losses. About $5 billion dollars lost in 2024. https://www.cnbc.com/2024/09/27/openai-sees-5-billion-loss-t...
they would be break-even if all they did was serve existing models and got rid of everything related to R&D
Earlier quoted context omitted.
But they’re ASICs so any big architecture changes will be painful for them right?
TPUs are accelerators that accelerate the common operations found in neural nets. A big part is simply a massive number of matrix FMA units to process enormous matrix operations, which comprises the bulk of doing a forward pass through a model. Caching enhancements and massively growing memory was necessary to facilitate transformers, but on the hardware side not a huge amount has changed and the fundamentals from ye…
Earlier quoted context omitted.
Yes. Google is probably gonna win the LLM game tbh. They had a massive head start with TPUs which are very energy efficient compared to Nvidia Cards.
Hasn’t the Inferentia chip been around long enough to make the same argument? AWS and Google probably have the same order of magnitude of their own custom chips
Earlier quoted context omitted.
So people find more value than they thought so they'll just up the price. Meanwhile, they still make more money per inference than they lose.
Currently, they lose more money per inference than they make for Pro subscriptions, because they are essentially renting out their service each month instead of charging for usage (per token).
Earlier quoted context omitted.
Even is the AI bubble does not pops, your prediction about those servers being available on ebay in 10 years will likely be true, because some datacenters will simply upgrade their hardware and resell their old ones to third parties.
Someone's take on AI was that we're collectively investing billions in data centers that will be utterly worthless in 10 years. Unlike the investments in railways or telephone cables or roads or any other sort of architecture, this investment has a very short lifespan. Their point was that whatever your take on AI, the present investment in data centres is a ridiculous waste and will always end up as a huge net loss…
Datacenters could go into the business of making personal PC's or workstations using the older NVIDIA cards and sell them.
Earlier quoted context omitted.
So people find more value than they thought so they'll just up the price. Meanwhile, they still make more money per inference than they lose.
This assumes that the value obtained by customers is high enough to cover any possible actual cost. Many current AI uses are low value things or one time things (for example CV generation, which is killing online hiring).
Many current AI uses are low value things or one time things (for example CV generation, which is killing online hiring).
One trick is that multiple prompts are combined into a batch that is subject to the token prediction simultaneously. I've had Google Gemini start talking to be in the context of someone else's chat, unrelated to mine. ME: Does POSIX have any API for creating a filesystem object (such as a file) with a predetermined inode number supplied by the caller? AI: \documentclass{article} \usepackage{amsmath} \usepackage{amsfo…
> There was no such instruction. I've never chatted with any AI about LaTeX. it leaked from the tokens of someone else's chat. Nope. That's not how it works. Attention doesn't work across multiple independent prompts queued in the same batch. It's not physically possible for the tokens of another chat to leak. What most likely happened is that the model glitched out to the instructions in its (hidden) system prompt,…
https://openai.com/index/march-20-chatgpt-outage/
"We took ChatGPT offline earlier this week due to a bug in an open-source library which allowed some users to see titles from another active user’s chat history. It’s also possible that the first message of a newly-created conversation was visible in someone else’s chat history if both users were active around the same time."
You are probably right about this particular LaTeX issue though.