421–426 of 426 posts
Renting GPU servers at AWS is just stupid. Did you ever see bitcoin miners calculate the price of mining one bitcoin by looking at renting AWS GPU servers?
Earlier quoted context omitted.
Yep, batching is a feature I really wish the OpenAI API had. That and the ability to intelligently cache frequently used prompts. Much easier to achieve this with a hosted OS model, so I guess it's a speed + customizability/cost tradeoff for the time being.
imo they dont have batching because they pack sequences before passing through the model. so a single sequence in a batch on OpenAI might have requests from multiple customers in it
We just started a service different open source models and with an OpenAI compatible API [1]. The pricing isn't final and we haven't officially launched yet but you should be able to save at least 75% compared to GPT 3.5. [1] https://lemonfox.ai/
Earlier quoted context omitted.
> Microsoft will reportedly get a 75% share of OpenAI's profits until it makes back the money on its investment, after which the company would assume a 49% stake in OpenAI. 49% isn't _just_ a share, it's a significant portion of the company.
It's what we call a non-controlling interest. Definitely not something that qualifies as an acquisition.
Open source always wins, in the end. This is a fluff piece.
Earlier quoted context omitted.
Moore's law is irrelevant. Large language models are going to leave the digital paradigm behind altogether. Neural nets don't need fully precise digital computing. Especially with quantization we're seeing that losing a bit of precision in the weights isn't impactful. Now that we're serving huge foundation models with static weights there's an enormous incentive to develop analog hardware to run them. Mark my words,…
> Just think, how many multiplications per second is the light refracting through your window right now clocking? It really depends where you draw the lines, because you could also say that one single transistor in my electrical CPU is doing a kerjillion calculations for all of the atoms and electrons involved.