SEQUOIA: Exact Llama2-70B on an RTX4090 with half-second per-token latency
infini-ai-lab.github.io
SEQUOIA: Exact Llama2-70B on an RTX4090 with half-second per-token latency
1–10 of 65 posts
Re: SEQUOIA: Exact Llama2-70B on an RTX4090 with half-second per-token latency
#2So this is 8x faster for serving these models than before? Or is this about it being more deterministic? I can't quite tell from reading it.
Re: SEQUOIA: Exact Llama2-70B on an RTX4090 with half-second per-token latency
#3Someone get this into koboldcpp
Re: SEQUOIA: Exact Llama2-70B on an RTX4090 with half-second per-token latency
#4this is quite worrying for OpenAI as the rate token prices have been plummeting thanks to Meta and its going to have to keep cutting its prices while capex remains flat. whatever Sam says in interviews just think the opposite and the whole picture comes together.
It's almost a mathematical certainty that people who invested in OpenAI will need to reincarnate in multiple universes to ever see that money again but no bother many are probably NVIDIA stock holders to even out the damage.
Re: SEQUOIA: Exact Llama2-70B on an RTX4090 with half-second per-token latency
#5So this is 8x faster for serving these models than before? Or is this about it being more deterministic? I can't quite tell from reading it.
The idea is to serve models that would normally be considered too large for GPU memory (70 billion parameters at 16 bytes each for 140 GB of memory required). Some people figured out you can offload the model and only have parts of it loaded so a 24 GB GPU like the 4090 can still serve the model, but it goes a lot slower. They have a new way to serve the same model on the same GPU but 8x better throughput. Something about decoding tokens on a smaller model maybe, then just checking multiple tokens on the larger model in a single batch. Magic, but ultimately its the same model, same GPU, same output as before, but much better throughout.
Re: SEQUOIA: Exact Llama2-70B on an RTX4090 with half-second per-token latency
#6this is quite worrying for OpenAI as the rate token prices have been plummeting thanks to Meta and its going to have to keep cutting its prices while capex remains flat. whatever Sam says in interviews just think the opposite and the whole picture comes together. It's almost a mathematical certainty that people who invested in OpenAI will need to reincarnate in multiple universes to ever see that money again but no b…
I suspect that when it costs 0.5c per 100 million generated token, and you can generate 1000 tokens per second, they'll be very happy.
Re: SEQUOIA: Exact Llama2-70B on an RTX4090 with half-second per-token latency
#7this is quite worrying for OpenAI as the rate token prices have been plummeting thanks to Meta and its going to have to keep cutting its prices while capex remains flat. whatever Sam says in interviews just think the opposite and the whole picture comes together. It's almost a mathematical certainty that people who invested in OpenAI will need to reincarnate in multiple universes to ever see that money again but no b…
Isn't that why he's making rounds to lock down the biggest AI's?
Re: SEQUOIA: Exact Llama2-70B on an RTX4090 with half-second per-token latency
#8I'm looking at buying 2 X RTX 3060s to run LLama 70b for my new PC I just purchased.
Will this work, or do I need a Tesla P40 or two?
Re: SEQUOIA: Exact Llama2-70B on an RTX4090 with half-second per-token latency
#9[flagged]
Re: SEQUOIA: Exact Llama2-70B on an RTX4090 with half-second per-token latency
#10I'm looking at buying 2 X RTX 3060s to run LLama 70b for my new PC I just purchased. Will this work, or do I need a Tesla P40 or two?
Theoretically there's no reason why this shouldn't work, but you likely will find the software isn't designed for multi-GPU and have to reimplement/fix things yourself.
You will also be getting about 720GB/s of memory bandwidth with 2x3060; instead of 1TB/s with the 4090; so expect lower performance.