Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

SEQUOIA: Exact Llama2-70B on an RTX4090 with half-second per-token latency

infini-ai-lab.github.io

Post reply on HN
Print view View original

41–50 of 65 posts

Re: SEQUOIA: Exact Llama2-70B on an RTX4090 with half-second per-token latency

#41

Post by mft_ »

jiggawatts wrote: post #13
spxneo wrote: post #4

this is quite worrying for OpenAI as the rate token prices have been plummeting thanks to Meta and its going to have to keep cutting its prices while capex remains flat. whatever Sam says in interviews just think the opposite and the whole picture comes together. It's almost a mathematical certainty that people who invested in OpenAI will need to reincarnate in multiple universes to ever see that money again but no b…

There’s a Pareto frontier where Meta is pushing out the boundaries along the “private” and “cheap” axes. Open AI can release GPT 4.5 or 5 and push out the boundary in the direction of “correctness” and “multimodality”. Either way, we win as customers while the the level of competition remains this hot. I personally want a smart AI much more than a cheap or fast one. Your mileage may vary.

Well, Pareto is about optimisation, not either/or. I want a model that’s smart enough, while also being locally-executable.

I don’t know whether/when we’ll get there, and whether it will be improvements in models, or underlying model technology, or GPU/TPUs with larger memory at a consumer price point, or something else, that will deliver it.

Re: SEQUOIA: Exact Llama2-70B on an RTX4090 with half-second per-token latency

#42

Post by ashu1461 »

anon373839 wrote: post #26
jstummbillig wrote: post #22

Earlier quoted context omitted.

I disagree. a) A year after GPT-4 set the bar, it's still the best model, despite everyone else not having to do it first. Just copy, and just software. And that's not for lack of trying by every other viable prime player on the planet with unprecedented acceleration. Imagine any other piece of software, where the incumbent has a mere 2-3 year head start, in which they had to work out the entire product that everyone…

> A year after GPT-4 set the bar, it's still the best model Debatable. Many people find Claude Opus superior, and I know I've found it consistently better for challenging coding questions. More importantly, the delta between GPT-4 and everything else is getting smaller and smaller. Llama 3 is basically interchangeable with GPT-4 for a huge number of tasks, despite its smaller size.

Agree, the delta is getting smaller. And for majority of the tasks you can use the Claude Sonet which is better than 3.5 and also fast.

But at the same time when you actually want to solve a complicated problem, deep down you know that only GPT 4 can crack it.

Re: SEQUOIA: Exact Llama2-70B on an RTX4090 with half-second per-token latency

#43

Post by tarruda »

aussieguy1234 wrote: post #8

I'm looking at buying 2 X RTX 3060s to run LLama 70b for my new PC I just purchased. Will this work, or do I need a Tesla P40 or two?

Note that 2 RTX 3060 will probably be significantly slower than RTX4090.

Even with RTX 4090, 2 tokens per second is very slow and likely not ideal for most tasks. It is impressive (much faster than previous solutions), but still very slow for real time use.

If you want to run Llama 3 70b, might be better to purchase a mac studio with 64gb RAM (more for longer contexts) and run with 4-bit quantization.

My 2 cents: For most common tasks Llama 3 8b will be more than enough, and you can run that with full precision using a single rtx 3090. At a much lower cost, you can also run Llama 3 8b with 8-bit quantization in a single RTX 3060, if it has 12GB RAM.

Re: SEQUOIA: Exact Llama2-70B on an RTX4090 with half-second per-token latency

#44

Post by qrios »

zwaps wrote: post #37

Is it me or is this paper basically missing all technical information? I get that Therese proprietary technology, but if so, can we please not put this on arxiv and pretend it’s a scientific contribution?

The linked github repo [1] seems to have the code available and well documented.

[1] https://github.com/Infini-AI-Lab/Sequoia/tree/main/Engine

Re: SEQUOIA: Exact Llama2-70B on an RTX4090 with half-second per-token latency

#45

Post by hackerlight »

spxneo wrote: post #4

this is quite worrying for OpenAI as the rate token prices have been plummeting thanks to Meta and its going to have to keep cutting its prices while capex remains flat. whatever Sam says in interviews just think the opposite and the whole picture comes together. It's almost a mathematical certainty that people who invested in OpenAI will need to reincarnate in multiple universes to ever see that money again but no b…

Depends how good their next model is, and if they prevent leaks and departures so they can prolong the lead for an undetermined amount of time.

Re: SEQUOIA: Exact Llama2-70B on an RTX4090 with half-second per-token latency

#46

Post by CapeTheory »

Hendrikto wrote: post #39
moralestapia wrote: post #20

Earlier quoted context omitted.

Disclaimer: not a fan of "Open"AI Everyone could say anything about open source models, but they're comparing themselves to what OpenAI released a year ago. They haven't shown all of their cards yet and they have a decent moat already in place; some say they have no moat, I disagree, they have one of the best moats possible which is brand awareness. Sora on its own could bring in billions in revenue; an open-source S…

> they have one of the best moats possible which is brand awareness. Do they though? If you talk to "regular people", everybody knows ChatGPT, but nobody knows or cares about OpenAI. And most of them don‘t even really know that name. They call it ChatUuuuhm, ChatThingy, Chad Gippity, or similar. I think they will just switch, when something better comes along.

Good old Chatty Jeeps

Re: SEQUOIA: Exact Llama2-70B on an RTX4090 with half-second per-token latency

#47

Post by jstummbillig »

ashu1461 wrote: post #42
anon373839 wrote: post #26

Earlier quoted context omitted.

> A year after GPT-4 set the bar, it's still the best model Debatable. Many people find Claude Opus superior, and I know I've found it consistently better for challenging coding questions. More importantly, the delta between GPT-4 and everything else is getting smaller and smaller. Llama 3 is basically interchangeable with GPT-4 for a huge number of tasks, despite its smaller size.

Agree, the delta is getting smaller. And for majority of the tasks you can use the Claude Sonet which is better than 3.5 and also fast. But at the same time when you actually want to solve a complicated problem, deep down you know that only GPT 4 can crack it.

Even more important, you know that GPT-4 will probably also not crack it. Which is why the SOTA is not terribly interesting. The delta between GPT-4 and the competition has been closing but why anyone would assume that this is a trend and that it would continue with GPT-4.5 to competition, or GPT-5 to competition instead of the other way around is a mystery to me.

I am not saying it could not be true. But extrapolating from differences between current bad models to a future with better models is weird, specially when everyone seems to pretty much agree that scale is the difference between the two and scale is hard and exclusive.

Re: SEQUOIA: Exact Llama2-70B on an RTX4090 with half-second per-token latency

#48

Post by 34679 »

aussieguy1234 wrote: post #8

I'm looking at buying 2 X RTX 3060s to run LLama 70b for my new PC I just purchased. Will this work, or do I need a Tesla P40 or two?

I picked up a couple RTX 4060ti in the 16GB version for $450 each a couple days ago from Bestbuy. Had been looking at the 3060 like yourself. Installed LM Studio and have been trying out a bunch of models with varying levels of quantization, completely pain free.

Re: SEQUOIA: Exact Llama2-70B on an RTX4090 with half-second per-token latency

#49

Post by anon373839 »

jstummbillig wrote: post #47
ashu1461 wrote: post #42

Earlier quoted context omitted.

Agree, the delta is getting smaller. And for majority of the tasks you can use the Claude Sonet which is better than 3.5 and also fast. But at the same time when you actually want to solve a complicated problem, deep down you know that only GPT 4 can crack it.

Even more important, you know that GPT-4 will probably also not crack it. Which is why the SOTA is not terribly interesting. The delta between GPT-4 and the competition has been closing but why anyone would assume that this is a trend and that it would continue with GPT-4.5 to competition, or GPT-5 to competition instead of the other way around is a mystery to me. I am not saying it could not be true. But extrapolati…

There’s a scatterplot that’s been circulating on Twitter. The trend lines show that since the time of GPT-2, open weights models have improved at a steeper rate than proprietary models, with the two on a path to intersect.

Re: SEQUOIA: Exact Llama2-70B on an RTX4090 with half-second per-token latency

#50

Post by lightbritefight »

torginus wrote: post #40
michelsedgh wrote: post #16

Earlier quoted context omitted.

My speculation is that internally they have much stronger models like Q* but they won’t be able to release them to public even if they want to for lack of compute and safety and other reasons they see probably…

Honestly I'm pretty puzzled by this mystical fog that hangs over OpenAIs skunkworks projects - don't people leave for other jobs/go to conferences etc.? I'm surprised that nobody call tell what they infact do or do not have.

Truth tends to take the wind out of hypes sails.

With hundreds of billions on the line for the founders and a whole lot of likely unvested stock options for the employees, it doesnt seem like anyone wants to open up about whats actually going on day to day.

Post reply on HN