Other than portability and privacy, are there any benefits to running a local model with a 4090, versus running the same model on-demand on a cloud service with the same or more powerful card?
SEQUOIA: Exact Llama2-70B on an RTX4090 with half-second per-token latency
11–20 of 65 posts
Re: SEQUOIA: Exact Llama2-70B on an RTX4090 with half-second per-token latency
#12this is quite worrying for OpenAI as the rate token prices have been plummeting thanks to Meta and its going to have to keep cutting its prices while capex remains flat. whatever Sam says in interviews just think the opposite and the whole picture comes together. It's almost a mathematical certainty that people who invested in OpenAI will need to reincarnate in multiple universes to ever see that money again but no b…
I agree with you somewhat. You are correct unless they have a much better GPT model that have not released for whatever reason. They are a year ahead than competitors and GPT4 is pretty old now. I find it hard to believe they don’t have much more capable models now. We Will see though
Re: SEQUOIA: Exact Llama2-70B on an RTX4090 with half-second per-token latency
#13this is quite worrying for OpenAI as the rate token prices have been plummeting thanks to Meta and its going to have to keep cutting its prices while capex remains flat. whatever Sam says in interviews just think the opposite and the whole picture comes together. It's almost a mathematical certainty that people who invested in OpenAI will need to reincarnate in multiple universes to ever see that money again but no b…
There’s a Pareto frontier where Meta is pushing out the boundaries along the “private” and “cheap” axes.
Open AI can release GPT 4.5 or 5 and push out the boundary in the direction of “correctness” and “multimodality”.
Either way, we win as customers while the the level of competition remains this hot.
I personally want a smart AI much more than a cheap or fast one. Your mileage may vary.
Re: SEQUOIA: Exact Llama2-70B on an RTX4090 with half-second per-token latency
#14this is quite worrying for OpenAI as the rate token prices have been plummeting thanks to Meta and its going to have to keep cutting its prices while capex remains flat. whatever Sam says in interviews just think the opposite and the whole picture comes together. It's almost a mathematical certainty that people who invested in OpenAI will need to reincarnate in multiple universes to ever see that money again but no b…
I agree with you somewhat. You are correct unless they have a much better GPT model that have not released for whatever reason. They are a year ahead than competitors and GPT4 is pretty old now. I find it hard to believe they don’t have much more capable models now. We Will see though
There's wide speculation that what will be branded as either GPT-4.5 or GPT-5 has finished pretraining now and is undergoing internal testing for a fairly near-term release.
Re: SEQUOIA: Exact Llama2-70B on an RTX4090 with half-second per-token latency
#15this is quite worrying for OpenAI as the rate token prices have been plummeting thanks to Meta and its going to have to keep cutting its prices while capex remains flat. whatever Sam says in interviews just think the opposite and the whole picture comes together. It's almost a mathematical certainty that people who invested in OpenAI will need to reincarnate in multiple universes to ever see that money again but no b…
Most of the big "investments" in OpenAI are in the form of compute credits. I fail to see the downside of that.
Re: SEQUOIA: Exact Llama2-70B on an RTX4090 with half-second per-token latency
#16Earlier quoted context omitted.
I agree with you somewhat. You are correct unless they have a much better GPT model that have not released for whatever reason. They are a year ahead than competitors and GPT4 is pretty old now. I find it hard to believe they don’t have much more capable models now. We Will see though
There's wide speculation that what will be branded as either GPT-4.5 or GPT-5 has finished pretraining now and is undergoing internal testing for a fairly near-term release.
My speculation is that internally they have much stronger models like Q* but they won’t be able to release them to public even if they want to for lack of compute and safety and other reasons they see probably…
Re: SEQUOIA: Exact Llama2-70B on an RTX4090 with half-second per-token latency
#17Other than portability and privacy, are there any benefits to running a local model with a 4090, versus running the same model on-demand on a cloud service with the same or more powerful card?
There are always going to be pros and cons. That's why solutions like managed databases are reality. From an expert perspective it seems like there's more to lose but from the perspective of a company with employee turn over, possible data loss, security etc. the benefits start to far outweigh the costs.
This reasoning can mostly be applied here. If you want to learn about and pull the LLM apart. Perhaps fine-tune and tinker then 100% go ahead running locally. You however won't be able to scale this up easily for a consumer base and the electricity use and heat output starts to become a problem.
At some point it's more beneficial to pay the provider for inference, this includes upkeep, latest models, faster generation, stability, hosting etc.
Pros and cons! Choice is important and Meta is doing the right thing by the AI community and tech community in general by being realistic with these programs. The ecosystem is giving back by being able to access these high quality models.
Re: SEQUOIA: Exact Llama2-70B on an RTX4090 with half-second per-token latency
#18Earlier quoted context omitted.
There's wide speculation that what will be branded as either GPT-4.5 or GPT-5 has finished pretraining now and is undergoing internal testing for a fairly near-term release.
My speculation is that internally they have much stronger models like Q* but they won’t be able to release them to public even if they want to for lack of compute and safety and other reasons they see probably…
They don't actually care about safety, that's a lie, so compute and business strategy is the only thing stopping them.
SoRA is the same. It's not ready and it's too slow.
Re: SEQUOIA: Exact Llama2-70B on an RTX4090 with half-second per-token latency
#19Other than portability and privacy, are there any benefits to running a local model with a 4090, versus running the same model on-demand on a cloud service with the same or more powerful card?
Guaranteed uptime.
you are the guarantor but that's good enough.
Re: SEQUOIA: Exact Llama2-70B on an RTX4090 with half-second per-token latency
#20this is quite worrying for OpenAI as the rate token prices have been plummeting thanks to Meta and its going to have to keep cutting its prices while capex remains flat. whatever Sam says in interviews just think the opposite and the whole picture comes together. It's almost a mathematical certainty that people who invested in OpenAI will need to reincarnate in multiple universes to ever see that money again but no b…
Disclaimer: not a fan of "Open"AI
Everyone could say anything about open source models, but they're comparing themselves to what OpenAI released a year ago. They haven't shown all of their cards yet and they have a decent moat already in place; some say they have no moat, I disagree, they have one of the best moats possible which is brand awareness.
Sora on its own could bring in billions in revenue; an open-source Sora will take at least another year, if not two, to come out. Then more time until it can run on commodity hardware. An open source model that only runs in a dedicated H100 is actually less useful than a closed model behind an API call; not to detract from open source, I think it's the way to go but I'm just being pragmatic and realistic. There's a reason why MS Office is still the top productivity app in the world, even though dozens of open source alternatives exist.