21–30 of 62 posts
Anything to compete against the cloud AI models and instead run them locally for $0 for free on your own machine and anyone who is able to build such models has a moat such as Meta and is at the finish line in the AI race to zero. At this point it is unstoppable and $0 free local AI models will only accelerate and OpenAI and Google knows it.
Sun, DEC, Microsoft, etc just assumed they would use their cash and market presence to dominate the web up, down, left, and right.
LAMP showed up and ate their lunch. Today open source is the underpinnings for basically every startup over the past 25 years and everything from Android to MacOS to every browser rendering engine. An exclusionary list would be easier.
A 2008 study[0] from the Linux Foundation estimated the development cost of Fedora (at the time) to be almost $11B - in 2008 dollars on a 15 year old codebase. Wonder what it is now?
Point is, OpenAI raising a few billion dollars sounds like a lot of money until you look at the aggregate collective.
FOSS has always had commercial backers, entities with interesting business plans, non-obvious strategy (at the time), etc.
It’s perfectly clear to me why Meta is doing this. It’s also damn near breathtaking to watch the FOSS communities build, build, and build on top of the work of those like Meta as well as each other.
Personally I don’t understand how anyone could think OpenAI, etc vs FOSS is going to be special or unique against what we know to be obvious. I don’t see how this could possibly go any different than the early web. Why, just because it’s “AI” instead of “WWW” this time?
OpenAI will end up like Windows - used by some of the masses and corporate/Enterprise applications, users, etc (think Office integration). Everything else will be a herd (yes, Llama/Alpaca/Vicuna/etc reference.
[0] - https://www.linuxfoundation.org/press/press-release/linux-fo...
Anything to compete against the cloud AI models and instead run them locally for $0 for free on your own machine and anyone who is able to build such models has a moat such as Meta and is at the finish line in the AI race to zero. At this point it is unstoppable and $0 free local AI models will only accelerate and OpenAI and Google knows it.
Let’s not pretend they’re free. Even if you’re heavily optimizing, one of the best local models you can run today is CodeLlama 34B. To run that effectively, you need at least a used RTX 3090: $800 (assuming your computer already has everything else you need to effectively use one). They take 350W by default, but you can power limit them to 200W and preserve most of the LLM performance. If you’re in an average place w…
Earlier quoted context omitted.
Cool, well then we may as well opensource the Columbia-class submarine design!
I for one agree we should not, but also we shouldn't underestimate our opponents and assume that we are or will stay ahead.
I see OSS LLMs along the same lines as the Rolls Royce Nene being sold to the USSR in 1946, or the captured AIM-9 being "the Rosetta Stone" for Soviet engineers as far as missile design.
I mean, will Kremlin or CCP engineers contribute their PRs to OSS LLMs? This seems so utterly naive to me on the part of the OSS religious.[0] And maybe just egoistical on the part of Meta. As in, Meta has to win, no matter what. See: Rohingya genocide.
I am really curious if there are any real AI safely folks at Meta, and what they think of this.
[0] I was once religious about OSS, but then I realized that not all products are the same.
Anything to compete against the cloud AI models and instead run them locally for $0 for free on your own machine and anyone who is able to build such models has a moat such as Meta and is at the finish line in the AI race to zero. At this point it is unstoppable and $0 free local AI models will only accelerate and OpenAI and Google knows it.
Let’s not pretend they’re free. Even if you’re heavily optimizing, one of the best local models you can run today is CodeLlama 34B. To run that effectively, you need at least a used RTX 3090: $800 (assuming your computer already has everything else you need to effectively use one). They take 350W by default, but you can power limit them to 200W and preserve most of the LLM performance. If you’re in an average place w…
Obviously we all know now that it was well worth it, this is how we got our start, and likely why we’re on HN today.
That RTX 3090 can also run thousands upon thousands of other models for any number of use cases at the same cap and op ex. It can enable and build anything from a sense of wonder to an extremely valuable, fulfilling, and long lasting career. Worst case it’s fun, a relatively cheap hobby. Speaking of fun it also plays games.
The RTX 3090 is an investment, not a cost.
That said, some fair points generally but:
1) 25 hours of use is A LOT. GPUs scale power consumption too. The GPU will use slightly higher than idle power to keep the model in memory, then spike to whatever your power limit is while the model is executing. Let’s say 30 seconds to generate as worst case. Human reads that, comprehends it, does something. Meanwhile the power scales back down until you hit it again. Point is, 25 hours is 3,000 30 second generation sessions - 100/day, every day. For $1 (plus base/idle load of machine). You can also suspend the machine or turn it off to cut down on idle-idle waste.
2) Are you doing any optimization? I mostly work on high scale and inference serving frameworks (not on 3090s) but if I had to guess with something like lmdeploy, AWQ int4, int8 kv cache, etc you should be far exceeding ChatGPT in tokens/s. For example, I do concurrency testing with 13b on the RTX 4090 in my workstation. It hits 115 tokens/s - beating the 15 tokens/s or so from ChatGPT should be pretty straightforward with 34b on an RTX 3090.
In terms of making something that could beat a turing test the 65B/70B llama1/2 already are better than openai's currently offered models like gpt3.5-turbo or even the currently available output from gpt3.5 text-davinci-003. They've been so heavily "aligned" that they're insufferable and "As a large language model," everything even given an extensive pre-prompt in text completion mode. It wasn't always this way. Back…
Anything to compete against the cloud AI models and instead run them locally for $0 for free on your own machine and anyone who is able to build such models has a moat such as Meta and is at the finish line in the AI race to zero. At this point it is unstoppable and $0 free local AI models will only accelerate and OpenAI and Google knows it.
It’s the early web all over again. Sun, DEC, Microsoft, etc just assumed they would use their cash and market presence to dominate the web up, down, left, and right. LAMP showed up and ate their lunch. Today open source is the underpinnings for basically every startup over the past 25 years and everything from Android to MacOS to every browser rendering engine. An exclusionary list would be easier. A 2008 study[0] fr…
APIs... it's all about APIs and ease of use. Companies have data centers or could have data centers if they wanted, but a lot use AWS, MSFT, GCP, and friends.
In terms of making something that could beat a turing test the 65B/70B llama1/2 already are better than openai's currently offered models like gpt3.5-turbo or even the currently available output from gpt3.5 text-davinci-003. They've been so heavily "aligned" that they're insufferable and "As a large language model," everything even given an extensive pre-prompt in text completion mode. It wasn't always this way. Back…
Call me when it can compete with GPT-4. It continues to astound me.
Earlier quoted context omitted.
It’s the early web all over again. Sun, DEC, Microsoft, etc just assumed they would use their cash and market presence to dominate the web up, down, left, and right. LAMP showed up and ate their lunch. Today open source is the underpinnings for basically every startup over the past 25 years and everything from Android to MacOS to every browser rendering engine. An exclusionary list would be easier. A 2008 study[0] fr…
> Personally I don’t understand how anyone could think OpenAI, etc vs FOSS is going to be special or unique against what we know to be obvious. APIs... it's all about APIs and ease of use. Companies have data centers or could have data centers if they wanted, but a lot use AWS, MSFT, GCP, and friends.
Let’s say I’m a law firm or a medical group. Do I need ChatGPT to write me a bedtime story or tell me about the Great Chicago Fire, or write code?
No, I need it to help me write my briefs, summarize patient notes, whatever.
The next crop of startups will be (and already are) LegalGPT and ScribeGPT (or whatever) with fine tuned highly specific, highly curated, regulatory compliant, quality controlled, decision controlled, PHI-scrubbed, tuned on your own data, etc, AI solutions largely based on open foundational models, open serving frameworks, data pipelines, etc. Available via end-user interface and/or API.
Then watch LegalGPT be acquired by LegalZoom, LexisNexus, whoever. ScribeGPT by Epic, etc. or vice-versa.
ChatGPT is jack of all trades, truly master of none. As I said it will always have its place but it will recede more and more.
AI is trying to catch up to human intelligence. Do you go to a mechanic when you should be going to a dentist?