What Facebook did to the community and their leaked torrent accelerated everything.
Google “We have no moat, and neither does OpenAI”
121–130 of 1001 posts
Re: Google “We have no moat, and neither does OpenAI”
#122Some snippets for folks who come just for the comments: > While our models still hold a slight edge in terms of quality, the gap is closing astonishingly quickly. Open-source models are faster, more customizable, more private, and pound-for-pound more capable. They are doing things with $100 and 13B params that we struggle with at $10M and 540B. And they are doing so in weeks, not months. > A tremendous outpouring of…
Meta's leaked model isn't open-source. I can found a business using Linux, that's open-source. The LLM piracy community are unpaid FB employees; it is not legal for anyone but Meta to use the results of their labor. I know this might be hard news but it needs to be said... if you want to put your time into working on open source LLMs, you need to get behind something you have a real (and yes, open source) license for…
Most of the code isn't specific to a model. It happens that LLaMA is approximately the best LLM currently available to the public to run on their own hardware, so that's what people are doing. But as soon as anyone publishes a better one, people will use that, using largely the same code, and there is no reason it couldn't be open source.
I'm also curious what the copyright status of these models even is, given the "algorithmic output isn't copyrightable" thing and that the models themselves are essentially the algorithmic output of a machine learning algorithm on third party data. What right does Meta have to impose restrictions on the use of that data against people who downloaded it from The Pirate Bay? Wouldn't it be the same model if someone just ran the same algorithm on the same public data?
(Not that that isn't an impediment to people who don't want to risk the legal expenses of setting a precedent, which models explicitly in the public domain would resolve.)
Re: Google “We have no moat, and neither does OpenAI”
#123Some snippets for folks who come just for the comments: > While our models still hold a slight edge in terms of quality, the gap is closing astonishingly quickly. Open-source models are faster, more customizable, more private, and pound-for-pound more capable. They are doing things with $100 and 13B params that we struggle with at $10M and 540B. And they are doing so in weeks, not months. > A tremendous outpouring of…
Meta's leaked model isn't open-source. I can found a business using Linux, that's open-source. The LLM piracy community are unpaid FB employees; it is not legal for anyone but Meta to use the results of their labor. I know this might be hard news but it needs to be said... if you want to put your time into working on open source LLMs, you need to get behind something you have a real (and yes, open source) license for…
LLaMA leaked intentionally?
Re: Google “We have no moat, and neither does OpenAI”
#124Some snippets for folks who come just for the comments: > While our models still hold a slight edge in terms of quality, the gap is closing astonishingly quickly. Open-source models are faster, more customizable, more private, and pound-for-pound more capable. They are doing things with $100 and 13B params that we struggle with at $10M and 540B. And they are doing so in weeks, not months. > A tremendous outpouring of…
Meta's leaked model isn't open-source. I can found a business using Linux, that's open-source. The LLM piracy community are unpaid FB employees; it is not legal for anyone but Meta to use the results of their labor. I know this might be hard news but it needs to be said... if you want to put your time into working on open source LLMs, you need to get behind something you have a real (and yes, open source) license for…
this is a temporary state. Open source alternatives are already available and more are being trained.
Re: Google “We have no moat, and neither does OpenAI”
#125The part of the post that resonates for me is that working with the open source community may allow a model to improve faster. And, whichever model improves faster, will win - if it can continue that pace of improvement. The author talks about Koala but notes that ChatGPT is better. GPT-4 is then significantly better than GPT-3.5. If you've used all the models and can afford to spend money, you'd be insane to not use…
> Linux won in servers and supercomputing, but not in end user computing "End user computing" these days means mobile, and mobile is dominated by Linux (in Apple's case BSD, but we're splitting hair) and Chrome/WebKit - which began as KHTML. The only area where opensource failed is the desktop, and that's also because of Microsoft's skill in defending their moats.
The kernel isn't the OS/environment. Distiling iOS to BSD is just not useful in the context of this discussion.
Re: Google “We have no moat, and neither does OpenAI”
#126This has been my speculation about the people pushing for regulation in this space: it’s an attempt at regulatory capture because there really is little moat with this tech.
I can already run GPT-3 comparable models on a MacBook Pro. GPT-4 level models that can run on at least higher end commodity hardware seem close.
Models trained on data scraped from the net may not be defensible via copyright and they certainly are not patentable. It also seems possible to “pirate” models by training a model on another model. Defending against this or even detecting it would be as hard as preventing web scraping.
Lastly the adaptive nature of the tech makes it hard to achieve lock in via API compatibility. Just tell the model to talk a different way. The rigidity of classical von Neumann computing that facilitates lock in just isn’t there.
So that leaves the old fashioned way: frighten and bribe the government into creating onerous regulations that you can comply with but upstarts cannot. Or worse make the tech require a permit that is expensive and difficult to obtain.
Re: Google “We have no moat, and neither does OpenAI”
#127Some snippets for folks who come just for the comments: > While our models still hold a slight edge in terms of quality, the gap is closing astonishingly quickly. Open-source models are faster, more customizable, more private, and pound-for-pound more capable. They are doing things with $100 and 13B params that we struggle with at $10M and 540B. And they are doing so in weeks, not months. > A tremendous outpouring of…
Meta's leaked model isn't open-source. I can found a business using Linux, that's open-source. The LLM piracy community are unpaid FB employees; it is not legal for anyone but Meta to use the results of their labor. I know this might be hard news but it needs to be said... if you want to put your time into working on open source LLMs, you need to get behind something you have a real (and yes, open source) license for…
Meh, you can experiment on it for personal use as much as you want and that's all what's needed in this short period of time before powerful, open base models start appearing like mushrooms at which point the whole thing is going to be moot.
Re: Google “We have no moat, and neither does OpenAI”
#128Wow, an open source gift economy beating the closed-source capitalistic model? You don't say.
Wikipedia handily beat Britannica (the most well-known and prestigious encyclopedia, sold door to door) and Encarta (supported by Microsoft)
The Web beat AOL, CompuServe, MSN, newspapers, magazines, radio and TV stations, etc.
Linux beat closed source competitors on tons of environments
Apache and NGinX beat Microsoft Internet Information Server and whatever else proprietary servers.
About the only place it doesn't beat, is consumer-facing frontends. Because open-source does take skill to use and maintain. But that's why the second layer (sysadmins, etc.) have chosen it.
Re: Google “We have no moat, and neither does OpenAI”
#129Some snippets for folks who come just for the comments: > While our models still hold a slight edge in terms of quality, the gap is closing astonishingly quickly. Open-source models are faster, more customizable, more private, and pound-for-pound more capable. They are doing things with $100 and 13B params that we struggle with at $10M and 540B. And they are doing so in weeks, not months. > A tremendous outpouring of…
> the one clear winner in all of this is Meta. Because the leaked model was theirs, they have effectively garnered an entire planet’s worth of free labor. Since most open source innovation is happening on top of their architecture, there is nothing stopping them from directly incorporating it into their products.
This
Re: Google “We have no moat, and neither does OpenAI”
#130So I use ChatGPT every day. I like it a lot and it is useful but it is overhyped. Also from 3.5 to 4 the jump was nice but seemed relatively marginal to me. I think the head start OpenAi has will vanish. Iteration will be slow and painful giving google or whoever more than enough time to catch up. ChatGPT was a fantastic leap getting us say 80% to Agi but as we have seen time and time again the last 20% are excruciat…
GPT feels like an upgrade from MapQuest to Garmin.
Garmin was absolutely a better user experience. Less mental load, dynamically updating next steps, etc, etc.
However, both MapQuest and Garmin still got things wrong. Interestingly, with Garmin, the lack of mental load meant people blindly followed directions. When it come something wrong, people would do really stupid stuff.