121–130 of 269 posts
Earlier quoted context omitted.
Wait… what? Can you elaborate.
https://en.wikipedia.org/wiki/Embrace,_extend,_and_extinguis...
Please make more effort next time than to provide me with a Wiki article.
Earlier quoted context omitted.
Wait… what? Can you elaborate.
He’s speculating that Microsoft is nerfing OpenAI / chatGPT to funnel narrow capabilities to silos like CoPilot.
The bit about plugins not having PMF is interesting and possibly flawed. I, like many others, got access to plugins but not the browsing or code interpreter plugins which feel like the bedrock plugins that make the whole offering useful. I think there's also just education that has to happen to teach users how to effectively use other plugins, and the UX isn't really there to help new users figure out what to even do…
PMF meaning "product market fit"? I had to look it up, curious if I found the right thing or not.
> He reiterated his belief in the importance of open source and said that OpenAI was considering open-sourcing GPT-3. Part of the reason they hadn’t open-sourced yet was that he was skeptical of how many individuals and companies would have the capability to host and serve large LLMs. Am I reading this right? "We're not open sourcing GPT-3 because we don't think it would be useful to anyone else"
I think I worded this poorly. What he said was that a lot of people say they want open-source models but they underestimate how hard it is to serve them well. So he wondered how much real benefit would come from open-sourcing them. I think this is reasonable. Giving researchers access is great but for most small companies they're likely better off having a service provider manage inference for them rather than naviga…
“It’s too hard, trust us” doesn’t really make sense in that context. If it is indeed too hard for small orgs to self host then they won’t. Hiding behind the guise of protecting these people by not open sourcing it seems a bit disingenuous.
If you look at their API limits, no serious company can use this to scale up beyond say 10k users. 3500 Reqs per min for gpt3.5 turbo. They have a long way to go to make it usable for the rest of the 95%
Earlier quoted context omitted.
What about Meta’s megabyte? Also nice proposal?
Yes. Solving context length has been tried in hundreds of different approaches, and yet most LLMs are almost identical to the original one from 2017. Just to name a few families of approaches: Sparse Attention, Hierachical Attention, Global-Local Attention,Sliding Window Attention, Locality sensitive hashing Attention, State space model, EMA gated attention.
Notably, human working memory isn't great either. Which begs the question (if the comparison is valid) as to whether that limitation might be fundamental.
> He reiterated his belief in the importance of open source and said that OpenAI was considering open-sourcing GPT-3. Part of the reason they hadn’t open-sourced yet was that he was skeptical of how many individuals and companies would have the capability to host and serve large LLMs. Am I reading this right? "We're not open sourcing GPT-3 because we don't think it would be useful to anyone else"
Also OpenAI: Meta is pissing in our moat, let's drop a hint about open sourcing our shit too!
The roadmap here is completely focused on ChatGPT and GPT-4. I wonder what portion of their resources is still going to other areas (DALL-E, audio/ video processing, etc.)