171–180 of 1001 posts
Facebook has a terrible reputation, and if they can open source their model, it would transform their reputation at least among techies.
This is easily among the rare highest quality articles/comments I've read in the past weeks, perhaps months (on LLMs/AI since that's what I am particularly interested in). And this was for internal consumption before it was made public. Reinforces my recent impression that so much that's being made for public consumption now is shallow and it is hard to find the good stuff. And sadly, increasing so even on HN. As I w…
The memo sounds like spin because it is. The surface argument is equivalent to arguing that no one could sell closed source software because open source exists, and that open source must also be commodotized (oddly, Apple and Microsoft are doing just fine). The implied argument is that Google Research was doing fine giving away their trade secrets and giving negative value to Google because it was going to happen any…
I disagree with this. It's too expensive to train high quality models. For example I don't see how anyone would make an open-source GPT4 unless OpenAI leaks their model to the public.
ELI5 How is it too expensive? I know ChatGPT was expensive to train but Vicuna-13b is said to have cost $300 to train [ https://lmsys.org/blog/2023-03-30-vicuna/ ]
Some snippets for folks who come just for the comments: > While our models still hold a slight edge in terms of quality, the gap is closing astonishingly quickly. Open-source models are faster, more customizable, more private, and pound-for-pound more capable. They are doing things with $100 and 13B params that we struggle with at $10M and 540B. And they are doing so in weeks, not months. > A tremendous outpouring of…
One interesting related point to this is Zuck's comments on Meta's AI strategy during their earnings call: https://www.reddit.com/r/MachineLearning/comments/1373nhq/di...
Summary:
""" Some noteworthy quotes that signal the thought process at Meta FAIR and more broadly
We’re just playing a different game on the infrastructure than companies like Google or Microsoft or Amazon
We would aspire to and hope to make even more open than that. So, we’ll need to figure out a way to do that.
...lead us to do more work in terms of open sourcing, some of the lower level models and tools
Open sourcing low level tools make the way we run all this infrastructure more efficient over time.
On PyTorch: It’s generally been very valuable for us to provide that because now all of the best developers across the industry are using tools that we’re also using internally.
I would expect us to be pushing and helping to build out an open ecosystem.The part of the post that resonates for me is that working with the open source community may allow a model to improve faster. And, whichever model improves faster, will win - if it can continue that pace of improvement. The author talks about Koala but notes that ChatGPT is better. GPT-4 is then significantly better than GPT-3.5. If you've used all the models and can afford to spend money, you'd be insane to not use…
Some snippets for folks who come just for the comments: > While our models still hold a slight edge in terms of quality, the gap is closing astonishingly quickly. Open-source models are faster, more customizable, more private, and pound-for-pound more capable. They are doing things with $100 and 13B params that we struggle with at $10M and 540B. And they are doing so in weeks, not months. > A tremendous outpouring of…
Meta's leaked model isn't open-source. I can found a business using Linux, that's open-source. The LLM piracy community are unpaid FB employees; it is not legal for anyone but Meta to use the results of their labor. I know this might be hard news but it needs to be said... if you want to put your time into working on open source LLMs, you need to get behind something you have a real (and yes, open source) license for…
> Giant models are slowing us down. In the long run, the best models are the ones which can be iterated upon quickly. We should make small variants more than an afterthought, now that we know what is possible in the Maybe this is true for the median query/conversation that people are having with these agents - but it certainly has not been what I have observed in my experience in technical/research work. GPT-4 is leg…
So I use ChatGPT every day. I like it a lot and it is useful but it is overhyped. Also from 3.5 to 4 the jump was nice but seemed relatively marginal to me. I think the head start OpenAi has will vanish. Iteration will be slow and painful giving google or whoever more than enough time to catch up. ChatGPT was a fantastic leap getting us say 80% to Agi but as we have seen time and time again the last 20% are excruciat…