The graph titled "Comparison between Colossal-AI and current major open source projects in the same period" has no label on the Y axis, which shows quantities in thousands. WTH?
Y axis are Github stars. They sort of mention it in the preceding paragraph.
Open source solution replicates ChatGPT training process
151–158 of 158 posts
Re: Open source solution replicates ChatGPT training process
#152Earlier quoted context omitted.
I have been using it as a search replacement for most of the past month and only found two subtly wrong answers. This covers legal questions, researching product differences, wiring diagrams, suggesting books to read, correcting misremembered quotes, and about a hundred other tasks. Of course still relying on google in the background, but increasingly rarely, and presuming all the negative commentary we've been seein…
I don't know what you've been searching for that you've only found two subtly wrong answers. It frequently gives me incorrect answers, some of which are subtle and some of which are obvious. It's given me incorrect code, told me about incorrect APIs, explained deep learning concepts incorrectly, given me wrong answers about science-related questions, made up characters wholesale when I asked it about Irish mythology,…
Re: Open source solution replicates ChatGPT training process
#153"hitting 100 million monthly active users 2 months after its launch". I'm deeply suspicious of that number. It came from Similarweb, who track these things through analytics gathered from browser extensions. I trust this article more: https://www.nytimes.com/2023/02/03/technology/chatgpt-openai... "But two months after its debut, ChatGPT has more than 30 million users and gets roughly five million visits a day, two p…
Can someone tell me what the hell they use ChatGPT for? I tried it a few times and it always confidently gave me wrong results to basic things. What is this thing supposedly “disrupting”? Is it really just marketing cranking out metric tons of spam blogs?
Re: Open source solution replicates ChatGPT training process
#154Earlier quoted context omitted.
Can someone tell me what the hell they use ChatGPT for? I tried it a few times and it always confidently gave me wrong results to basic things. What is this thing supposedly “disrupting”? Is it really just marketing cranking out metric tons of spam blogs?
> Can someone tell me what the hell they use ChatGPT for? I play DnD with my friends and I’m usually the dungeon master. I use ChatGPT to help me world build, and flesh out details. Don’t imagine asking ChatGPT what should happen in the next session. More like asking for options for the name and title of a non-player character. Then it writes options, I twist them up, combine them and select the one I like the best.…
Re: Open source solution replicates ChatGPT training process
#155Earlier quoted context omitted.
there is https://github.com/laion-ai/open-assistant being built in the open already. you can contribute too. please also notice that the article you linked is about the text classifier of the frontend and not the LLM itself
That's what I love about this particular AI revolution. The technologies are developed in such a non-siloed manner that open source is able to replicate the largest steps forward in a manner of a year.
but hey, those are the same people who made the dataset (laion5b) for stable diffusion. I have hope.
Re: Open source solution replicates ChatGPT training process
#156> On a single multi-GPUs server, even with the highest-end A100 80GB GPU, PyTorch can only launch ChatGPT based on small models like GPT-L (774M), due to the complexity and memory fragmentation of ChatGPT. Hence, multi-GPUs parallel scaling to 4 or 8 GPUs with PyTorch's DistributedDataParallel (DDP) results in limited performance gains. Where are these numbers coming from? An 80GB A100 GPU is certainly more than capa…
For reference, GPT-NeoX is a 20B parameter model, and it runs on 45 GB of VRAM. On an 80 GB A100 you could probably run a 35B parameter model. Maybe 8 A100 cards to do inference on ChatGPT? Or 32 3090 cards, which would run you under $40k total.
Re: Open source solution replicates ChatGPT training process
#157Earlier quoted context omitted.
Exactly! I'm doing this for German language. It is rare for translators to get correctly how people talk in German and chatGPT is astonishingly good at that task.
Out of curiosity: Have you tried DeepL.com and, if so, how do the two compare?
Re: Open source solution replicates ChatGPT training process
#158How good is the quality of this? BLOOM is a 176B parameter model, but it doesn't seem to compare to GPT-3 (175B parameters) in terms of output quality.
Out of curiosity, how did your measure their respective performances? My understanding is that BLOOM roughly comparable to GPT-3 in performance on most NLP tasks. Were you comparing OpenAI davinci to raw BLOOM by any chance?