> On a single multi-GPUs server, even with the highest-end A100 80GB GPU, PyTorch can only launch ChatGPT based on small models like GPT-L (774M), due to the complexity and memory fragmentation of ChatGPT. Hence, multi-GPUs parallel scaling to 4 or 8 GPUs with PyTorch's DistributedDataParallel (DDP) results in limited performance gains. Where are these numbers coming from? An 80GB A100 GPU is certainly more than capa…
> Once that dataset gets replicated, we can start talking about size. Hopefully LAION will deliver. Is LAION starting a community project to rate model outputs? I didn't see anything on their site.
Open source solution replicates ChatGPT training process
91–100 of 158 posts
Re: Open source solution replicates ChatGPT training process
#92Earlier quoted context omitted.
I'm going to echo other people's skepticism and give a concrete example that's easy to reproduce and which has virtually no dependence on real experience in the physical world. Try asking it about public transit wayfinding trivia. Pure text matching, well defined single letter / digit service names, closed system of semantic content. All there is are services and stations and each service is wholly defined by the lis…
I've been spending some time trying to get a sense of how it works by exploring where it fails. When it makes a mistake, you can ask questions in a socratic method until it says the true counterpart to its mistake. It doesn't comment on noticing a discrepancy even if you try to get it to reconcile its previous answer with the corrected version that you guided it to. If you ask specifically about the discrepancy it wi…
> If you ask specifically about the discrepancy it will usually deny the discrepancy entirely or double-down on the mistake.
I have had the exact opposite experience. I pasted error messages from code it generated, I corrected its Latin grammar, and I pointed out contradictions in its factual statements in a variety of ways. Every time, it responded with a correction and (the same) apology.
This makes me wonder if we got different paths in an AB test.
Re: Open source solution replicates ChatGPT training process
#93It would somehow be combined with an open source search engine
I wish I could @marginalia_nu here :)
Re: Open source solution replicates ChatGPT training process
#94"hitting 100 million monthly active users 2 months after its launch". I'm deeply suspicious of that number. It came from Similarweb, who track these things through analytics gathered from browser extensions. I trust this article more: https://www.nytimes.com/2023/02/03/technology/chatgpt-openai... "But two months after its debut, ChatGPT has more than 30 million users and gets roughly five million visits a day, two p…
Can someone tell me what the hell they use ChatGPT for? I tried it a few times and it always confidently gave me wrong results to basic things. What is this thing supposedly “disrupting”? Is it really just marketing cranking out metric tons of spam blogs?
Slightly out of left field - I'm using it to generate Chinese conversations for foreign language students, paired with TTS voices and 3D avatars, streaming live on Twitch.
https://www.twitch.tv/polyvoxlive
It's not 100% flawless, but for this task, it is exceptionally good (in particular, it makes fewer mistakes than the davinci models).
Re: Open source solution replicates ChatGPT training process
#95> On a single multi-GPUs server, even with the highest-end A100 80GB GPU, PyTorch can only launch ChatGPT based on small models like GPT-L (774M), due to the complexity and memory fragmentation of ChatGPT. Hence, multi-GPUs parallel scaling to 4 or 8 GPUs with PyTorch's DistributedDataParallel (DDP) results in limited performance gains. Where are these numbers coming from? An 80GB A100 GPU is certainly more than capa…
I think they are correctly referring to ChatGPT as GPT-3 + RLHF. In other words
ChatGPT = GPT-3 + RLHF. So, 80GB A100 GPU would be required for both GPT-L AND RLHF (PyTorch version). And it looks to me from the TFA that the main thing that takes a lot of space is actually RLHF.
>I don’t understand how they went from talking about 175B params across 32 cards to 774M on one card. 175B divided by 32 is 5.4B.
They claim 774M is the size of GPT-L which if run in conjunction with their RLHF would require 80GB A100 GPU to train (using their RLHF PyTorch implementation). They additionally claim that training GPT-3(175B params) plus RLHF would need 64 * 80gb = 5120gb of memory if using PyTorch implementation of RLHF or 32 * 80gb = 2560gb if going Colossal AI route.
To be honest, these numbers do look to me to be a bit of a cheesy ad for their product but hey they need to put food on their table too. I'm not sure if the dataset would be such a huge problem otherwise Britannica would still be ahead of Wikipedia. Given an army of volunteers willing to produce it OpenAI brigade of contractors has no chance.
Re: Open source solution replicates ChatGPT training process
#96Is cool~ Waiting for the day when I can run a model like this in a native language like Rust, without incurring the overhead of the Python interpreter. Python can be good for trying out methodologies but it's sort of yucky to set up ime.
Re: Open source solution replicates ChatGPT training process
#97Earlier quoted context omitted.
I've been spending some time trying to get a sense of how it works by exploring where it fails. When it makes a mistake, you can ask questions in a socratic method until it says the true counterpart to its mistake. It doesn't comment on noticing a discrepancy even if you try to get it to reconcile its previous answer with the corrected version that you guided it to. If you ask specifically about the discrepancy it wi…
> If you ask specifically about the discrepancy it will usually deny the discrepancy entirely or double-down on the mistake. I have had the exact opposite experience. I pasted error messages from code it generated, I corrected its Latin grammar, and I pointed out contradictions in its factual statements in a variety of ways. Every time, it responded with a correction and (the same) apology. This makes me wonder if we…
How the hell does one A/B test a language model that even the designers don’t fully understand?
Of course, I’m sure that once you start plugging engagement metrics into the model and the model itself conducts A/B tests on its output… hoo boy….
Re: Open source solution replicates ChatGPT training process
#98Is cool~ Waiting for the day when I can run a model like this in a native language like Rust, without incurring the overhead of the Python interpreter. Python can be good for trying out methodologies but it's sort of yucky to set up ime.
The python interpreter is not the bottleneck
Re: Open source solution replicates ChatGPT training process
#99"hitting 100 million monthly active users 2 months after its launch". I'm deeply suspicious of that number. It came from Similarweb, who track these things through analytics gathered from browser extensions. I trust this article more: https://www.nytimes.com/2023/02/03/technology/chatgpt-openai... "But two months after its debut, ChatGPT has more than 30 million users and gets roughly five million visits a day, two p…
Can someone tell me what the hell they use ChatGPT for? I tried it a few times and it always confidently gave me wrong results to basic things. What is this thing supposedly “disrupting”? Is it really just marketing cranking out metric tons of spam blogs?
The only actually useful thing I found for myself so far, was as a helper for TTRPGs. Essentially a random story (encounter, issues, background, anything) generator on steroids.
Re: Open source solution replicates ChatGPT training process
#100"hitting 100 million monthly active users 2 months after its launch". I'm deeply suspicious of that number. It came from Similarweb, who track these things through analytics gathered from browser extensions. I trust this article more: https://www.nytimes.com/2023/02/03/technology/chatgpt-openai... "But two months after its debut, ChatGPT has more than 30 million users and gets roughly five million visits a day, two p…
Can someone tell me what the hell they use ChatGPT for? I tried it a few times and it always confidently gave me wrong results to basic things. What is this thing supposedly “disrupting”? Is it really just marketing cranking out metric tons of spam blogs?
Cover letters. “Respond to requirements with qualifications and specific examples”