The graph titled "Comparison between Colossal-AI and current major open source projects in the same period" has no label on the Y axis, which shows quantities in thousands. WTH?
Open source solution replicates ChatGPT training process
121–130 of 158 posts
Re: Open source solution replicates ChatGPT training process
#122Why are gazillions of parameters needed in the first place? From an information perspective it feels that there might be some fundamentally inefficient use of parametric freedom. A brute force approach to combinatorial explosion so to speak. Are there any research efforts that look into how to reduce model complexity (without substantially sacrificing performance obviously).
Re: Open source solution replicates ChatGPT training process
#123"hitting 100 million monthly active users 2 months after its launch". I'm deeply suspicious of that number. It came from Similarweb, who track these things through analytics gathered from browser extensions. I trust this article more: https://www.nytimes.com/2023/02/03/technology/chatgpt-openai... "But two months after its debut, ChatGPT has more than 30 million users and gets roughly five million visits a day, two p…
Can someone tell me what the hell they use ChatGPT for? I tried it a few times and it always confidently gave me wrong results to basic things. What is this thing supposedly “disrupting”? Is it really just marketing cranking out metric tons of spam blogs?
It helps like a very low cost assistent; for instance, people complain about it writing ‘wrong code’. So do most programmers.
The confidently wrong is simply because it is not human (even though many humans do the same); it is in fact not confident; you just attribute that to it because of the language it uses. In fact, it feels nothing confident or otherwise; it just gives you crap and then you can ask it to think about it again.
Like I was asking it something and it told me the answer was 11, then it explained why the answer was 11 but at the end of the explanation it concluded the answer was 12. So I asked what it is 11 or 12; it said 12 and 11 was a mistake. Which was indeed correct. Again, that is not unlike humans at all; humans often say something with confidence and after a big of pondering, they correct themselves. Difference is, humans actually do feel confident or unsure etc.
You write and then you fine tune and fix; that is what it does. But faster and cheaper than humans. You engage in a conversation about the problem; it generates code, you check the code and tell what’s wrong, it will generate again with fixes etc. Especially with boring things like structural transformations aka a large json doc transformed to another large json doc with all kinds of operations in between, chatgpt is simply a lot faster than I would be, even with errors. Just paste a two json doc and ask typescript types and code for both to transform them one to another. Then add operations by tell on which nodes and what they should do. Ask to make jest tests.
Re: Open source solution replicates ChatGPT training process
#124Earlier quoted context omitted.
If someone created a folding@home to crowd train an actually open ChatGPT, I'd gladly donate my spare resources to the cause.
That's unlikely to work. The memory has to be fast with low latency, even switching from on-board VRAM to system RAM slows performance at least 10-100x. The bottleneck isn't computing power it's I/O. Total bus bandwidth of a common small AI cluster is around 1 terabyte per second. We really shouldn't be building an "open source" AI in the first place though, and it's going to be illegal to do so soon. The weaponizati…
>We really shouldn't be building an "open source" AI in the first place though, and it's going to be illegal to do so soon. The weaponization power will be made clear soon and that will justifiably spook everyone.
Encryption was illegal not that long ago for the same reasons. Now it's the basis of all the digital economy. If we made it illegal again of the top 10 tech companies by market cap only Nvidia and TSMC would not be outright illegal to operate.
The timid cowardice that's taken over tech will not serve it well in the coming 20 years.
Re: Open source solution replicates ChatGPT training process
#125"hitting 100 million monthly active users 2 months after its launch". I'm deeply suspicious of that number. It came from Similarweb, who track these things through analytics gathered from browser extensions. I trust this article more: https://www.nytimes.com/2023/02/03/technology/chatgpt-openai... "But two months after its debut, ChatGPT has more than 30 million users and gets roughly five million visits a day, two p…
Can someone tell me what the hell they use ChatGPT for? I tried it a few times and it always confidently gave me wrong results to basic things. What is this thing supposedly “disrupting”? Is it really just marketing cranking out metric tons of spam blogs?
I used it to successfully write a job posting for a few engineering positions. Those things are 90% fluff anyway. The prompt was something like:
>Write a job listing for a front end engineer with 3 years in vue and mention some bullshit about how innovative $company_name is and how great it is to work a $company_name.
Two pages of corporatese and got a bunch of good responses.
Re: Open source solution replicates ChatGPT training process
#126The graph titled "Comparison between Colossal-AI and current major open source projects in the same period" has no label on the Y axis, which shows quantities in thousands. WTH?
Y axis are Github stars. They sort of mention it in the preceding paragraph.
Re: Open source solution replicates ChatGPT training process
#127Why are gazillions of parameters needed in the first place? From an information perspective it feels that there might be some fundamentally inefficient use of parametric freedom. A brute force approach to combinatorial explosion so to speak. Are there any research efforts that look into how to reduce model complexity (without substantially sacrificing performance obviously).
One way to think about it is that the model needs to essentially encode the entirety of human knowledge. If you can do it with just 175b parameters then it looks quite efficient to me. GPT-3 is about 400gb in size which would even fit in some modern IPhones! Another metric to consider is that there are about 100 trillion connections in the human brain. If you roughly equate brain connection to a model parameter then GPT-3 would be only 0.175% size of human brain.
Re: Open source solution replicates ChatGPT training process
#128Why are gazillions of parameters needed in the first place? From an information perspective it feels that there might be some fundamentally inefficient use of parametric freedom. A brute force approach to combinatorial explosion so to speak. Are there any research efforts that look into how to reduce model complexity (without substantially sacrificing performance obviously).
One way to think about it is that the model needs to essentially encode the entirety of human knowledge. If you can do it with just 175b parameters then it looks quite efficient to me. GPT-3 is about 400gb in size which would even fit in some modern IPhones! Another metric to consider is that there are about 100 trillion connections in the human brain. If you roughly equate brain connection to a model parameter then…
A model parameter is not the same as a "fact". Facts can multiply uncontrollably, but the logical relationships between facts that (at least we as humans) care about are much more economical. It feels that this approach is missing some key abstractions that might help reduce redundancy in encoding. But its just a hunch. Need to dig deeper to understand at least conceptually why this dimensional explosion.
Re: Open source solution replicates ChatGPT training process
#129Earlier quoted context omitted.
I’ve used it to write out 45 minute long lesson plans, help write complicated text message where all I’ve got is a bunch of points to make, I’ve had it correct my Portuguese since I’m not a native speaker, I’ve had it give me a baseline SQL table design to achieve a specific goal, I’ve had it come up with different ways to phrase things since I’m not creative enough, I’ve had it write marketing copy, created design b…
Exactly! I'm doing this for German language. It is rare for translators to get correctly how people talk in German and chatGPT is astonishingly good at that task.
Out of curiosity: Have you tried DeepL.com and, if so, how do the two compare?
Re: Open source solution replicates ChatGPT training process
#130"hitting 100 million monthly active users 2 months after its launch". I'm deeply suspicious of that number. It came from Similarweb, who track these things through analytics gathered from browser extensions. I trust this article more: https://www.nytimes.com/2023/02/03/technology/chatgpt-openai... "But two months after its debut, ChatGPT has more than 30 million users and gets roughly five million visits a day, two p…
Can someone tell me what the hell they use ChatGPT for? I tried it a few times and it always confidently gave me wrong results to basic things. What is this thing supposedly “disrupting”? Is it really just marketing cranking out metric tons of spam blogs?
* I'm learning a new programming language. "How do I do in ?" I get enough of an answer that I can experiment with the results.
* I have to write business emails. Instead of spending 20 minutes trying to think of the right politically correct terminology, I feed it the bullet points and it spits out a mostly proper email which I then spend another 5 minutes re-typing to get it the rest of the way.
* I've always wanted to start a blog, but I hate writing. Same idea as the previous point, but for blog posts.
I don't blindly trust it's output, but it saves me a ton of time in handling the to me bs extra stuff by filling in the edges.