Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Open source solution replicates ChatGPT training process

hpc-ai.tech

Post reply on HN
Print view View original

131–140 of 158 posts

Re: Open source solution replicates ChatGPT training process

#131

Post by imtringued »

simonw wrote: post #35
de6u99er wrote: post #29

Earlier quoted context omitted.

GPT-3 has been publicly covered in scientific publications. Same as GPT-2, and GPT. Those are all pre-trained models, where GPT is the abbreviation of Generative Pretrained Transformer. Transformers have been invented in 2017 at Google Brain [1]. -> https://medium.com/walmartglobaltech/the-journey-of-open-ai-... GPT-4 is around the corner, and it's allegedly 100x more powerful than it'd predecessor. -> https://medium…

That "100x" figure is extremely poorly sourced. I don't believe that at all.

And yet the intimidating pictures of a small and large circle keep getting posted everywhere.

Re: Open source solution replicates ChatGPT training process

#132

Post by imtringued »

college_physics wrote: post #122

Why are gazillions of parameters needed in the first place? From an information perspective it feels that there might be some fundamentally inefficient use of parametric freedom. A brute force approach to combinatorial explosion so to speak. Are there any research efforts that look into how to reduce model complexity (without substantially sacrificing performance obviously).

Given a large enough model, model architecture becomes increasingly less relevant as any specialized architecture can be discovered by the larger model automatically.

The only benefit of a specialized architecture is minimizing resource usage.

Re: Open source solution replicates ChatGPT training process

#133

Post by leobg »

Der_Einzige wrote: post #58
lossolo wrote: post #16

Earlier quoted context omitted.

It's because BLOOM is undertrained, you can prune a lot of weights in BLOOM and it doesn't impact performance. Look at Chinchilla paper[1], 70B model outperforms 175B GPT-3 model. https://arxiv.org/abs/2203.15556

In general, most giant LLMs are extremely undertrained at this time. Consider that most of the gains in RoBerta vs bert were from just continuing to train.

So can we continue training RoBERTa to get it to, say, GPT3 Ada level

Re: Open source solution replicates ChatGPT training process

#134

Post by flangola7 »

dodslaser wrote: post #106
flangola7 wrote: post #104

Earlier quoted context omitted.

That's unlikely to work. The memory has to be fast with low latency, even switching from on-board VRAM to system RAM slows performance at least 10-100x. The bottleneck isn't computing power it's I/O. Total bus bandwidth of a common small AI cluster is around 1 terabyte per second. We really shouldn't be building an "open source" AI in the first place though, and it's going to be illegal to do so soon. The weaponizati…

There's a significant number of people working hard on making certain tech illegal or at least heavily restricted. E2EE and Onion Routing comes to mind. That doesn't mean we should abandon them. In fact, in many cases it's an indicator that we should keep going. Why do you think we should avoid an open source AI?

How do you plan to have differential technological development and careful alignment research if anyone is allowed to build Skynet in their garage?

I use and generally support E2EE and onion routing. E2EE and onion routing aren't inherently existential risks to the continued existence of life on Earth.

Re: Open source solution replicates ChatGPT training process

#135

Post by flangola7 »

throw009 wrote: post #124
flangola7 wrote: post #104

Earlier quoted context omitted.

That's unlikely to work. The memory has to be fast with low latency, even switching from on-board VRAM to system RAM slows performance at least 10-100x. The bottleneck isn't computing power it's I/O. Total bus bandwidth of a common small AI cluster is around 1 terabyte per second. We really shouldn't be building an "open source" AI in the first place though, and it's going to be illegal to do so soon. The weaponizati…

>We really shouldn't be building an "open source" AI in the first place though, and it's going to be illegal to do so soon. The weaponization power will be made clear soon and that will justifiably spook everyone. Encryption was illegal not that long ago for the same reasons. Now it's the basis of all the digital economy. If we made it illegal again of the top 10 tech companies by market cap only Nvidia and TSMC woul…

How do you plan to have differential technology development and thoughtful and cautious alignment research if we go building these things without a speed limit?

Giving a baby a hand grenade would be more responsible.

Re: Open source solution replicates ChatGPT training process

#136

Post by flangola7 »

ahtihn wrote: post #110
flangola7 wrote: post #104

Earlier quoted context omitted.

That's unlikely to work. The memory has to be fast with low latency, even switching from on-board VRAM to system RAM slows performance at least 10-100x. The bottleneck isn't computing power it's I/O. Total bus bandwidth of a common small AI cluster is around 1 terabyte per second. We really shouldn't be building an "open source" AI in the first place though, and it's going to be illegal to do so soon. The weaponizati…

> We really shouldn't be building an "open source" AI in the first place though, and it's going to be illegal to do so soon. How do you make that illegal while still allowing private corporations to build AI? How do you legally define AI without applying it to all kinds of existing applications and without stopping all research on AI? And while staying broad enough that simply using a slightly different technique wou…

Replace "AI" with "uranium enrichment and nuclear research" and the answers fill themselves in.

Re: Open source solution replicates ChatGPT training process

#137

Post by Taek »

sillysaurusx wrote: post #57

> On a single multi-GPUs server, even with the highest-end A100 80GB GPU, PyTorch can only launch ChatGPT based on small models like GPT-L (774M), due to the complexity and memory fragmentation of ChatGPT. Hence, multi-GPUs parallel scaling to 4 or 8 GPUs with PyTorch's DistributedDataParallel (DDP) results in limited performance gains. Where are these numbers coming from? An 80GB A100 GPU is certainly more than capa…

For reference, GPT-NeoX is a 20B parameter model, and it runs on 45 GB of VRAM. On an 80 GB A100 you could probably run a 35B parameter model. Maybe 8 A100 cards to do inference on ChatGPT?

Or 32 3090 cards, which would run you under $40k total.

Re: Open source solution replicates ChatGPT training process

#138

Post by dodslaser »

flangola7 wrote: post #134
dodslaser wrote: post #106

Earlier quoted context omitted.

There's a significant number of people working hard on making certain tech illegal or at least heavily restricted. E2EE and Onion Routing comes to mind. That doesn't mean we should abandon them. In fact, in many cases it's an indicator that we should keep going. Why do you think we should avoid an open source AI?

How do you plan to have differential technological development and careful alignment research if anyone is allowed to build Skynet in their garage? I use and generally support E2EE and onion routing. E2EE and onion routing aren't inherently existential risks to the continued existence of life on Earth.

LLMs are not going to pose an existential risk to anyone. Also, making AI development less accessible to the general public will not make it any safer.

I am willing to bet all this fear mongering singularity bullshit is just being peddled by large corporations with a vested interest to keep AI development out of reach from the general public.

Re: Open source solution replicates ChatGPT training process

#139

Post by throw009 »

flangola7 wrote: post #135
throw009 wrote: post #124

Earlier quoted context omitted.

>We really shouldn't be building an "open source" AI in the first place though, and it's going to be illegal to do so soon. The weaponization power will be made clear soon and that will justifiably spook everyone. Encryption was illegal not that long ago for the same reasons. Now it's the basis of all the digital economy. If we made it illegal again of the top 10 tech companies by market cap only Nvidia and TSMC woul…

How do you plan to have differential technology development and thoughtful and cautious alignment research if we go building these things without a speed limit? Giving a baby a hand grenade would be more responsible.

Caution is for losers.

Re: Open source solution replicates ChatGPT training process

#140

Post by brainbag »

harleypig wrote: post #130
jackblemming wrote: post #23

Earlier quoted context omitted.

Can someone tell me what the hell they use ChatGPT for? I tried it a few times and it always confidently gave me wrong results to basic things. What is this thing supposedly “disrupting”? Is it really just marketing cranking out metric tons of spam blogs?

* I'm learning a new programming language. "How do I do in ?" I get enough of an answer that I can experiment with the results. * I have to write business emails. Instead of spending 20 minutes trying to think of the right politically correct terminology, I feed it the bullet points and it spits out a mostly proper email which I then spend another 5 minutes re-typing to get it the rest of the way. * I've always wante…

The internet and books primarily focus on beginner-to-intermediate process, so there's very little resources beyond that. I've found ChatGPT to be exceptional for explaining things beyond it, like getting into more advanced Rust topics lately.

I think the key is to treat it like an experienced mentor that can make mistakes because of imperfect memory, not a perfect talking encyclopedia. Web searches don't always have the right answer, and even experts with decades of experience (cough) still get things wrong regularly. It's a collaborative conversation.

Post reply on HN