121–130 of 260 posts
And we kind of just stumbled on the design by throwing massive data and neural networks together?
Earlier quoted context omitted.
It will be interesting to see the attitude of Microsoft towards this project in the light of their "Microsoft loves open source" propaganda.
I don't know where people got this idea that Microsoft can't participate positively in Open Source, and do that sincerely, without open sourcing absolutely everything. Of course they can - just because you contribute to open source, and do that because you also benefit from open source projects, doesn't mean you have to do absolutely everything under open source. Especially considering OpenAI isn't even Microsoft's I…
I'm not claiming that. Of course there is place for closed and open elements of their offerings. Let me clarify.
In the past, Microsoft was very aggressive about open source. When they realized this strategy of FUD brings little result, they changed their attitude 180 and decided to embrace it putting literal hearts everywhere.
Personally, I find it hypocritical. There is no love/hate, just business. They will use whatever strategy works to get their advantage. What I find strange is that people fell for it.
(I don't know how the model is accessed - are users of mainline GPT-3 given a .pb and a stack of NDAs, or do they have to access it through access-controlled API?)
Wherever data is desired by many but held by a few, a pirate crew inevitably emerges.
Earlier quoted context omitted.
I don't know where people got this idea that Microsoft can't participate positively in Open Source, and do that sincerely, without open sourcing absolutely everything. Of course they can - just because you contribute to open source, and do that because you also benefit from open source projects, doesn't mean you have to do absolutely everything under open source. Especially considering OpenAI isn't even Microsoft's I…
> I don't know where people got this idea that Microsoft can't participate positively in Open Source, and do that sincerely, without open sourcing absolutely everything. I'm not claiming that. Of course there is place for closed and open elements of their offerings. Let me clarify. In the past, Microsoft was very aggressive about open source. When they realized this strategy of FUD brings little result, they changed…
But even when Microsoft can't open source it because it's not theirs, we still have people posting in this thread that this is further evidence that Microsoft is hypocritical. It sounds a lot like a form of Confirmation Bias to me where any evidence is used as proof that Microsoft is 'anti-open-source'.
Earlier quoted context omitted.
I don't know where people got this idea that Microsoft can't participate positively in Open Source, and do that sincerely, without open sourcing absolutely everything. Of course they can - just because you contribute to open source, and do that because you also benefit from open source projects, doesn't mean you have to do absolutely everything under open source. Especially considering OpenAI isn't even Microsoft's I…
I don't know where people got the idea that companies can be "sincere." Sincerity is faithfully communicating your mental state to others. A company's mental state can change on a dime based on the decisionmaking of people who rely on the company for the degree of profit it generates. Any analog to sincerity that you think you see can probably be eliminated by firing one person after an exceptionally bad quarter (or…
And I think companies can be sincere - because companies are really just groups of people and assets when you get down to the nuts and bolts of it.
The intention behind it is pretty good. Best of luck to them. I wonder if I can donate computing power to this remotely. Like the old SETI or protein folding things. Use idle CPU to calculate for the network. Otherwise the estimates I have seen on how much it would take to train these models are enormous.
This way, you never have to synchronize the weights of the entire model across the participants — you only need to send the gradients/activations to a set of peers. Slow connections are mitigated with asynchronous SGD and unreliable/disconnected experts can be discarded, which makes it more suitable for Internet-like networks.
Disclaimer: I work on this project. We're currently implementing a prototype, but it's not yet GPT-3 sized. Some issues like LR scheduling (crucial for Transformer convergence) and shared parameter averaging (for gating etc.) are tricky to implement for decentralized training over the Internet.
Serious question: is there a warez scene for trained models yet? (I don't know how the model is accessed - are users of mainline GPT-3 given a .pb and a stack of NDAs, or do they have to access it through access-controlled API?) Wherever data is desired by many but held by a few, a pirate crew inevitably emerges.
In my experience, the output from GPT-3, DALL-E, et al is similar to what you get from googling the prompt and stitching together snippets from the top results. These transformers are trained on "what was visible to google", which provides the limitation on their utility. I think of the value proposition of GPT-X as "what would you do with a team of hundreds of people who can solve arbitrary problems only by googling…
What would you do with a team of hundreds of people who can instantly access an archive comprising the sum total of digitized human knowledge and use it to solve problems?
Earlier quoted context omitted.
Content farms are hardly a productive application.
You missed the forest for the trees. If you got a tool that can use StackOverflow to solve simple programming tasks, or to generally solve any simple task with Google, then you're sitting on a gold mine.
It may be useful to hire less low skilled employees and keep a few senior ones that take input from machine and decide what to keep and what to throw away. I'm not sure if a senior engineer would be more productive patching up code written by a bot or writing it from scratch. It's going to be a hard sell while you still need human supervisors.
You can't trust a machine that can't reason with code implementation, or even content creation. You need a human to supervise or a better machine.
We already have AI based auto-completion for code, gpt-3 can be useful for that (but at what cost? Storing a huge model on your disk or making a slow / unsafe http request to the cloud?)