241–250 of 260 posts
Earlier quoted context omitted.
GPT-3 can’t play Go.
It almost definitely can to some extent given that gpt2 could play chess [0]. 0. https://slatestarcodex.com/2020/01/06/a-very-unlikely-chess-...
Earlier quoted context omitted.
It's likely to be bad, such as: Massively plagiarize articles and the search engine probably have no way to identify which is the original content. It's like to rewrite everything on the internet using your own words, this may lead to the internet filled with this kind of garbage. Reddit and platforms alike filled with bots say bullshits all the time but hard to identify by the human in the first place (current model…
>(current model is pretty good at generating metaphysical bullshits, but rarely insightful content) Wait, are we talking about bots posting crap, or the average political discussion?
Earlier quoted context omitted.
BTW, we should mandatorily tag generated content for search engines in order to exclude it from future training sets.
Apart from that, hopefully the people building training sets use gltr or something similar to prevent training on generated text. http://gltr.io/
Earlier quoted context omitted.
Not directly related, but the Learning@home [1] project aims to achieve precisely that goal of public, volunteer-trained neural networks. The idea is that you can host separate "experts," or parts of your model (akin to Google's recent Switch Transformers paper) on separate computers. This way, you never have to synchronize the weights of the entire model across the participants — you only need to send the gradients/…
Do you have a personal Twitter account I can follow? Your career is one I'd like to follow.
https://github.com/EleutherAI/gpt-neo (Couldn't find it on the website)
They also pivoted to dall-e to cover up their complete failure to deliver on any of their promises with gpt3, which was an interesting move.
Earlier quoted context omitted.
Not directly related, but the Learning@home [1] project aims to achieve precisely that goal of public, volunteer-trained neural networks. The idea is that you can host separate "experts," or parts of your model (akin to Google's recent Switch Transformers paper) on separate computers. This way, you never have to synchronize the weights of the entire model across the participants — you only need to send the gradients/…
How do you deal with adversarial/byzantine updates that attempt to degrade performance or even install a backdoor? Do you use plain averaging, or some other aggregation algorithm like Multi-Krum?
A draft solution would be for the central server to measure the goodness of each update and drop the ones that don't perform well. This could somehow work since inference is much cheaper than gradients computing.
Earlier quoted context omitted.
I don't know where people got this idea that Microsoft can't participate positively in Open Source, and do that sincerely, without open sourcing absolutely everything. Of course they can - just because you contribute to open source, and do that because you also benefit from open source projects, doesn't mean you have to do absolutely everything under open source. Especially considering OpenAI isn't even Microsoft's I…
How about when Steve Ballmer said something along the lines of “Linux is a cancer that attaches itself in an intellectual property sense to everything it touches” Pretty sure that is hostile towards open source? Linux being one of the flagship projects of open source. [edit] source https://www.zdnet.com/article/ex-windows-chief-heres-why-mic...
In my experience, I work at a completely different company than the one that Ballmer ran. Nearly everyone I talk to speaks of the "Ballmer era" in a negative light, and confirms that Satya literally turned the entire company on its head.
Many things happen every day that would never have happened under Ballmer.
Earlier quoted context omitted.
Not directly related, but the Learning@home [1] project aims to achieve precisely that goal of public, volunteer-trained neural networks. The idea is that you can host separate "experts," or parts of your model (akin to Google's recent Switch Transformers paper) on separate computers. This way, you never have to synchronize the weights of the entire model across the participants — you only need to send the gradients/…
Your project looks so interesting. Have u thught of putting the experts on a distributed market where their expertise and work can be exchanged for some token (obvlsy using a blockchain). This would encourage people to host experts in your network and would create value.
Also, I believe that for some projects (e.g. GPT-3 replication effort) people would want to join the network regardless of the incentive mechanism, as demonstrated by Leela Chess Zero [1].
Earlier quoted context omitted.
Not directly related, but the Learning@home [1] project aims to achieve precisely that goal of public, volunteer-trained neural networks. The idea is that you can host separate "experts," or parts of your model (akin to Google's recent Switch Transformers paper) on separate computers. This way, you never have to synchronize the weights of the entire model across the participants — you only need to send the gradients/…
How do you deal with adversarial/byzantine updates that attempt to degrade performance or even install a backdoor? Do you use plain averaging, or some other aggregation algorithm like Multi-Krum?