231–240 of 260 posts
Earlier quoted context omitted.
People don’t read it, search engines do.
BTW, we should mandatorily tag generated content for search engines in order to exclude it from future training sets.
Earlier quoted context omitted.
On CPU the latency would be absolutly prohibitive to the point of being useless.
For training yes, but not for inference.
> Earlier this year, researchers at NVIDIA announced MegatronLM, a massive transformer model with 8.3 billion parameters (24 times larger than BERT)
> The parameters alone weigh in at just over 33 GB on disk. Training the final model took 512 V100 GPUs running continuously for 9.2 days.
Running this model on a "regular" machine at some useful rate is probably not possible at this time.
Serious question: is there a warez scene for trained models yet? (I don't know how the model is accessed - are users of mainline GPT-3 given a .pb and a stack of NDAs, or do they have to access it through access-controlled API?) Wherever data is desired by many but held by a few, a pirate crew inevitably emerges.
I'd gladly contribute (power and) few of idle GTX cards I have to public peer/volunteer/seti@home-like project if result snapshot(s) are available publicly/to registered, active contributors.
SETI@home style distributed computation is not suitable for training something like GPT-3, unlike for SETI, the unit of work a node can do before needing to share it's output with the next node is really small, so very fast interconnect between the nodes is needed (Infiniband and NVLink is used in clusters to train it). It would probably take a decade to train such a model over regular internet.
Earlier quoted context omitted.
Maybe a case for a community colocation cloud where I a consumer can buy a system and colocate it in a large data center with great internal networking. Edit: typo
Handling heterogenous (and potentially untrustworthy) systems also adds overhead, not to mention that buying hardware in bulk is cheaper, so it makes the most sense just to raise the money and buy the hardware.
The intention behind it is pretty good. Best of luck to them. I wonder if I can donate computing power to this remotely. Like the old SETI or protein folding things. Use idle CPU to calculate for the network. Otherwise the estimates I have seen on how much it would take to train these models are enormous.
Not directly related, but the Learning@home [1] project aims to achieve precisely that goal of public, volunteer-trained neural networks. The idea is that you can host separate "experts," or parts of your model (akin to Google's recent Switch Transformers paper) on separate computers. This way, you never have to synchronize the weights of the entire model across the participants — you only need to send the gradients/…
Of course, that's not nearly as sexy.
Yes, there are lots of incredible positive impacts of such technology, just like there was with fire, or nuclear physics. But that doesn't mean that safeguards aren't absolutely critical if you want it to be net win for society.
These negative impacts are not theoretical. They are obvious and already a problem for anyone who works in the right parts of the security and disinformation world.
We've been through all this before... https://aviv.medium.com/the-path-to-deepfake-harm-da4effb541...
Of course, some of the same people who ignored recommendations[1] for harm mitigations in visual deepfake synthesis tools (which ended up being used for espionage and botnets) seem to be working on this.
[1] e.g. https://www.technologyreview.com/2019/12/12/131605/ethical-d...
In my experience, the output from GPT-3, DALL-E, et al is similar to what you get from googling the prompt and stitching together snippets from the top results. These transformers are trained on "what was visible to google", which provides the limitation on their utility. I think of the value proposition of GPT-X as "what would you do with a team of hundreds of people who can solve arbitrary problems only by googling…
Have you tried conversing with it, after a few lines of setting a proper context? Like two scientist talking or something like that. It can provide very interesting outputs that are not googlable. Yes, every time you see something that for human obviously doesn't make sense it makes you dismiss it. You would look at that output differently though if you were talking with a child. Just like a child can miss some infor…
The intention behind it is pretty good. Best of luck to them. I wonder if I can donate computing power to this remotely. Like the old SETI or protein folding things. Use idle CPU to calculate for the network. Otherwise the estimates I have seen on how much it would take to train these models are enormous.
Not directly related, but the Learning@home [1] project aims to achieve precisely that goal of public, volunteer-trained neural networks. The idea is that you can host separate "experts," or parts of your model (akin to Google's recent Switch Transformers paper) on separate computers. This way, you never have to synchronize the weights of the entire model across the participants — you only need to send the gradients/…
This would encourage people to host experts in your network and would create value.
The intention behind it is pretty good. Best of luck to them. I wonder if I can donate computing power to this remotely. Like the old SETI or protein folding things. Use idle CPU to calculate for the network. Otherwise the estimates I have seen on how much it would take to train these models are enormous.
Not directly related, but the Learning@home [1] project aims to achieve precisely that goal of public, volunteer-trained neural networks. The idea is that you can host separate "experts," or parts of your model (akin to Google's recent Switch Transformers paper) on separate computers. This way, you never have to synchronize the weights of the entire model across the participants — you only need to send the gradients/…