41–50 of 212 posts
Earlier quoted context omitted.
This post may be tongue in cheek, but realistically that's right. I ran the Facebook chatbot that was posted maybe a month ago and didn't get good performance until I was using many Tesla v100 GPUs, which are $8k each. Thankfully modern CSPs like Azure allow renting rigs that can handle this pretty easily.
How much does that cost?
This part really freaked me out... GPT-2 couldn't do math: Context → Passage: Saint Jean de Br´ebeuf was a French Jesuit missionary who travelled to New France in 1625. There he worked primarily with the Huron for the rest of his life, except for a few years in France from 1629 to 1633. He learned their language and culture, writing extensively about each to aid other missionaries. In 1649, Br´ebeuf and another missi…
It seems it has (rudimentarily) learned concepts general enough to be logic itself. That is general intelligence. Now hook it up to reinforcement circuitry and make it even larger and it will mark the end to life as we know it. GTP-3 has 175 billion parameters, but the human brain has 100 trillion synapses, so 0.175%. NN model capacity currently has a 3.4 month doubling time.[1] In 7-10 doublings we'll be in a simila…
What is this and why does it take the top two spots on HN?
How do you go about running a model this large?
Read though most of the paper and here's what GPT-3 is: If you wanted to generate poems with GPT-2, you'd need to have a lot of poems to fine-tune GPT-2 to get reasonable results. With GPT-3, you use few-shot learning instead (without the need to do gradient updates with each example) The paper is long and filled with how it stacks with models like Grover and T5 and it does well... given that this is a 175 B param mo…
Would you have any easy to explain insight in to how these perform better than larger models? I’ve always wanted to understand that as a technically adept and somewhat familiar (briefly) person who has explored what such models can do.
Is it sentient yet? /s Real question, are they going to release the full model?
It took them a while to release GOT-2 full model because of the implications for things like spambots. The GPT-3 paper indicates that they have been monitoring forums and noticed that bad actors haven't really been using GPT-2 for their own devices. That's unsurprising because GPT-2 takes a lot of hardware to run and I assume it messes with the economics of spamming. GPT-3 will take significantly more resources to ru…
GPT-2 doesn't require as many resources to run as you would expect: even from the 1.5B model, you can mass-produce passing spam comments for less than a dollar an hour in GPU costs: https://docs.aitextgen.io/tutorials/generate_1_5b/
Pure text spam in general is less effective in 2020; it's content that harder to fake (e.g. deepfakes) that shakes up social media, and why it's good FB/Twitter have proactively taken a stance against it.
Earlier quoted context omitted.
One thread will (probably) be merged into the other, but GPT-2 was an extremely popular OpenAI project that generated long, realistic-sounding text/articles if you gave it a simple starting sentence or topic sentence. GPT-3 is an iteration on that, so it's likely a huge improvement.
It doesn't sound like it's an improvement at all, but instead requires less training data to produce worse results?
They missed an opportunity to be the first paper to measure their computation in mole flops.