Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

AI training method exceeds GPT-3 performance with fewer parameters

infoq.com

Post reply on HN
Print view View original

31–40 of 82 posts

Re: AI training method exceeds GPT-3 performance with fewer parameters

#31

Post by fnord123 »

lzzzfelipe wrote: post #5

GPT3's strength is on language generation, so using *GLUE for evaluating it (where encoder type models are just better) and claiming to have 99.9% less parameters is sensationalism.

Welcome to benchmarklandia, where you find the top algorithm in an area and write a stripped down version that outcompetes it on a different set of data and get good results.

See: hadoop->spark->spark with infiniband->spark with nvme->timely dataflow on a laptop; csv parsers that don't support scientific notation floats, etc. I'm sure others can give moren interesting examples.

Re: AI training method exceeds GPT-3 performance with fewer parameters

#33

Post by yorwba »

antirez wrote: post #21

You do better than gpt3 with fewer parameters if you can have the same or better performances over the amount of tasks gpt3 can do well. To test a general model like gpt3 against something that can address a single problem does not make sense.

It does make sense if your aim is to address that single problem. They're comparing against GPT-3 because that was the previous record holder (in the few-shot setting), despite being a more general model.

Re: AI training method exceeds GPT-3 performance with fewer parameters

#34

Post by panpanna »

mhuffman wrote: post #22
jppope wrote: post #14

Sure they outperformed GTP-3... but can they DDoS my twitter account via a bunch of software engineers that won't stop talking about their model???

... even more, is the new model "so goddamn powerful" that it can't even be trusted with normal humans, and must be doled out (for money) a little bit at a time so that some normie doesn't accidentally create skynet?

It is starting to become a meme, but it sure achieved its PR goal?

Re: AI training method exceeds GPT-3 performance with fewer parameters

#35

Post by op03 »

desmap wrote: post #23

Slightly OT: Will GPT-3 itself will have any impact IRL? I ask because of its lacking options/competition on the hosting side (there's just Azure). This implies high prices hindering most use cases to break even/be profitable.

Mostly more effective spam. I see it as having access to the text generation capability of an infinite number of 15 year olds.

Re: AI training method exceeds GPT-3 performance with fewer parameters

#37

Post by emteycz »

dvfjsdhgfv wrote: post #10
visarga wrote: post #6

Earlier quoted context omitted.

Yes, it can do simple yes/no questions but can it write dad jokes like GPT-3?

For me another question is crucial: is it open (i.e. is there an open source reference implementation) or closed like the so-called "OpenAI" products/services?

Can you please explain to me (I don't understand it) why do you think GPT-3 is closed? Yes, they won't share the trained model, but they're sharing the research here[0][1] so you can reproduce easily, aren't they? As I understand it now, it's very fair - training the model is a separate thing from doing (and sharing) the research, is very costly, and would not happen if they were forced to open that too - I also don't understand why should they be.

[0] https://arxiv.org/abs/2005.14165

[1] https://github.com/openai/gpt-3

Re: AI training method exceeds GPT-3 performance with fewer parameters

#38

Post by dj_mc_merlin »

emteycz wrote: post #37
dvfjsdhgfv wrote: post #10

Earlier quoted context omitted.

For me another question is crucial: is it open (i.e. is there an open source reference implementation) or closed like the so-called "OpenAI" products/services?

Can you please explain to me (I don't understand it) why do you think GPT-3 is closed? Yes, they won't share the trained model, but they're sharing the research here[0][1] so you can reproduce easily, aren't they? As I understand it now, it's very fair - training the model is a separate thing from doing (and sharing) the research, is very costly, and would not happen if they were forced to open that too - I also don'…

> so you can reproduce easily

That's like saying you can look at the Eiffel tower and it's schematics so what's so hard about getting a spare million dollars and building it.

Re: AI training method exceeds GPT-3 performance with fewer parameters

#39

Post by fxtentacle »

emteycz wrote: post #37
dvfjsdhgfv wrote: post #10

Earlier quoted context omitted.

For me another question is crucial: is it open (i.e. is there an open source reference implementation) or closed like the so-called "OpenAI" products/services?

Can you please explain to me (I don't understand it) why do you think GPT-3 is closed? Yes, they won't share the trained model, but they're sharing the research here[0][1] so you can reproduce easily, aren't they? As I understand it now, it's very fair - training the model is a separate thing from doing (and sharing) the research, is very costly, and would not happen if they were forced to open that too - I also don'…

Yes they tell you how to do it.

But training a model of this size requires you to use thousands of GPUs, or wait forever. That will sum up to millions of $$$ in rental and electricity costs.

Re: AI training method exceeds GPT-3 performance with fewer parameters

#40

Post by Jugurtha »

m3at wrote: post #8

If you want to read the source, I recommend skipping to their follow-up paper (same authors): https://arxiv.org/abs/2009.07118 Edit: tangentially related but for those who like to have a glance from their phone, arxiv-vanity is great instead of squinting at a pdf: https://www.arxiv-vanity.com/papers/2009.07118/

Nice. We're building a machine learning platform[0] where you can schedule a training notebook job, watch it on your phone. Examples[1][2]. Docs[3]

- [0]: https://iko.ai

- [1]: https://twitter.com/jugurthahadjar/status/130762501783286989...

- [2]: https://twitter.com/jugurthahadjar/status/131380072874244915...

- [3]: https://iko.ai/docs/notebook/#long-running-notebooks

Post reply on HN