Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Better Language Models and Their Implications

blog.openai.com

Post reply on HN
Print view View original

101–110 of 138 posts

Re: Better Language Models and Their Implications

#101

Post by paraschopra »

Anyone who’s done large scale model training like this, an you shed light on following questions:

What is the process like? Do you prototype locally? How do you confidence that the only limitation to good results is more compute power and NOT the model architecture or applicability of deep learning to a particular task? At what point do you decide that shelling many tens of thousands is OK? How often do you do large scale training only to find non-impressive results and hence the money wasted?

Re: Better Language Models and Their Implications

#102

Post by slashcom »

paraschopra wrote: post #101

Anyone who’s done large scale model training like this, an you shed light on following questions: What is the process like? Do you prototype locally? How do you confidence that the only limitation to good results is more compute power and NOT the model architecture or applicability of deep learning to a particular task? At what point do you decide that shelling many tens of thousands is OK? How often do you do large…

There’s a natural way to parallelize these models so that using 128 GPUs is the same as a 128x batch size. You can similarly simulate 128x batch size by accumulated gradients before backpropping. So you can test on just one or a few GPUs before you run the full thing.

By that point you know it’s going to work, it’s just a matter of how well and whether you could’ve done nominally better with different tuning.

There’s been enough research leading up to this paper to suspect that just scaling larger would play out.

Re: Better Language Models and Their Implications

#103

Post by paraschopra »

slashcom wrote: post #102
paraschopra wrote: post #101

Anyone who’s done large scale model training like this, an you shed light on following questions: What is the process like? Do you prototype locally? How do you confidence that the only limitation to good results is more compute power and NOT the model architecture or applicability of deep learning to a particular task? At what point do you decide that shelling many tens of thousands is OK? How often do you do large…

There’s a natural way to parallelize these models so that using 128 GPUs is the same as a 128x batch size. You can similarly simulate 128x batch size by accumulated gradients before backpropping. So you can test on just one or a few GPUs before you run the full thing. By that point you know it’s going to work, it’s just a matter of how well and whether you could’ve done nominally better with different tuning. There’s…

Thanks.

>By that point you know it’s going to work, it’s just a matter of how well and whether you could’ve done nominally better with different tuning.

This can't be true in all cases, right? I'm assuming that for many initially promising results on less-compute when they scale it, the results aren't impressive. I'm very curious to know what is the trials-to-success rate of publishable results when big-compute is thrown in the mix.

Re: Better Language Models and Their Implications

#104

Post by jeremysalwen »

wuthefwasthat wrote: post #3
legatus wrote: post #2

I was honestly surprised by the quality of the generated text. While I can't say I've been following the state of the art in the last months, this seems like a pretty important step forward. Furthermore, at the end of the post they note that the samples are somewhat representative of their results. Maybe they should consider releasing a text file with some more (not hand-chosen) samples? Whatever the case, fantastic…

Thank you! We've released 500 random unconditional samples from GPT-2 at https://github.com/openai/gpt-2/blob/master/gpt2-samples.txt

Is this the future of politics...

===================================

Photo: DEA / W.H. 038 WITHDRAWN: Rep. Allen West to run again in 2016

Former Rep. Allen West will run for the U.S. Senate in 2016 -- okay, maybe not quite against Florida GOP Sen. Marco Rubio.

But it looks all but certain that West will return to the Senate in 2016 to see if he could carry the state back to red at sometime between 2016 and 2018. Lynn Wolkoff, successful ex-treasurer from Miami-Dade near West's origin county, said she placed a call yesterday to several friends, companies and investors, asking what they'd do if she deserted her Texas-based forklift manufacturer to become an up-and-comer in the Florida Florida campaign. And in the process, she learned that West himself might want to run.

"Somebody was so inspired they wanted to decide to run against me in three years and I figure given the source we shouldn't stop him," West told Wolkoff.

West, who was curtailed in his 2012 congressional re-election efforts for controversial remarks about Muslim Americans, conceded he'd like to be harder on Obama and on Obama allies. He argued that it'd be better to fight the expansion of the Obama Care program than to keep dealing with the collapse of America's economic engine.

"Taxes should be increased for very high incomes. We should go after the highest-income earners. Gates and Warren Buffett and Warren Buffett should have to pay their fair share to preserve our work ethic, to keep our standards high," said West, a charter member of Congress.

Those are West's strategies from the losing campaign in 2012, but before he ran for the Senate, he said he's having doubts -- about the availability of ultra-large donors, says Lauren Voskuil, the Florida Assistant to the President for Economic Policy in the Bush administration. He pushed former Florida Gov. Jeb Bush against "self-funding" laws that ban him from using some super PACs - and West's strategy is strong, Voskuil said, calling Westbrook "a master strategist".

"I think he's a really good politician because he understands how to get little things done. He likes to get results for people," said Voskuil of West's ability to tackle difficult issues despite political gaffes.

Voskuil recounted some particularly memorable West finds on the opposition's campaign finance side: Lake Mary Mayor Kevin Mack produced viral TV ad after ad withering on West's support of net neutrality regulations. Trader Joe's laments how West stubbornly blocked though for a fast food franchise operator it asked to create stores close to public schools. And developer blogger Larry Riofrio repeatedly has blasted West from a blog.

"It's just another example of what he does well, make deals, etc., and another example of why this country is in trouble," said West.

Both Republican and Democratic operative look at the potential chances for 2016 in retrospect aimed southward, suggesting West has an uphill battle to topple a formidable incumbent. Terry Dalsia took aim at the stated motivation to run

"Mr. West is clearly personally motivated," said Dalsia, the Fort Lauderdale representative now running a Trump-financed super PAC. "He is running because he can do it in 2044. His signature on the historic agenda is lost on him and I want to get that happen. He will be leaving the Senate with every GOP vote in 2014 ... just not before. Long way off."

Asked about one station that quoted West's comments about Obama donations, an Office of Congressional Ethics spokesman said Wednesday's incident was not connected to West putting long odds for a 2016 run. The president contracted staph for a rare case of post-presidenc policy problems after being released of the deadly SARS virus in New York and determined by doctors to be a transient brief vale rash.

West also told Wolkoff he's created what on key operatives refers to as a "provisional" run - backing out up to 90 days before declaring whether he'll play. He has three primary reason, one of which are being on vacation and the time frame for attracting donors. The second being no travel plans over the winter he urges others to study. The last boils down to the fact he really, really thinks he could win. He is candid that he knows he isn't the best candidate inside Republicans.

Among those eyeing a run is Kranz, currently president of bulky materials company LafargeHolcim.lt. Briefly considered for the job of visit by Sen. John McCain, Goldwater's son said he opposes Texas Gov. Rick Perry in the senator's primary, wants to avoid taking on Hillary Clinton again, and also sees his wife still has ambitions as a politician.

"When I heard Sen. West said he's on the way out, I told myself -- that's great, he knows he's not

Re: Better Language Models and Their Implications

#105

Post by lordnacho »

This is so crazy good, someone needs to do a Turing test by sending it to some unsuspecting publishers.

I get the feeling that debatepocolypse is not far away. Every forum can now be spammed with reasonable sounding gibberish that humans will have to slog through.

Re: Better Language Models and Their Implications

#106

Post by sanxiyn »

lordnacho wrote: post #105

This is so crazy good, someone needs to do a Turing test by sending it to some unsuspecting publishers. I get the feeling that debatepocolypse is not far away. Every forum can now be spammed with reasonable sounding gibberish that humans will have to slog through.

We must run a bot on HN with upvotes/downvotes for comments acting as reinforcement.

Re: Better Language Models and Their Implications

#107

Post by lostmsu »

resters wrote: post #7

While censoring the full data set seems in some way to support the rationale of the OpenAI charter, it also means that only state actors and very well-funded entities will be able to use the work to create models of the size necessary to do the impressive stuff in the write up. Based on the concerns, it would seem that restricting the capabilities only to state actors would have the opposite of the intended effect. W…

Or, in 3 years, anyone. I suggest everyone start thinking on ramifications now.

Re: Better Language Models and Their Implications

#110

Post by yazr »

Cybiote wrote: post #79
ninjin wrote: post #67

Earlier quoted context omitted.

> …we do NOT need new architectures… You do realise the irony in stating this regarding transformers that arguably made their first appearance as decomposable attention in 2016 [1] and then as transformers in 2017 [2]? It is not as if this is a vanilla RNN straight out of the 90s sweeping the floor with decades of model innovations, rather it looks like we are seeing the rise of a new, simple model category that work…

Agreed, the transformer's approach to memory/attention is certainly novel and a non-trivial departure from RNNs. It's also questionable the extent to which this is not over-fitting. Some are already finding phrases on the web the model is clearly stitching together. This is not to underplay the impressiveness of stitching the correct phrases and n-grams in a thematically coherent way for paragraphs at a time, it's st…

Do u have any recent links to program synthesis (with transformer or other novel attempts ?)
Post reply on HN