Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Better Language Models and Their Implications

blog.openai.com

Post reply on HN
Print view View original

131–138 of 138 posts

Re: Better Language Models and Their Implications

#131

Post by riku_iki »

pmontra wrote: post #114
cs702 wrote: post #54

Earlier quoted context omitted.

> Agreed that "simply" scaling up with more compute will result in progress and useful systems, and work in that direction is interesting and valuable. But, while we may not need new architectures or training objectives to make progress, we do need them to approach human level sample complexity. Yes, agreed. Nothing I said above contradicts that! :-) > Humans don't need to read through 40 GB of text multiple times to…

After 40 GB of text the model doesn't know anything about how the world works, and it shows many times in the examples. Nobody would do some of those mistakes, not even young kids. Other mistakes are more subtle but still show a total lack of understanding. Then yes, it's enough to write text that nobody really cares about and that could cover a lot of what we read.

It's because nobody dumps 40GB on kid. Kids go through long process of feedback-corrections. I imagine if there would be some crowd funded project to provide feedback about mistakes to this model, it would learn and produce better results fast.

Re: Better Language Models and Their Implications

#132

Post by riku_iki »

sytelus wrote: post #115
rfdearborn wrote: post #91

Sure, not releasing the full trained model probably delays it, but sooner or later a bad actor will do their own scraping and train their own model and share it around and the genie will be out of the bottle. Then what? I think we need to be conducting AI research (and building software generally) under the assumption that all of it will eventually be repurposed by bad actors. How would our practices be different if…

CommonCrawl already has open dataset in petabyte size ready on AWS. Even if it didn’t exist, scrapping 80GB of data in AWS is trivial. I am surprised authors considered this as such a big deal. Also notice that performance is not anywhere close to humans. It sort of works and it’s astonishing that it does but long way to go before we have to fear weaponizing text generation.

I think the big deal is the size of model, BERT large is 300M params, and this one is 1.5B. Bert has been trained on pod with 64 TPUs, and this model requires even larger GPU/TPU cluster. There is no way indie underfunded researcher can train such model.

Re: Better Language Models and Their Implications

#133

Post by riku_iki »

slashcom wrote: post #102
paraschopra wrote: post #101

Anyone who’s done large scale model training like this, an you shed light on following questions: What is the process like? Do you prototype locally? How do you confidence that the only limitation to good results is more compute power and NOT the model architecture or applicability of deep learning to a particular task? At what point do you decide that shelling many tens of thousands is OK? How often do you do large…

There’s a natural way to parallelize these models so that using 128 GPUs is the same as a 128x batch size. You can similarly simulate 128x batch size by accumulated gradients before backpropping. So you can test on just one or a few GPUs before you run the full thing. By that point you know it’s going to work, it’s just a matter of how well and whether you could’ve done nominally better with different tuning. There’s…

I doubt 1.5B params will fit any single GPU. I think they spread parts of models between GPUs/TPUs similarly to mesh-tensorflow: https://arxiv.org/abs/1811.02084

Re: Better Language Models and Their Implications

#134

Post by airstrike »

schoen wrote: post #73
wuthefwasthat wrote: post #3

Earlier quoted context omitted.

Thank you! We've released 500 random unconditional samples from GPT-2 at https://github.com/openai/gpt-2/blob/master/gpt2-samples.txt

Wow, some of these really go off the rails but those that only kinda go off the rails are absolutely hilarious and/or bizarre. A few summaries of ones that I looked at which appeared to be more or less staying on a single topic: Sample 1: An Austin nonvegetarian vegetarian restaurant encounters a series of difficulties in opening, as its nonexistent but extensive menu depicts a wide range of food options and the rest…

Blackwater USA was born January 21, 2003, when Blackwater founder Erik Prince met Bill Clinton in March of that year. In an email dated 08/28/99, Bill Clinton wrote to "Uncle Ray." Erik Prince was a small business owner and business leader. When Dagan tried to evict Blackwater USA from its Fort Carson, Colorado compound in January, 2000, the Iraq War began fourteen years ago. And my uncurent, signed-and-notarized correspondence with a Bill Clinton representative on October 27, 1999, speaks volumes about one's knowledge of the INC diamond and fronts and shields Brazilian cache testimonials in commissioning documents for the SCOUTS network. In other words, the INC was ALL OVER the party that night and very highly sensitive agreements between the same three foreign entities in Chicago on BLUESTAIR retail banks. Yes, I believe the whole "####BETTIN Powers intact." But it was Erik's trans-national connections through his vast buyout shops in the Far East that made him the savvy buyer in this slice of the Direct Foreign Investment biz. Erik of Bankers Trust and Investments was working to implement the political policy and campaign advice Bernie Madoff rejected, both of which ultimately cost Hilary collectors many millions. Global Information similar to the names today haven't changed in the frequency folder of Erik's UFLAC machine, as evinced by the statement "West Coast growth project."

First members moved to Las Vegas and a quiet retreat in Mesa, Arizona came into existence titled Predius Group LLC " rel="nofollow">http://www.predius.com/>

The massive Madison Avenue bank and hedge fund structure that devised illegal high rollers for such organizations as SS&XM 13/CCGB, Marc Rich and use of the old Puerto Rican nuclear smuggling routes from Santo Domingo continues, as deeply buried secrets leaked to me show that a master of legal portals exited from a long, multimillion compensation firm in Wisconsin for BP with a large bolillo-stuffed look like the federal prosecutor U.S. Grand Nomen Suisse makes to seem "human." One intelligence utility broke for their relative comfort in several layers of a "cheque factory/lobby/university."

Blackwater USA may not like to admit what a true dick the injustice has wrought. What the rape of Alexandria, a slap in the face to the American public seems to belong able to early final order technology, esp. Skyjack Special Forces Weapons, at the price of just 3 million followed by a back-door, fired check into Hillary's back pocket allows us to view this sub rosa of tax-proof lottery economics as messy FleetMicrosoft. investors that bloom with sedition, all hands seem tied at the start. We may never learn who the infamous Bradie Burglars were and for a directors' distance in Texas. obscurity that loafed Fletcher Marshall bridges Hab Saber guaranteed earned59,5/83/67 which for this two is two sacrificed, locks dread beneath the cave like bankers shielded. Noir in the wake of JFK stimulates a game of IDEAS over Hypothetically recruTCater John Waldincyv. The chairman of Sears Remedy Strategies and GSD Group Inc. New York had once taught a class called with the subject Logic and Order Manufactured Domination subject17 Excitement vacuum playoffs NASA policies war against Al Qaeda. A homeland security branch of the FBI using its HAWK MAN wooden stake farm & an AK assault rifle on Muslims , refuses to bend its political opinions in line with a Supreme Court accepting Hollywood records, magazines of political horrors, such post minority weekend get blown off. As a result of projecting it at God they sound and dissent singing. while boghoum badges still as nasty MO. Uncovered I act as the CERO, Scarlet Crusade presence. and begins to operate out of a converted American Air National Guard SOC air attache tube at the embassy port of entry. I adapt mode that accommodates the supervisor/chief and begins to adjust to alienation anomaly Lebanese charities making intercepts. ownsACK board of USCT to go from that I CAN READ AND DOB less than 09 or 2005 what is running on twitter, like We are to blame motherfuckersmobile. We BLACKWATERim audiences lesson stronger then in multiple appearances in 2008 or 2009, on their website is clearly for everyone to see.

Re: Better Language Models and Their Implications

#135

Post by dannyw »

DenisM wrote: post #82
minimaxir wrote: post #12

The generated text sounds too good; is it possible that the model overfit the source material (especially since the n-previous-tokens value is infinite, while other approaches like char-rnns/textgenrnn use a fixed window length)? It's something I've encountered many times while working with text generation.

It is uncanny. I would expect a language model to pull up different person names for each time that one was called for. For a person name to be consistently used through several paragraphs it is not enough to rely on word co-occurrence. If I had to produce a text like this I would simply take an existing text and replace randomly chosen words with other similar words (as hinted by amvalo). Similar as in - words that…

That’s because the transformer model used has memory.

Re: Better Language Models and Their Implications

#137

Post by resters »

mikeyouse wrote: post #39
skybrian wrote: post #33

Earlier quoted context omitted.

I would guess this reduces the risk. Why would you say it does the opposite? My argument: state actors might misuse this tech, but letting any script kiddie do whatever they want almost guarantees someone will misuse it.

Which is exactly what happens whenever there's a leak of NSA or other foreign government 'hacking' tools. As soon as they're public, ransomware authors and other shitty actors all deploy them to steal as much as possible before systems are patched. Case-in-point: Wannacry: https://en.wikipedia.org/wiki/WannaCry_ransomware_attack NotPetya: https://en.wikipedia.org/wiki/Petya_(malware)#2017_cyberatta...

I think it’s a bit different because the attacks are not entirely clear. The power of many of the attacks would be for them to fly under the radar because nobody was expecting it.

So having lots of script kiddie attacks (which would be sloppier and easier to notice) would lead to a more rapid adoption of safeguards.

Re: Better Language Models and Their Implications

#138

Post by Retra »

lordnacho wrote: post #126
Retra wrote: post #125

Earlier quoted context omitted.

A proper Turing test allows you to ask questions whose answers must indicate novel introspection, learning, and social awareness. It's absurd to think any computer system today would pass such a test.

This thing does exactly that. You can see in the text it generates, it sounds like someone who's thought about things and keeps a consistent tone about the subject.

That is not what it needs to do to pass a Turing test. It needs to sustain ongoing dialog and thought, while maintaining a consistent awareness of another human's perspective in real time. It needs to do what a human does while having a conversation. And humans don't just spit out responses to questions. They get bored. They get distracted. They display tonal inconsistencies in response to emotions.
Post reply on HN