Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

OpenAI Trains Language Model, Mass Hysteria Ensues

approximatelycorrect.com

Post reply on HN
Print view View original

71–80 of 119 posts

Re: OpenAI Trains Language Model, Mass Hysteria Ensues

#71

Post by sgt101 »

moconnor wrote: post #43
modeless wrote: post #26

Earlier quoted context omitted.

> The _whole point_ is that our model is not special and that other people can reproduce and improve Only people with a large amount of money and a lot of expertise. What you are doing is the opposite of democratizing AI.

Actually this shows why OpenAI matters. Google have been training and refining Transformer architectures for years; how unlikely is it nobody tried training a language model at this scale or larger with similar results? Yet from Google we heard nothing. Which is the optimal decision for them - they only lose by blowing the whistle.

A lot of people have results similar to this - but most people generating a paragraph of slightly_weird_but_plausible_if_you_read_quickly text using a primped version of BERT one time out of 25 regarded it as more or less pointless. But journalists don't.

This would be ok if this is the first time that anyone had a media go wild over AI story. But actually this has happened 10000 times this year already.

Re: OpenAI Trains Language Model, Mass Hysteria Ensues

#72

Post by pas »

roenxi wrote: post #54
pishpash wrote: post #42

Earlier quoted context omitted.

Exactly. This is like holding up spam samples or how spammers operate from the spam detecting work. That side (and the cultural discussions) needs all the headstart it can get, not be complacent that some arbitrary "experts" will patronizingly "protect" them.

If you look at it as a PR stunt, it is almost certainly a good idea. If a bad actor can auto-generate text that is not really distinguishable from something written by a human, how does a community with open membership (eg, HN) protect itself? I imagine this technology will enable interesting new attacks against online communities; we havn't seen that for a while. OpenAI are extremely sensible to draw attention to th…

But ... it's not novel. We could already generate convincing gibberish years ago.

Now the novelty is that this can be better targeted. But even simple Markov-chain based text generators were good enough to fool people for a bit.

And there was always people that had too much free time to write. A lot. (See for example the crackpots and conspiracy theorists that bombard physics forums. See the 9/11, Zeitgeists, etc. movies. See how much has been written about anti-vaxx, about quantum woo, etc.)

Reputation systems work pretty well for countering spammers.

And against APTs (advanced persistent threats, spearfishing attacks, etc) there's no real "universal" protection anyways. (You need a competent security team to out think and out resource the attackers in every possible dimension.)

This AI is the same as the paid Russian trolls and the unpaid scammers, and so on.

Re: OpenAI Trains Language Model, Mass Hysteria Ensues

#73

Post by czr »

sp332 wrote: post #62

Does someone have a description of the network somewhere? Does it use LSTM for memory or what? Is there anything unusual about the size or structure of the network? Does it use an attention mechanism?

I would recommend reading the paper: https://d4mucfpksywv.cloudfront.net/better-language-models/l...

and the previous paper

https://s3-us-west-2.amazonaws.com/openai-assets/research-co...

It's a transformer, not LSTM, and it's very large but not structured in a particularly unusual way.

Re: OpenAI Trains Language Model, Mass Hysteria Ensues

#74

Post by jph00 »

In other fields such as infosec, responsible disclosure is a standard approach. You don't just throw a zero-day out there because you can. Whilst the norms for AI research needn't be identical, they should at least be informed by the history in related fields.

The lead policy analyst at OpenAI has already tried to engage the community in discussing the malicious use of AI, on many occasions, including this extremely well-researched piece with input from many experts: https://maliciousaireport.com/ . But until OpenAI actually published examples, the conversation didn't really start.

In the end, there's no right answer - both releasing the model, and not releasing the model, have downsides. But we need a respectful and informed discussion about AI research norms. I've written more detailed thoughts here: https://www.fast.ai/2019/02/15/openai-gp2/

Re: OpenAI Trains Language Model, Mass Hysteria Ensues

#75

Post by Cacti »

bilbo0s wrote: post #58
Cacti wrote: post #38

Earlier quoted context omitted.

> - The _whole point_ is that our model is not special and that other people can reproduce and improve upon what we did. We hope that when they do so, they too will reflect about the consequences of releasing their very powerful text generation models. If this is your whole point, then I think you are missing something fundamental. Implementing these models doesn't require reflection, or introspection, or any sort of…

> You are fooling yourself if you think your stance makes a single mote of difference in this arms race... In fairness, if that's true, then no one has any need of her model. More seriously speaking, why does anyone need, say, "training set x", or "model y", to make their implementation work? You don't. So I don't really understand why everyone is so worked up about not releasing this stuff? If you want to do it, do…

No one is saying "I demand everyone do it." There are two points:

- If they are going to publish the research, and want to claim it as research (which they will, either by submitting it to a conference or putting on arxiv for the citations), then they should publish the supporting material, because without the supporting material it is impossible for reviewers or other researchers to evaluate. This is not just the model--they are also not publishing the training code or the dataset.

In short, they want to have it both ways, by having their work accepted as scientific research, yet providing absolutely no way of determining if the results are reproducible. That is a horrible, horrible standard. (other companies are guilty of this as well, btw). I mean, think about how absurd it is that they are saying "our scientific results are too good to publish. Trust us." Why is this acceptable? because it sure as hell wouldn't be acceptable if was a random person releasing a paper claiming incredible accomplishments, yet they provided absolutely no evidence.

- The other criticism is that the justification for why they aren't publishing (which is that they are too concerned with the moral and ethical implications of their work) is, well, a load of crap. They aren't doing anything to contribute to the ethical or moral use of these tools by doing this and they aren't slowing research into the area one bit. If they really wanted to have an impact here they should have just not said anything (but of course, then the authors couldn't put this on their resume...).

Whether they are releasing the model is not the issue, own its own, and I don't think anyone is throwing a fit because someone doesn't release their model. It's the _why_ and the implications that bother people.

Re: OpenAI Trains Language Model, Mass Hysteria Ensues

#76

Post by pas »

cma wrote: post #61
Permit wrote: post #5

Earlier quoted context omitted.

Is volume really what dictates whether or not you can impersonate someone? It's never seemed that way to me.

It lets you impersonate a crowd, or various crowds.

"PR firms" already have an army of fake/paid accounts on every important platform.

This new AI could help them with that. They can let go of the paid writers and hire an IT guy/gal to operate the bots - and the VPNs. (Or they can just pay a lot less to the paid trolls just for their home ADSL/Cable/4G connection.)

But so far this AI is not going to pass a Turing test. Sure, maybe it can be integrated with a chatbot. And it'll be interesting how internet communities will react.

Re: OpenAI Trains Language Model, Mass Hysteria Ensues

#77

Post by pas »

malux85 wrote: post #31
ilyasut wrote: post #23

Ilya from OpenAI here. Here's our thinking: - ML is getting more powerful and will continue to do so as time goes by. While this point of view is not unanimously held by the AI community, it is also not particularly controversial. - If you accept the above, then the current AI norm of "publish everything always" will have to change - The _whole point_ is that our model is not special and that other people can reprodu…

So a small number of individuals decided what's best for everybody? How is that open? How is that not centralization of power?

The paper is open https://d4mucfpksywv.cloudfront.net/better-language-models/l...

Re: OpenAI Trains Language Model, Mass Hysteria Ensues

#79

Post by m0zg »

pakitan wrote: post #64
BoiledCabbage wrote: post #15

Earlier quoted context omitted.

It's Bulk. Same reason why spam is a problem.

The problem for spammers, as well as for fake news writers, has never been in coming up with the text for the spam email or the fake news story. This is already cheap and easy enough. The problem is with distribution and getting enough eyeballs. This new and so very dangerous AI may enable you to come up with 1M fake news stories with the click of a button but it won't get any of those stories published in NYTimes.

>> get any of those stories published in NYTimes

I wouldn't be so sure about that. Take their reporting of Charlottesville events and Trump's comments about them. Here's what Trump _actually_ said: https://twitter.com/ZiaErica/status/1096572062196486144. Pretty reasonable point of view, all things considered. What was NYTimes "reporting"? That Trump is "defending white supremacists", of course. Don't believe me? See for yourself: https://www.google.com/search?q=trump+charlottesville+nytime.... Why was NYTimes doing that? It's either deliberate malice or incompetence, both of which would make NYTimes quite friendly to automatically generated fake news as long as they fit their narrative.

But there's a bigger issue with all of this. When people see this tech, they immediately think that it'll be used to generate fake news (which it will be, to be sure). BUT, it could also be used to do the exact opposite: take facts and summarize them without agenda-driven omissions, without "reading minds" or inventing "sources" "familiar with" someone's "thinking", or passing off uncorroborated dossiers or book chapters as gospel truth.

Re: OpenAI Trains Language Model, Mass Hysteria Ensues

#80

Post by repolfx »

roenxi wrote: post #54
pishpash wrote: post #42

Earlier quoted context omitted.

Exactly. This is like holding up spam samples or how spammers operate from the spam detecting work. That side (and the cultural discussions) needs all the headstart it can get, not be complacent that some arbitrary "experts" will patronizingly "protect" them.

If you look at it as a PR stunt, it is almost certainly a good idea. If a bad actor can auto-generate text that is not really distinguishable from something written by a human, how does a community with open membership (eg, HN) protect itself? I imagine this technology will enable interesting new attacks against online communities; we havn't seen that for a while. OpenAI are extremely sensible to draw attention to th…

I'm not sure it has much in the way of implications.

There is no real profit to be made by generating realistic looking text. Spammers don't work that way, spammers haven't cared about realistic looking text for years. Nor have spam filters cared much about text for a long time, exactly because it's so easy to randomise. Anti-spam is not a good reason to hold back on language generation models, in my view.

As for HN, if bots can write posts as good as humans, great, why hold back?

Post reply on HN