Better Language Models and Their Implications

It's becoming ever more certain that the transformer architecture is one of the largest contributions to AI (not merely machine learning, but AI), often beating LSTMs despite LSTMs being expressive enough to capture Turing Equivalence (at least in theory). Its main ideas are three: shorter paths help gradient flow, the training setup and the final key aspect, unhelpfully called self-attention. Self-attention is better thought of as a form of similarity gated key-value soft memory on which learning operations allows Transformers to learn non-trivial programs with contextual weights look-ups.

I also notice reported tries, suggesting some level of curation. While this level of generation is undoubtedly impressive and a sign of non-trivial levels of understanding, the ability to project along arbitrary dimensions of similarity at a fine-grained level and to learn from text instruction is more useful than text generation. Although the unicorn story was a really fun read, better than many humans already, I doubt it could have gone on for much longer. It maintains a theme but not coherently or fluently (see especially the Kennedy nanotech and recycling examples, comparing the dis-fluency there versus the excellence of the Civil War report suggest at least some over-fitting). These relatively minor caveats aside, this is unambiguously an outstanding result.

Winograd Schemas are the single metric to track if interested in understanding how language understanding is truly improving. OpenAI reports 71% and wrongly report the previous record as 63%. The current record here is at 65% https://gluebenchmark.com/leaderboard though not fully comparable. Will OpenAI be submitting? Note that you can get to 60% using about 1-2 orders of magnitude less data and compute.

It concerns me that results here are so far dependent on such large data and computation. However, based on several papers I've read, I do not believe this to be inherent even in transformers. I plan to do some experiments on this when I free up some bandwidth.

If everyone is pulled in by the glamour of working for a well funded, prestigious operation then it should be no surprise that they do not consider paths which operate on several orders of magnitude less data and computational resources.

We all should consider bringing about a group of researchers who swear to an austere computational life of a single GPU, no more than 4-8x average RAM and CPUs that do not cross 90 Watts. The Bicameral Order would be a good name for such a group.

"you need to be thinking about implications" Thinking about it, won't put food on your table.

These are not necessarily similar to "exploit" codes, but more like a box of innocuous tricks often needed for good purposes. Gather enough of those throw enough compute, then you go from something that a human can perceive as obviously false, to something human-like.

There are plenty of people out there, with the competence to replicate those results, yet only a few big companies reap most the rewards, monopolize user data and interactions, and open-source for free results which could have been monetized by a company to provide useful services to users.

Seriously, how can one expect that most of those Phd who didn't get recruited don't use AI for nefarious uses?

There was an old science fiction novel set in a world like that. I've forgotten the name, but they called the robot-written stuff "wordwooze". (It doesn't work as a search term because some publisher is using it now.)

Added: it was The Silver Eggheads by Fritz Leiber. An odd, forgettable book itself.

I like how it effortlessly switches into "git diff" mode at the end of sample 112. Sadly it doesn't do whitespace.

> Showing 1 changed file with 4 additions and 19 deletions. +4 −19 png_source/colors/pointer.py Show comments View 8 png_source/colors/pointer.py @@ -35,6 +35,7 @@ def _draw_hull_class_level(self): repr(Shape[td_get_framepanel_pcs(dc) for dc in xrange(dc.cols)]), self.doublesize.values) \ } def _draw_hull_class_level [etc.]

It also inserted a helpful reference link to http://wiki.openarcade.com/wiki/List_of_Programmer_Constants (I've had to check: no, there is no openarcade wiki.)

Also, sample 217 is some mediocre Java, with comments and all. Impressive how a single model can handle this all at once.