Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Show HN: ChessCoach – A neural chess engine that comments on each player's moves

chrisbutner.github.io

Post reply on HN
Print view View original

41–50 of 92 posts

Re: Show HN: ChessCoach – A neural chess engine that comments on each player's moves

#41

Post by cbutner »

a_t48 wrote: post #21
cbutner wrote: post #10

Earlier quoted context omitted.

Yeah, that's a massive problem with the natural language domain all across machine learning. Unfortunately it's very difficult to track down training data for chess commentary in the first place, let alone trim down biases. For reference, I was able to gather about 1 million samples, but it really needs a billion. Hopefully through data augmentation and better general intelligence models we can make better progress o…

You might be able to kludge a fix to tokenize the output and replace he/him/she/her with them/their. It's not as sexy as the engine outputting the correct words, but it should get the job done.

Yes, in this case as long as they still agree when it actually names people, I don't think it would be too difficult. There may be factors I'm not considering though.

Harder would be more general models like GPT-2 and GPT-3.

Re: Show HN: ChessCoach – A neural chess engine that comments on each player's moves

#42

Post by cbutner »

phendrenad2 wrote: post #37

Looking at the screenshot, I was briefly hopeful that they used actual Chess commentary and gpt3 or something. But it looks like a simple rule engine is translating "user did X but I (the AI) would have done Y" into a slightly nicer phrasing. Meh. Still cool.

It is using a full-sized transformer decoder, trained on about 1 million data samples, but with far fewer neural network parameters and training samples than GPT-2 or GPT-3.

Re: Show HN: ChessCoach – A neural chess engine that comments on each player's moves

#43

Post by a_t48 »

cbutner wrote: post #41
a_t48 wrote: post #21

Earlier quoted context omitted.

You might be able to kludge a fix to tokenize the output and replace he/him/she/her with them/their. It's not as sexy as the engine outputting the correct words, but it should get the job done.

Yes, in this case as long as they still agree when it actually names people, I don't think it would be too difficult. There may be factors I'm not considering though. Harder would be more general models like GPT-2 and GPT-3.

Singular "they" doesn't care about the gender of the person named, so it should be good.

Re: Show HN: ChessCoach – A neural chess engine that comments on each player's moves

#44

Post by alana314 »

potatoman22 wrote: post #30
butwhywhyoh wrote: post #28

Earlier quoted context omitted.

What makes it unfortunate that it used masculine pronouns? If it had used purely female pronouns would that make you feel better?

Probably not. The issue with always using one set of pronouns is that people with other pronouns may feel isolated. For example, always assuming software devs are male could cause female devs to feel like they don't belong. Realistically, it's really hard to fix this encoded bias in language models.

It does. Thanks for expressing this.

Re: Show HN: ChessCoach – A neural chess engine that comments on each player's moves

#45

Post by cbutner »

thomasahle wrote: post #26

So do I understand correctly: This is a new head on top of the AlphaZero model? That is, in addition to the usual evaluation and policy heads, this takes the intermediate board representation and outputs a seed vector that is fed into a transformer text generator? Or do other things go into the seed? Like the search tree somehow? Otherwise I suppose the commentary will not be able to comment on deeper tactics? Or may…

The original hope was for this to be a third head on top of the AlphaZero model, but I couldn't think of a way to generate commentary during self-play (such that it would gradually improve), and trying to rotate supervised commentary training into the main schedule ended up hurting both sides because of the disjoint datasets.

So, now the commentary decoder is just trained separately on the final primary model. The previous and current game positions are fed into the primary model, and the outputs are taken from the final convolutional layer, just before the value and policy heads. Then, that data plus the side to play is positionally encoded and fed into a transformer decoder.

It would be better for a search tree/algorithm to be used for commentary too so that tactics could be better understood, but that would need some kind of subjective BLEU equivalent, and metrics like those don't work well for chess commentary.

You can see a diagram of the architecture here: https://chrisbutner.github.io/ChessCoach/high-level-explanat...

Re: Show HN: ChessCoach – A neural chess engine that comments on each player's moves

#46

Post by owenmarshall »

eeegnu wrote: post #19

Just tried it out, really interesting to read post game: https://lichess.org/dKbbqymG/ I don't think this comment was accurate: > 23. c4: "White tries to get his pawns moving. I am still thinking that I have to move my bishop, but that is too slow. I think white should have moved his king back to c3 to prevent my pawn from becoming a passer." The idea behind moving the pawn was because black playing c4 would have ins…

The gameplay seems solid, but the comments are all over the map:

https://lichess.org/4l1urWeU

I wonder if we are getting snippets of variations in some cases.

Re: Show HN: ChessCoach – A neural chess engine that comments on each player's moves

#47

Post by cassonmars »

potatoman22 wrote: post #30
butwhywhyoh wrote: post #28

Earlier quoted context omitted.

What makes it unfortunate that it used masculine pronouns? If it had used purely female pronouns would that make you feel better?

Probably not. The issue with always using one set of pronouns is that people with other pronouns may feel isolated. For example, always assuming software devs are male could cause female devs to feel like they don't belong. Realistically, it's really hard to fix this encoded bias in language models.

Yeah it’s always annoyingly jarring when people assume that. You can even see in the comments on this topic that people attack the suggestion of gender neutral language, which further compounds on the feeling like we don’t belong or are explicitly unwanted.

Re: Show HN: ChessCoach – A neural chess engine that comments on each player's moves

#48

Post by mcyc »

cbutner wrote: post #2

This took about a year and a half – a little over a year coding in between experiments and training. It's a chess engine with a primary neural network just like AlphaZero or Leela Chess Zero's, but it adds on a secondary "commentary decoder" network based on Transformer architecture to comment on positions and moves. All of the code and data for training and search is from scratch, although it does use Stockfish code…

This is a fantastic project. Thanks for sharing!

I had a nice long conversation with two of the authors of [0] at ACL.

One thing we discussed was the reverse problem. That is, as a player, could I give commands to the model and have the engine figure the moves that would best satisfy them.

This ranges from concrete like "take the black square bishop" (there is still variability like which piece should take it or if it's even possible) to more complex positional stuff like "set up to attack the kingside."

Any thoughts on this line of research?

[0] Automated Chess Commentator Powered by Neural Chess Engine (Zang, Yu & Wan, 2019) https://arxiv.org/pdf/1909.10413.pdf

Re: Show HN: ChessCoach – A neural chess engine that comments on each player's moves

#49

Post by cbutner »

owenmarshall wrote: post #46
eeegnu wrote: post #19

Just tried it out, really interesting to read post game: https://lichess.org/dKbbqymG/ I don't think this comment was accurate: > 23. c4: "White tries to get his pawns moving. I am still thinking that I have to move my bishop, but that is too slow. I think white should have moved his king back to c3 to prevent my pawn from becoming a passer." The idea behind moving the pawn was because black playing c4 would have ins…

The gameplay seems solid, but the comments are all over the map : https://lichess.org/4l1urWeU I wonder if we are getting snippets of variations in some cases.

It does train on variations too, given the scarcity of data available, so that can hurt accuracy, mood, etc.

Re: Show HN: ChessCoach – A neural chess engine that comments on each player's moves

#50

Post by cbutner »

mcyc wrote: post #48
cbutner wrote: post #2

This took about a year and a half – a little over a year coding in between experiments and training. It's a chess engine with a primary neural network just like AlphaZero or Leela Chess Zero's, but it adds on a secondary "commentary decoder" network based on Transformer architecture to comment on positions and moves. All of the code and data for training and search is from scratch, although it does use Stockfish code…

This is a fantastic project. Thanks for sharing! I had a nice long conversation with two of the authors of [0] at ACL. One thing we discussed was the reverse problem. That is, as a player, could I give commands to the model and have the engine figure the moves that would best satisfy them. This ranges from concrete like "take the black square bishop" (there is still variability like which piece should take it or if i…

SentiMATE[1] looks at one of the reverse problems in a way - training an engine on commentary data - although it's not exactly what you're talking about.

I think this line of thinking could eventually lead to automated metrics for commentary evaluation, which could in turn lead to better methods than top-k/top-p for turning a bunch of sequential logits into a sentence or paragraph - basically treat it like MCTS/PUCT also.

The problem is that if you look at high-level commentary - maybe Radjabov-MVL on https://www.chess.com/news/view/2021-champions-chess-tour-fi... (I'm not the best judge, just a quick search) - it's not often possible to predict the move starting with the comment. And if you did, you might end up with very dry metrics and reverse commentary.

But this direction has a lot of potential I think, beyond just chess, into more of an algorithmic/generational support for pure NN-based language models.

[1] https://arxiv.org/pdf/1907.08321.pdf

Post reply on HN