Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Go grandmaster Shin defeats AI KataGo with a two-stone handicap

kedglobal.com

Post reply on HN
Print view View original

71–80 of 187 posts

Re: Go grandmaster Shin defeats AI KataGo with a two-stone handicap

#71

Post by DiogenesKynikos »

hyperpape wrote: post #37
rao-v wrote: post #5

It’s worth understanding that Shin Jinse has been significantly stronger than his nearest human opponents for a while now, more so than Magnus was even at his very peak. In go ELO like scoring he’s something like 120 points over the next strongest player. No other player has ever broken a 3800 rating let alone 3850. Ke Jie (the previous long time champion) peaked at 3755. Shin Jinseo’s strength graph is the most absu…

You're right that Shin Jinseo is a generational talent, and more dominant than anyone since Lee Changho (peaked in the 90s and was strong into the early-mid 2000s). However, you can't compare goratings over time, the top ranks are not nearly stable enough. https://www.goratings.org/en/history/ (I think it's believable Shin Jinseo is better than Lee Changho, but not that there has been steady progress since the days o…

You can't compare Elo ratings over long stretches of time, period.

Ratings drift over time, based on the total population of people competing. I think the most accurate way to view Elo ratings is as a measure of skill vs. the average rated player.

If you want to compare Magnus Carlsen's peak rating of 2882 in the year 2014 to Garry Kasparov's peak rating of 2851 in the year 1999, you have to know how strong the total pool of players (including all the amateurs who compete at lower levels) was in the year 2014 vs. 1999.

The only way to actually anchor the Elo system over long periods of time would be to have rated humans occasionally play against a set of unchanging computer players, which could then serve as static rating calibrators. You could use those games to then calibrate Elo ratings from different time periods to a common scale, by asserting that the computer ratings don't change.

Re: Go grandmaster Shin defeats AI KataGo with a two-stone handicap

#72

Post by tucnak »

dlevine wrote: post #4

Apparently 2 stones is a huge advantage. An estimate is that the computer is roughly 4-600 ELO stronger on an even match. Also, the human played a strategy tailored to that huge initial advantage. He said that the AI did not handle this particularly well, and played high probability moves instead of trying to lure him into a mistake. Also, even though this was the best Go engine, it was not running on a supercomputer…

I'm sorry, but Elo is not an acronym for Electronic Light Orchestra. You don't write ELO, but simply Elo.

Re: Go grandmaster Shin defeats AI KataGo with a two-stone handicap

#73

Post by Davidzheng »

thangalin wrote: post #16

In Go, there are exchanges of plays called "joseki". Professionals consider the outcome of joseki to be an equal result for both players. Most joseki are only a handful of moves, but some, such as the "flying knife" joseki have variations that continue for upwards of 50 moves. A traditional 19x19 go board has 361 intersections. Shin's genius was to play out a complex variation of the flying knife joseki that was, in…

In odds chess bots, the bots would willingly take more disadvantageous positions which are more complicated--probably the bots in GO which are trained for odds do similar? Why does it not avoid such a joseki & play a worse response which it believes the human cannot read?

Re: Go grandmaster Shin defeats AI KataGo with a two-stone handicap

#74

Post by somenameforme »

hyperpape wrote: post #33
zelphirkalt wrote: post #32

Earlier quoted context omitted.

I read this comment before looking at the article and thought that the grandmaster beat the AI even giving the AI 2 stones. Too bad. But this way around is of course more realistic. And of course you would need to take into consideration the scale of go ratings and chess ratings when making that comparison. With top chess ratings being around 2800, being 1000 less than the top go ratings, one would have to apply a fa…

If you scaled Shin Jinseo to 2800, you would have players with extremely negative ratings. This page shows ratings of European players on a roughly aligned scale: https://europeangodatabase.eu/EGD/createalleuro3.php?country... . It still has negative numbers on it, and this only contains players who have attended a tournament (though it's more common for beginners to play tournaments in the west, since it's hard to f…

I'm unfamiliar with Go ratings, but chess ratings are based on the Elo system which is a simple mathematical prediction system. Borrowing some figures from Wiki [1] we get:

  1.00 +800
  0.99 +677
  0.9 +366
  0.8 +240
  0.7 +149
  0.6 +72
  0.5 0
  0.4 −72
  0.3 −149
  0.2 −240
  0.1 −366
  0.01 −677
  0.00 −800
The second column is your rating minus your opponent's, and the left is your predicted result. So if you are rating 1849 and your opponent is rated 1700 then you'd be expected to score about 70%. To have a 1% expected score against Magnus, you'd need a rating of about 2150.

[1] - https://en.wikipedia.org/wiki/Elo_rating_system

Re: Go grandmaster Shin defeats AI KataGo with a two-stone handicap

#75

Post by choedev »

Davidzheng wrote: post #73
thangalin wrote: post #16

In Go, there are exchanges of plays called "joseki". Professionals consider the outcome of joseki to be an equal result for both players. Most joseki are only a handful of moves, but some, such as the "flying knife" joseki have variations that continue for upwards of 50 moves. A traditional 19x19 go board has 361 intersections. Shin's genius was to play out a complex variation of the flying knife joseki that was, in…

In odds chess bots, the bots would willingly take more disadvantageous positions which are more complicated--probably the bots in GO which are trained for odds do similar? Why does it not avoid such a joseki & play a worse response which it believes the human cannot read?

Go AIs tend to naturally be quite bad at playing handicap games, due to the horizon effect. To massively simplify, when the AI sees that there's a large score gap, it realizes that every move it plays has a very low/high win rate, so it basically picks one at random. The early AIs played lots of slack moves when they were ahead, often making small endgame mistakes but winning by half a point in the end.

To account for this, KataGo uses a "playout doubling factor". When the AI plays against itself to learn, the developers set one instance of the AI to use fewer playouts compared to the other one, but gives the weaker AI some handicap. This allows the AI with more playouts to learn that although it may be in a losing position, if it makes the board position chaotic enough, it may still win.

The flying knife is objectively an extremely complicated position, so the AI played it assuming that the opponent would be forced into a very complicated reading battle where they could make some mistakes. Unfortunately, Shin has memorized the flying knife joseki more thoroughly than any other human on the planet, so he could play exactly like a very strong AI. It would probably be possible to train an adversarial network specifically to beat players like Shin, but that would take a substantial amount of effort, and Shin is strong enough that it probably wouldn't make too much of a difference -- Shin won by 11.5 points in game 3 without a flying knife shenanigans, only losing 7 points of value throughout the entire game.

Re: Go grandmaster Shin defeats AI KataGo with a two-stone handicap

#76

Post by somenameforme »

DiogenesKynikos wrote: post #71
hyperpape wrote: post #37

Earlier quoted context omitted.

You're right that Shin Jinseo is a generational talent, and more dominant than anyone since Lee Changho (peaked in the 90s and was strong into the early-mid 2000s). However, you can't compare goratings over time, the top ranks are not nearly stable enough. https://www.goratings.org/en/history/ (I think it's believable Shin Jinseo is better than Lee Changho, but not that there has been steady progress since the days o…

You can't compare Elo ratings over long stretches of time, period. Ratings drift over time, based on the total population of people competing. I think the most accurate way to view Elo ratings is as a measure of skill vs. the average rated player. If you want to compare Magnus Carlsen's peak rating of 2882 in the year 2014 to Garry Kasparov's peak rating of 2851 in the year 1999, you have to know how strong the total…

Even that would be slightly malinformative because of opening theory. If you warped a very strong player from the past to the present, he'd do very poorly at first simply because of advances in opening theory. But give him a bit to catchup and he'd likely have his rating zoom on up. So modern players would do better against the static computer because of the same advantage, but that doesn't mean they're necessarily stronger in the sense that we hope to measure. The question people always want to know are things like how would a Morphy, Capablanca, or Alekhine do in modern times with access to modern theory and the like - not how well would they do against Carlsen if they went in with nothing but the knowledge of their era.

Re: Go grandmaster Shin defeats AI KataGo with a two-stone handicap

#77

Post by 317070 »

logicallee wrote: post #27
bluecalm wrote: post #18

KataGo isn't very good at exploiting weaker opponents. In chess people were convinced a grandmaster can never be beaten with a knight odds. It's just too easy to simplify the position and win. It was very easy (for a grandmaster) vs already super human Stockfish. It was still kinda easy (for a strong GM) vs 200+ ELO stronger NNUE Stockfish. And then someone made a net optimized for exploiting humans. Its games are am…

can the grandmasters still win with rook odds?

they can win with any odds. But, they can only barely win even with queen odds: https://lqo.leumon.com/

Re: Go grandmaster Shin defeats AI KataGo with a two-stone handicap

#78

Post by somenameforme »

harshreality wrote: post #56
afthonos wrote: post #24

Earlier quoted context omitted.

Important to note that KataGo was double-handicapped. 20 seconds per move maximum; it couldn’t read deep. Against an amateur, it doesn’t matter, but against a historically strong pro it matters a lot.

...on a 4x 3090 rig. The game ran 299 moves, giving katago 100 minutes if it exhausted time on each move (which must be the optimal strategy under that time control). Shin used about 205 minutes, over twice as much time and of course had leeway to spend more time on difficult moves. Based on the youtube video, it looks like katago was only using 16 seconds per move, is that right? https://www.youtube.com/watch?v=-86z…

As another question, does it not operate similarly to the top chess engines? The way the neural network systems work is by using the probabilistic matching paired with a Monte Carlo simulation. So you can get to extreme depth very rapidly. Obviously the breadth is going to be limited, but if the neural network side is well tuned (so high probability hits are indeed generally the most challenging moves), then that's not such a problem.

And you can run a huuuuuuuge number of sims in 16 seconds.

Re: Go grandmaster Shin defeats AI KataGo with a two-stone handicap

#79

Post by geocar »

xhevahir wrote: post #23

> "This series taught me that rather than trying to imitate AI, it is far more important to build the board according to my own style." It's never made sense to me that so many go players study AI go play in the hope of emulating it in a human game. We're not machines. We can't do thousands of Monte Carlo tree searches per second.

> It's never made sense to me that so many go players study AI go play in the hope of emulating it in a human game.

> We're not machines. We can't do thousands of Monte Carlo tree searches per second.

You sound so sure of that, but I have seen people catch a ball, and I am not so sure that any artificial person should be any more aware of the tremendous maths they are "solving"

From the perspective of a game with far fewer rules, "trying to imitate AI" might not mean anything like what you are thinking.

Re: Go grandmaster Shin defeats AI KataGo with a two-stone handicap

#80

Post by CurtMonash »

mtlmtlmtlmtl wrote: post #30
rao-v wrote: post #5

It’s worth understanding that Shin Jinse has been significantly stronger than his nearest human opponents for a while now, more so than Magnus was even at his very peak. In go ELO like scoring he’s something like 120 points over the next strongest player. No other player has ever broken a 3800 rating let alone 3850. Ke Jie (the previous long time champion) peaked at 3755. Shin Jinseo’s strength graph is the most absu…

Probably a better comparison from the chess world(in reasonably modern times, though perhaps players like Capablanca and Lasker could be mentioned as well. Alas, I don't think FIDE rating existed back then) is Bobby Fischer. In the july 1972 FIDE rating list he held a rating of 2785, the highest in history at the time, with Spassky in second sitting at a "measly" 2660, and only 13 players being above 2600 even.

If I counted corrected, Fischer went 24-3 in the world championship series around then. That excludes many draws and one forfeit vs. Spassky, a few draws vs. Petrosian, and nothing at all in his sweeps of Larsen and Taimonov.
Post reply on HN