121–130 of 212 posts
Whis was built on Vercel and its shitting the bed right now
Well, you're not wrong :) Vercel is not the one to blame here, it's my skill issue. Entire thing was vibecoded by me — product manager with no production dev experience. Not to promote vibecoding, but I couldn't do it myself the other way.
We know how to compute the best moves in poker (it's computationally challenging; the more choices and players are present, the more likely it is that most attempts only even try at heads-up).
With all that said, I do think there's a way to use attention and BERT to solve poker (when trained on non-text sequences). We need a better corpus of games and some training time on unique models. If anyone is interested, my email is elliott.neil.clark @ gmail.com
I have PhD in algorithmic game theory and worked on poker. 1) There are currently no algorithms that can compute deterministic equilibrium strategies [0]. Therefore, mixed (randomized) strategies must be used for professional-level play or stronger. 2) In practice, strong play has been achieved with: i) online search and ii) a mechanism to ensure strategy consistency. Without ii) an adaptive opponent can learn to exp…
> LLMs do not have a mechanism for sampling from given probability distributions. They could have a tool for that, tho.
In general games of imperfect information such as Poker, Diplomacy, etc are much much harder than perfect information games such as Chess.
Multiplayer (3+) poker in particular is interesting because you cannot achieve a nash equilibrium (e.g. it is not zero sum).
That is part of the reason they are a fantastic venue for exploration of the capabilities of LLMs. They also mirror the decision making process of real life. Bezos framed it as "making decisions with about 70% of the information you wish you had."
As it currently stands having built many poker AIs, including what I believe to be the current best in the world, I don't think LLMs are remotely close to being able to do what specialized algorithms can do in this domain.
All of the best poker AI's right now are fundamentally based on counter factual regret minimization. Typically with a layer of real time search on top.
Noam Brown (currently director of research at OpenAI) took the existing CFR strategies which were fundamentally just trying to scale at train time and added on a version of search, allowing it to compute better policies at TEST TIME (e.g. when making decisions). This ultimately beat the pros (Pluribus beat the pros at 6 max in 2018 I believe). It stands as the state of the art, although I believe that some of the deep approaches may eventually topple it.
Not long after Noam joined OpenAI they released the o1-preview "thinking" models, and I can't help but think that he took some of his ideas for test time compute and applied them on top of the base LLM.
It's amazing how much poker AI research is actually influencing the SOTA AI we see today.
I would be surprised if any general purpose model can achieve true human level or super human level results, as the purpose built SOTA poker algorithms at this point play substantially perfect poker.
Background:
- I built my first poker AI when I was in college, made half a million bucks on party poker. It was a pseudo expert system. - Created PokerTableRatings.com and caught cheaters at scale using machine learning on a database of all poker hands in real time - Sold my poker AI company to Zynga in 2011 and was Zynga Poker CTO for 2 years pre/post IPO - Most recently built a tournament version of Pluribus (https://www.science.org/doi/10.1126/science.aay2400). Launching as duolingo for poker at pokerskill.com
I am the author/maintainer of rs-poker ( https://github.com/elliottneilclark/rs-poker ). I've been working on algorithmic poker for quite a while. This isn't the way to do it. LLMs would need to be able to do math, lie, and be random. None of which are they currently capable. We know how to compute the best moves in poker (it's computationally challenging; the more choices and players are present, the more likely it…
E.g. given a small code execution environment, it could use some secure random generator to pick between options, it could use a calculator for whatever math it decides it can't do 'mentally', and they are very capable of deception already, even more so when the RL training target encourages it.
I'm not sure why you couldn't train an LLM to play poker quite well with a relatively simple training harness.
I am the author/maintainer of rs-poker ( https://github.com/elliottneilclark/rs-poker ). I've been working on algorithmic poker for quite a while. This isn't the way to do it. LLMs would need to be able to do math, lie, and be random. None of which are they currently capable. We know how to compute the best moves in poker (it's computationally challenging; the more choices and players are present, the more likely it…
what makes you say this? modern LLMs (the top players in this leaderboard) are typically equipped with the ability to execute arbitrary Python and regularly do math + random generations.
I agree it's not an efficient mechanism by any means, but I think a fine-tuned LLM could play near GTO for almost all hands in a small ring setting
I have PhD in algorithmic game theory and worked on poker. 1) There are currently no algorithms that can compute deterministic equilibrium strategies [0]. Therefore, mixed (randomized) strategies must be used for professional-level play or stronger. 2) In practice, strong play has been achieved with: i) online search and ii) a mechanism to ensure strategy consistency. Without ii) an adaptive opponent can learn to exp…
What are you working on specifically? I've been vaguely following poker research since Libratus, the last paper I've read is ReBeL, has there been any meaningful progress after that? I was thinking about developing a 5-max poker agent that can play decently (not superhumanly), but it still seems like a kind of uncharted territory, there's Pluribus but limited to fixed stacks, very complex and very computationally dem…
CFR is still the best, however, like chess, we need a network that can help evaluate the position. Unlike chess, the hard part isn't knowing a value; it's knowing what the current game position is. For that, we need something unique.
I'm pretty convinced that this is solvable. I've been working on rs-poker for quite a while. Right now we have a whole multi-handed arena implemented, and a multi-threaded counterfactual framework (multi-threaded, with no memory fragmentation, and good cache coherency)
With BERT and some clever sequence encoding we can create a powerful agent. If anyone is interested, my email is: elliott.neil.clark@gmail.com
Earlier quoted context omitted.
How much is needed to get past those? The third one is solvable by giving them a basic tool call, or letting them write some code to run.
I agree, but they should come up with the distribution as well. If you directly give the distribution to the LLM, it is not doing anything interesting. It is just sampling from the strategy you tell it to play.
Earlier quoted context omitted.
>> but I recall there was some paper that showed you can get something like 98% of equilibrium utility in poker subgames, which could make deterministic strategy practical. (Can't find the paper now.) Yeah I can see that for sure. That's also a holy grail of a poker enthusiast "can we please have non-mixed solution that is close enough". The problem is that 2% or even 1% equilibrium utility is huge. Professional play…
I agree it is possible to build an LLM to play poker, with appropriate tool calling, in principle. I think it's useful to distinguish what LLMs can do in a) theory, b) non-LLM approaches we know work and c) how to do it with LLMs. In a) theory, LLMs with the "thinking" rollouts are equivalent to (finite-tape) Turing machine, so they can do anything a computer can, so a solution exists (given large-enough neural net/r…
>>On top of that, for poker, you'd need LLM to somehow implement continual resolving/ReBeL (for equilibrium guarantees). To do all of this, you need either i) LLM call the CPU implementation of the resolver or ii) the LLM to execute instructions like a CPU.
Maybe we don't. Maybe there are general patterns that LLM could pick up so it could make good decisions in all branches without resolving anything, just looking at the current state. For example LLM could learn to automatically scale calling/betting ranges depending on the bet size once it sees enough examples of solutions coming from algorithms that use resolving.
I guess what I am getting at is that intuitively there is not that much information in poker solutions in comparison to chess so there are more general patterns LLMs could pick up on.
I remember the discussion about the time heads-up limit holdem was solved and arguments that it's bigger than chess. I think it's clear now that solution to limit holdem is much smaller than solution to chess is going to be (and we haven't even started on compression there that could use internal structure of the game). My intuition is that no-limit might still be smaller than chess.
>>I see. The same applies for Chess however -- you can play mixed strategies there too, with similar property - you can linearly interpolate expected value between losing (-1) and winning (1).
I mean that in chess the same move in seemingly similar situation might be completely wrong or very right and a little detail can turn it from the latter to the former. You need a very "precise" pattern recognition to be able to distinguish between those situations. In poker if you know 100% calling with a top pair is right vs a river pot bet you will not make a huge mistakes if you 100% call vs 80% pot bet for example.
When NN based engines appeared (early versions of Lc0) it was instantly clear they have amazing positional "understanding" but get lost quickly when the position required a precise sequence of moves.