Building an AI that watches rugby

Better title: "LLM OCR on Rugby screenshots to read score and clock"

I don't quite get how diffing frames allows you to find the scores.

TFA mentions comparing a frame with and without - but how do you generate that frame without? If you can already do it, what's useful about doing that?

Why the focus on scorekeeping? I feel like an AI model is overkill here, when you have text-based sources readily available such as news apps, Twitter feeds, and apps such as Livescore which would be easier and cheaper to scrape. They probably cover more matches that aren't televised too.

I'd be curious to see what useful insights could be gleamed from the match commentary. You have the main commentator giving play-by-play objective reporting and then a 'colour' commentator giving some subjective analysis during breaks in play. I bet there's a lot of interesting ways this could be used.

The only interesting part of the model's output was

{ "current_play": "ruck", }

So the vision model can correctly identify that there's a ruck going on and that the ball is most likely in the ruck.

Why not build on this? Which team is in possession? Who was the ball carrier at the start of the ruck, and who tackled him? Who joined the ruck, and how quickly did they get there? How quickly did the attacking team get the ball back in hand, or the defending team turn over possession? What would be a good option for the outhalf if he got the ball right now?

All of these except the last would be straightforward enough for a human observer with basic rugby knowledge going through the footage frame by frame, and I bet it would be really valuable to analysts. It seems like computer vision technology is at a stage where this could be automated too.

I want AIs that clean my apartment while I watch rugby, not AIs that watch rugby while I clean the apartment. ;)

In seriousness, this is a cool project and show how sophisticated analysis LLMs can do in a plug and play manner. They may not always be the best solution but a fantastic baseline that can be deployed and adapted to a usecase in less than an hour.