Live data from Hacker News

Show HN: A GitHub Action that quizzes you on a pull request

github.com

1–10 of 35 posts

Show HN: A GitHub Action that quizzes you on a pull request

#1
A little idea I got from playing with AI SWE Agents. Can AI help make sure we understand the code that our AIs write?

PR Quiz uses AI to generate a quiz from a pull request and blocks you from merging until the quiz is passed. You can configure various options like the LLM model to use, max number of attempts to pass the quiz or min diff size to generate a quiz for. I found that the reasoning models, while more expensive, generated better questions from my limited testing.

Privacy: This GitHub Action runs a local webserver and uses ngrok to serve the quiz through a temporary url. Your code is only sent to the model provider (OpenAI).

Show HN: A GitHub Action that quizzes you on a pull request
github.com

Re: Show HN: A GitHub Action that quizzes you on a pull request

#3
> AI Agents are starting to write more code. How do we make sure we understand what they're writing?

This is a good question, but also how do we make sure that humans understand the code that _other humans_ have (supposedly) written? Effective code review is hard as it implies that the reviewer already has their own mental model about how a task could/would/should have been done, or is at the very least building their own mental model at reading-time and internally asking 'Does this make sense?'.

Without that basis code review is more like a fuzzy standards compliance, which can still be useful, but it's not the same as review process that works by comparing alternate or co-operatively competing models, and so I wonder how much of that is gained through a quiz-style interaction.

Re: Show HN: A GitHub Action that quizzes you on a pull request

#4
See, I think this is a good idea even for reviewing non-agentic human-written PRs!

We've got a huge LGTM problem where people approve PRs they clearly don't understand.

Recently we had a bug in some code of an employee that got laid off. The people who reviewed it are both still with the company, but neither of them could explain what the code did.

That triggered this angry tweet

https://x.com/donatj/status/1945593385902846118

Re: Show HN: A GitHub Action that quizzes you on a pull request

#6

> AI Agents are starting to write more code. How do we make sure we understand what they're writing? This is a good question, but also how do we make sure that humans understand the code that _other humans_ have (supposedly) written? Effective code review is hard as it implies that the reviewer already has their own mental model about how a task could/would/should have been done, or is at the very least building thei…

I imagine the quizzer could ask better questions along those lines with better context engineering (taking entire repo contents, design docs, discussions, etc and compressing those into a mental model). I just took the PR code changes and comments, so there's a lot of improvements that could be made there.

Re: Show HN: A GitHub Action that quizzes you on a pull request

#9
post #4

See, I think this is a good idea even for reviewing non-agentic human-written PRs! We've got a huge LGTM problem where people approve PRs they clearly don't understand. Recently we had a bug in some code of an employee that got laid off. The people who reviewed it are both still with the company, but neither of them could explain what the code did. That triggered this angry tweet https://x.com/donatj/status/194559338…

Could definitely be used for human PRs too! Though I'm sure companies would love to track the reviewer scores

Re: Show HN: A GitHub Action that quizzes you on a pull request

#10
That’s a fun take on a real issue, but…

> Your code is only sent to the model provider (OpenAI)

When has this become an acceptable « privacy » statement?

I feel we are reliving the era of free mobile apps at the expense of harvesting any user data for ads profiling before GDPR kicked in…

Post reply on HN