ChatGPT outperforms crowd-workers for text-annotation tasks

Edit: Upon further investigation it seems like a lot of those are not technically bills, but rather House or Senate resolutions. Explained here: https://www.senate.gov/legislative/common/briefing/leg_laws_... The govinfo.gov list seems to indicate that such resolutions are pretty common, but significantly less so than bills, so I'm not sure why the front page seems to consist only of resolutions right now.

1. The comment with the summary is made like any other comment. (In the case of Reddit, that’s perhaps mainly because the mods have no other option, but the point still stands.) Because of this, other users can downvote the bot comment if it is incorrect, as well as respond to it with corrections. Exactly the same way as you’d interact with a person in the past if their summary comment was incorrect; downvote them and/or correct them.

2. Include a disclaimer stating that the summary may be inaccurate. Encourage people to correct and/or downvote bad summaries in the disclaimer

MTurk is often used in these tasks in place of more expensive human annotators (e.g. grad students), and this paper says that for their case, at least, ChatGPT worked better, in the sense that, given the exact same instructions, it gave answers closer to the more-expensive annotators. Using MTurk seriously often entails extra steps intended to verify motivation, e.g. adding a question that says "select option 7" to make sure the person isn't just making random choices, or gathering more answers for questions where there was disagreement between annotators. What these extra steps have in common is that they take both more time when designing the labeling process, and cost more money.

On the surface of it, I'd expect ChatGPT to do very well at this. It's simple text and images, not many options and theoretically very limited context.

However the more I think about it the less sure I am. Firstly these weren't crowd-sourced reviews, they were trained reviewers, paid hourly not per review. Incentives were definitely in favour of the long term business relationship. Then there was the training doc, we maintained a vast disambiguation doc used to resolve things that were vague or could be interpreted multiple ways, this was constantly being revised. All necessary context should have been in that but it wasn't and reviewers definitely found patterns that worked and didn't. Lastly the reviewers were in a Slack channel where they would ask questions to their manager on our side, and while this might have only been ~1% of tasks, it was an important process.

So maybe you could point ChatGPT at it and let it run, but the oversight process we had would still be necessary. The disambiguation doc would have been too long for ChatGPT's context at the moment, but that will likely change in the near future. Would the workflow be to keep tweaking the prompt to add special case after special case? How do you scale "do this, but not that, but add this, but..." in prompting, and would ChatGPT become as confused as a human after enough of that – I expect so given that it's only a language model and that's not effective communication.

https://www.opensamizdat.com/posts/chatgpt_survey/