'pcwalton, I'm coming for you. You're going down. Kidding aside, the comments it picks out for us are a little random. For instance, this was an A+ predictive thread (it appears to be rating threads and not individual comments): https://news.ycombinator.com/item?id=10703512 But there's just 11 comments, only 1 for me, and it's like a 1-sentence comment. I do love that my unaccredited-access-to-startup-shares take is…
Auto-grading decade-old Hacker News discussions with hindsight
231–240 of 285 posts
Re: Auto-grading decade-old Hacker News discussions with hindsight
#232Earlier quoted context omitted.
"Boring but right" generally means that this prediction is already priced in to our current understanding of the world though. Anyone can reliably predict "the sun will rise tomorrow", but I'm not giving them high marks for that.
something like correctness^2 x novel information content rank?
Re: Auto-grading decade-old Hacker News discussions with hindsight
#233'pcwalton, I'm coming for you. You're going down. Kidding aside, the comments it picks out for us are a little random. For instance, this was an A+ predictive thread (it appears to be rating threads and not individual comments): https://news.ycombinator.com/item?id=10703512 But there's just 11 comments, only 1 for me, and it's like a 1-sentence comment. I do love that my unaccredited-access-to-startup-shares take is…
Hilariously, it seems you anticipated this happening and copyrighted your comments. Is karpathy's tool in violation of your copyright?!
Re: Auto-grading decade-old Hacker News discussions with hindsight
#234It's fun to read some of these historic comments! A while back I wrote a replay system to better capture how discussions evolved at the time of these historic threads. Here's Karpathy's list from his graded articles, in the replay visualizer: Swift is Open Source https://hn.unlurker.com/replay?item=10669891 Launch of Figma, a collaborative interface design tool https://hn.unlurker.com/replay?item=10685407 Introducing…
Comment dates on hn frontend are sometimes altered when submissions are merged, do you handle this case properly?
Re: Auto-grading decade-old Hacker News discussions with hindsight
#235I've found the opposite, since these models still fail pretty wildly at nuance. I think it's a conceptual "needle in the haystack sort of problem.
A good test is to find some thread where there's a disagreement and have it try to analyze the discussion. It will usually strongly misrepresent what was being said, by each side, and strongly align with one user, missing the actual divide that's causing the disagreement (a needle).
Re: Auto-grading decade-old Hacker News discussions with hindsight
#236The company has changed and it seems the mission has as well.
Re: Auto-grading decade-old Hacker News discussions with hindsight
#237> I realized that this task is actually a really good fit for LLMs I've found the opposite, since these models still fail pretty wildly at nuance. I think it's a conceptual "needle in the haystack sort of problem. A good test is to find some thread where there's a disagreement and have it try to analyze the discussion. It will usually strongly misrepresent what was being said, by each side, and strongly align with on…
Re: Auto-grading decade-old Hacker News discussions with hindsight
#238Earlier quoted context omitted.
How would you possibly grade comments if you change them?
Extract the concrete predictions, evaluate them as true/false/indeterminate, and grade the user on the number of true vs false?
I got an A for commenting on DF saying that I had not personally seen save corruption and listing weird bugs. It's true that weird bugs have long been a defining feature of DF, but I didn't predict it would remain that way or say that save corruption would never be a big thing, just that I hadn't personally seen it.
Another A for a comment on Google wallet just pointing out that users are already bad at knowing what links to trust. Sure, that's still true (and probably will remain true until something fundamental changes), but it was at best half a prediction as it wasn't forward looking.
Then something on hospital airships from the 1930s. I pointed out that one could escape pollution, I never said I thought it would be a big thing. Airships haven't really ever been much of a thing, except in fiction. Maybe that could change someday, but I kinda doubt it.
Then lastly there was the design patent famously referred to as the "rounded corner" patent. It dings me for simplifying it to that label, despite my actual statements being that yes, there's more, but just minor details like that can be sufficient for infringement. But the LLM says I'm right about ties to the Samsung case and still oversimplifying it. Either way, none of this was really a prediction to begin with.
Re: Auto-grading decade-old Hacker News discussions with hindsight
#239Earlier quoted context omitted.
I long had a similar idea for stocks. Analyze posts of people giving stock tips on WSB, Twitter, etc and rank by accuracy. I would be very surprised if this had not been done a thousand times by various trading firms and enterprising individuals. Of course in the above example of stocks there are clear predictions (HNWS will go up) and an oracle who resolves it (stock market). This seems to be a way harder problem fo…
> Analyze posts of people giving stock tips on WSB, Twitter, etc and rank by accuracy. Didn't somebody make an ETF once that went against the prediction of some famous CNBC stock picker, showing that it would have given you alpha in the past. > seems to be a way harder problem for generic free form comments. That's what prediction markets are for. People for whom truth and accuracy matters (often concentrated around…
https://finbold.com/inverse-cramer-leaves-sp-nasdaq-and-dow-...
Re: Auto-grading decade-old Hacker News discussions with hindsight
#240If you dig in, there are substantial flaws in the project's analysis and framing, such as the definition of a prediction, assessing comments, data quality overall, and more. Go spelunking through the comments here and notice people asking about methodology and checking the results.
Social science research isn't easy; it requires training, effort, and patience. I would be very happy if Karpathy added a Big Flashing Red Sign to this effect. It would raise awareness and focus community attention on what I think are the hardest and most important aspects of this kind of project: methodology, rigor, criticism, feedback, and correction.