Live data from Hacker News

Auto-grading decade-old Hacker News discussions with hindsight

karpathy.bearblog.dev

231–240 of 285 posts

Re: Auto-grading decade-old Hacker News discussions with hindsight

#231
post #69

'pcwalton, I'm coming for you. You're going down. Kidding aside, the comments it picks out for us are a little random. For instance, this was an A+ predictive thread (it appears to be rating threads and not individual comments): https://news.ycombinator.com/item?id=10703512 But there's just 11 comments, only 1 for me, and it's like a 1-sentence comment. I do love that my unaccredited-access-to-startup-shares take is…

Hilariously, it seems you anticipated this happening and copyrighted your comments. Is karpathy's tool in violation of your copyright?!

Re: Auto-grading decade-old Hacker News discussions with hindsight

#232
post #84

Earlier quoted context omitted.

"Boring but right" generally means that this prediction is already priced in to our current understanding of the world though. Anyone can reliably predict "the sun will rise tomorrow", but I'm not giving them high marks for that.

something like correctness^2 x novel information content rank?

Actually now thinking about it, incorrect information has negative value so the metric should probably reflect that.

Re: Auto-grading decade-old Hacker News discussions with hindsight

#233
post #231
post #69

'pcwalton, I'm coming for you. You're going down. Kidding aside, the comments it picks out for us are a little random. For instance, this was an A+ predictive thread (it appears to be rating threads and not individual comments): https://news.ycombinator.com/item?id=10703512 But there's just 11 comments, only 1 for me, and it's like a 1-sentence comment. I do love that my unaccredited-access-to-startup-shares take is…

Hilariously, it seems you anticipated this happening and copyrighted your comments. Is karpathy's tool in violation of your copyright?!

Karpathy, I'm coming for you next.

Re: Auto-grading decade-old Hacker News discussions with hindsight

#234

It's fun to read some of these historic comments! A while back I wrote a replay system to better capture how discussions evolved at the time of these historic threads. Here's Karpathy's list from his graded articles, in the replay visualizer: Swift is Open Source https://hn.unlurker.com/replay?item=10669891 Launch of Figma, a collaborative interface design tool https://hn.unlurker.com/replay?item=10685407 Introducing…

Comment dates on hn frontend are sometimes altered when submissions are merged, do you handle this case properly?

It is handled on the Unlurker front page (you will see a little note that says “time adjusted for second chance”). The replay doesn’t do any adjustment for it, but I think that makes it reflect the reality of when the comments came in since the adjustments are like a temporary bump

Re: Auto-grading decade-old Hacker News discussions with hindsight

#235
> I realized that this task is actually a really good fit for LLMs

I've found the opposite, since these models still fail pretty wildly at nuance. I think it's a conceptual "needle in the haystack sort of problem.

A good test is to find some thread where there's a disagreement and have it try to analyze the discussion. It will usually strongly misrepresent what was being said, by each side, and strongly align with one user, missing the actual divide that's causing the disagreement (a needle).

Re: Auto-grading decade-old Hacker News discussions with hindsight

#237
post #235

> I realized that this task is actually a really good fit for LLMs I've found the opposite, since these models still fail pretty wildly at nuance. I think it's a conceptual "needle in the haystack sort of problem. A good test is to find some thread where there's a disagreement and have it try to analyze the discussion. It will usually strongly misrepresent what was being said, by each side, and strongly align with on…

As always, which model versions did you use in your test?

Re: Auto-grading decade-old Hacker News discussions with hindsight

#238
post #224

Earlier quoted context omitted.

How would you possibly grade comments if you change them?

Extract the concrete predictions, evaluate them as true/false/indeterminate, and grade the user on the number of true vs false?

This doesn't even seem to look at "predictions" if you dig into what it actually did. Looking at my own example (#210 on https://karpathy.ai/hncapsule/hall-of-fame.html with 4 comments), very little of what I said could be construed as "predictions" at all.

I got an A for commenting on DF saying that I had not personally seen save corruption and listing weird bugs. It's true that weird bugs have long been a defining feature of DF, but I didn't predict it would remain that way or say that save corruption would never be a big thing, just that I hadn't personally seen it.

Another A for a comment on Google wallet just pointing out that users are already bad at knowing what links to trust. Sure, that's still true (and probably will remain true until something fundamental changes), but it was at best half a prediction as it wasn't forward looking.

Then something on hospital airships from the 1930s. I pointed out that one could escape pollution, I never said I thought it would be a big thing. Airships haven't really ever been much of a thing, except in fiction. Maybe that could change someday, but I kinda doubt it.

Then lastly there was the design patent famously referred to as the "rounded corner" patent. It dings me for simplifying it to that label, despite my actual statements being that yes, there's more, but just minor details like that can be sufficient for infringement. But the LLM says I'm right about ties to the Samsung case and still oversimplifying it. Either way, none of this was really a prediction to begin with.

Re: Auto-grading decade-old Hacker News discussions with hindsight

#239

Earlier quoted context omitted.

I long had a similar idea for stocks. Analyze posts of people giving stock tips on WSB, Twitter, etc and rank by accuracy. I would be very surprised if this had not been done a thousand times by various trading firms and enterprising individuals. Of course in the above example of stocks there are clear predictions (HNWS will go up) and an oracle who resolves it (stock market). This seems to be a way harder problem fo…

> Analyze posts of people giving stock tips on WSB, Twitter, etc and rank by accuracy. Didn't somebody make an ETF once that went against the prediction of some famous CNBC stock picker, showing that it would have given you alpha in the past. > seems to be a way harder problem for generic free form comments. That's what prediction markets are for. People for whom truth and accuracy matters (often concentrated around…

You probably mean Inverse Cramer:

https://finbold.com/inverse-cramer-leaves-sp-nasdaq-and-dow-...

Re: Auto-grading decade-old Hacker News discussions with hindsight

#240
Many people are impressed by this, and I can see why. Still, this much isn't surprising: the Karpathy + LLM combo can deliver quickly. But there are downsides of blazing speed.

If you dig in, there are substantial flaws in the project's analysis and framing, such as the definition of a prediction, assessing comments, data quality overall, and more. Go spelunking through the comments here and notice people asking about methodology and checking the results.

Social science research isn't easy; it requires training, effort, and patience. I would be very happy if Karpathy added a Big Flashing Red Sign to this effect. It would raise awareness and focus community attention on what I think are the hardest and most important aspects of this kind of project: methodology, rigor, criticism, feedback, and correction.

Post reply on HN