Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Auto-grading decade-old Hacker News discussions with hindsight

karpathy.bearblog.dev

Post reply on HN
Print view View original

151–160 of 285 posts

Re: Auto-grading decade-old Hacker News discussions with hindsight

#151

Post by potato3732842 »

modeless wrote: post #40

This is a cool idea. I would install a Chrome extension that shows a score by every username on this site grading how well their expressed opinions match what subsequently happened in reality, or the accuracy of any specific predictions they've made. Some people's opinions are closer to reality than others and it's not always correlated with upvotes. An extension of this would be to grade people on the accuracy of th…

>This is a cool idea. I would install a Chrome extension that shows a score by every username on this site grading how well their expressed opinions match what subsequently happened in reality, or the accuracy of any specific predictions they've made.

Why stop there?

If you can do that you can score them on all sorts of things. You could make a "this person has no moral convictions and says whatever makes the number go up" score. Or some other kind of score.

Stuff like this makes the community "smaller" in a way. Like back in the old days on forums and IRC you knew who the jerks were.

Re: Auto-grading decade-old Hacker News discussions with hindsight

#152

Post by medler »

hathawsh wrote: post #116
yunwal wrote: post #104

Earlier quoted context omitted.

What is an intellectual task? Once again, there's tons of stuff LLMs won't be trained on in the next 3 years. So it would be trivial to just find one of those things and say voila! LLMs aren't better than me at that. I'll make one prediction that I think will hold up. No LLM-based system will be able to take a generic ask like "hack the nytimes website and retrieve emails and password hashes of all user accounts" and…

I'll take a stab at this: LLMs currently seem to be rather good at details, but they seem to struggle greatly with the overall picture, in every subject. - If I want Claude Code to write some specific code, it often handles the task admirably, but if I'm not sure what should be written, consulting Claude takes a lot of time and doesn't yield much insight, where as 2 minutes with a human is 100x more valuable. - I ask…

> it agreed with me that its initial answer was wrong.

Most likely that was just its sycophancy programming taking over and telling you what you wanted to hear

Re: Auto-grading decade-old Hacker News discussions with hindsight

#154

Post by sigmar »

xpe wrote: post #148
sigmar wrote: post #129

Gotta auto grade every HN comment for how good it is at predicting stock market movement then check what the "most frequently correct" user is saying about the next 6 months.

I hope this is a joke. Forecasting and the meta-analysis of forecasters is fairly well studied. [1] is a good place to start. [1]: https://en.wikipedia.org/wiki/Superforecaster

> The conclusion was that superforecasters' ability to filter out "noise" played a more significant role in improving accuracy than bias reduction or the efficient extraction of information.

>In February 2023, Superforecasters made better forecasts than readers of the Financial Times on eight out of nine questions that were resolved at the end of the year.[19] In July 2024, the Financial Times reported that Superforecasters "have consistently outperformed financial markets in predicting the Fed's next move"

>In particular, a 2015 study found that key predictors of forecasting accuracy were "cognitive ability [IQ], political knowledge, and open-mindedness".[23] Superforecasters "were better at inductive reasoning, pattern detection, cognitive flexibility, and open-mindedness".

I'm really not sure what you want me to take from this article? Do you contend that everyone has the same competency at forecasting stock movements?

Re: Auto-grading decade-old Hacker News discussions with hindsight

#156

Post by losvedir »

ComputerGuru wrote: post #86

Looking at the results and the prompt, I would tweak the prompt to * ignore comments that do not speculate on something that was unknown or had not achieved consensus as of the date of yyyy-mm-dd * at the same time, exclude speculations for which there still isn’t a definitive answer or consensus today * ignore comments that speculate on minor details or are stating a preference/opinion on a subjective matter * it is…

Agreed. I feel like it's more just a collection of good comments. It doesn't surprise me to see tptacek, patio11, etc there. I think the "prediction" aspect is under weighted.

But it reminds me that I miss Manishearth's comments! What ever happened to him? I recall him being a big rust contributor. I'd think he'd be all over the place, with rust's adoption since then. I also liked tokenadult. interesting blast from the past.

Re: Auto-grading decade-old Hacker News discussions with hindsight

#157

Post by 0manrho »

Rperry2174 wrote: post #74

One thing this really highlights to me is how often the "boring" takes end up being the most accurate. The provocative, high-energy threads are usually the ones that age the worst. If an LLM were acting as a kind of historian revisiting today’s debates with future context, I’d bet it would see the same pattern again and again: the sober, incremental claims quietly hold up, while the hyperconfident ones collapse. Some…

It's because algorithmic feeds based on "user engagement" rewards antagonism. If your goal is to get eyes on content, being boring, predictable and nuanced is a sure way to get lost in the ever increasing noise.

Re: Auto-grading decade-old Hacker News discussions with hindsight

#160

Post by npunt »

One of the few use cases for LLMs that I have high hopes for and feel is still under appreciated is grading qualitative things. LLMs are the first tech (afaik) that can do top-down analysis of phenomena in a manner similar to humans, which means a lot of important human use cases that are judgement-oriented can become more standardized, faster, and more readily available.

For instance, one of the unfortunate aspects of social media that has become so unsustainable and destructive to modern society is how it exposes us to so many more people and hot takes than we have ability to adequately judge. We're overwhelmed. This has led to conversation being dominated by really shitty takes and really shitty people, who rarely if ever suffer reputational consequence.

If we build our mediums of discourse with more reputational awareness using approaches like this, we can better explore the frontier of sustainable positive-sum conversation at scale.

Implementation-wise, the key question is how do we grade the grader and ensure it is predictable and accurate?

Post reply on HN