Live data from Hacker News

Auto-grading decade-old Hacker News discussions with hindsight

karpathy.bearblog.dev

251–260 of 285 posts

Re: Auto-grading decade-old Hacker News discussions with hindsight

#251
post #241
post #214

Earlier quoted context omitted.

Echo chamber of rational, thoughtful and truthful speakers is what I’m looking for in Internet forums.

That’s what everyone living in an echo chamber (and especially one of their own creation) thinks they’re in.

"you're in an echo chamber" is one of the most frightfully overused opinions.

Re: Auto-grading decade-old Hacker News discussions with hindsight

#252
post #70

Earlier quoted context omitted.

I am not a Redditor, but RES sounds like it would increase the ‘echo-chamber’ effect, rather than improving one’s understanding of contributors’ calibration.

Reddit's current structure very much produces an echo chamber with only one main prevailing view. If everyone used an extension like this I would expect it to increase overall diversity of opinion on the site, as things that conflict with the main echo chamber view could still thrive in their own communities rather than getting downvoted with the actual spam.

Hacker News structure is identical though. Topics invite different demographics and downvotes suppress unpopular opinions. The front page shows most up voted stories. It's the same system.

Re: Auto-grading decade-old Hacker News discussions with hindsight

#253
post #154
post #148

Earlier quoted context omitted.

I hope this is a joke. Forecasting and the meta-analysis of forecasters is fairly well studied. [1] is a good place to start. [1]: https://en.wikipedia.org/wiki/Superforecaster

> The conclusion was that superforecasters' ability to filter out "noise" played a more significant role in improving accuracy than bias reduction or the efficient extraction of information. >In February 2023, Superforecasters made better forecasts than readers of the Financial Times on eight out of nine questions that were resolved at the end of the year.[19] In July 2024, the Financial Times reported that Superfore…

> I'm really not sure what you want me to take from this article?

I linked to the Wikipedia page as a way of pointing to the book Superforecasters by Tetlock and Gardner. If forecasting interests you, I recommend using it as a jumping off point.

> Do you contend that everyone has the same competency at forecasting stock movements?

No, and I'm not sure why you are asking me this. Superforecasters does not make that claim.

> I'm really not sure what you want me to take from this article?

If you read the book and process and internalize its lessons properly, I predict you will view what you wrote above in a different different light:

> Gotta auto grade every HN comment for how good it is at predicting stock market movement then check what the "most frequently correct" user is saying about the next 6 months.

Namely, you would have many reasons to doubt such a project from the outset and would pursue other more fruitful directions.

Re: Auto-grading decade-old Hacker News discussions with hindsight

#254
post #251
post #241

Earlier quoted context omitted.

That’s what everyone living in an echo chamber (and especially one of their own creation) thinks they’re in.

"you're in an echo chamber" is one of the most frightfully overused opinions.

The expression is an echo chamber in and of itself; it is self-fulfilling prophecy.

Re: Auto-grading decade-old Hacker News discussions with hindsight

#255
post #200

This is a cool idea. I would install a Chrome extension that shows a score by every username on this site grading how well their expressed opinions match what subsequently happened in reality, or the accuracy of any specific predictions they've made. Some people's opinions are closer to reality than others and it's not always correlated with upvotes. An extension of this would be to grade people on the accuracy of th…

Didn't Slashdot have something like the second point with their meta-moderation, many many years ago?

Sorta.

IIRC, when comment moderation and scoring came to Slashdot, only a random (and changing) selection of users were able to moderate.

Meta-moderation came a bit later. It allowed people to review prior moderation actions and evaluate the worth of those actions.

Those users who made good moderations were more likely to become a mod again in the future than those who made bad moderations.

The meta-mods had no idea whose actions they were evaluating, and previous/potential mods had no idea what their score was. That anonymity helped keep it honest and harder to game.

Re: Auto-grading decade-old Hacker News discussions with hindsight

#256
post #237
post #235

> I realized that this task is actually a really good fit for LLMs I've found the opposite, since these models still fail pretty wildly at nuance. I think it's a conceptual "needle in the haystack sort of problem. A good test is to find some thread where there's a disagreement and have it try to analyze the discussion. It will usually strongly misrepresent what was being said, by each side, and strongly align with on…

As always, which model versions did you use in your test?

Claude Opus 4.5, Gemini 3 Pro, ChatGPT 5.1. Haven't tried ChatGPT 5.2.

It requires that the discussion has nuance, to see the failure. Gemini is, by far the, worst at this (which fits my suspicion that they heavily weighted reddit posts).

I don't think this is all that strange though. The human, on one side of the argument, is also missing the nuance, which is the source of the conflict. Is there a belief that AI has surpassed the average human, with conversational nuance!?

Re: Auto-grading decade-old Hacker News discussions with hindsight

#257
post #26

Earlier quoted context omitted.

It's true that meta is the crack of internet forums, so we, er, crack down on it quite a bit. That's a longstanding view: https://hn.algolia.com/?dateRange=all&page=0&prefix=false&qu... Alternate metaphor: evil catnip - https://hn.algolia.com/?dateRange=all&page=0&prefix=true&que... But yesterday's thread and this one are clearly exceptions—far above the median. https://news.ycombinator.com/item?id=46212180 was parti…

Dang, posting links to searches for your own comments is so meta, no matter the topic, but even more meta when about meta crack. I love how the first hit of meta crack is this, your own message about meta crack.

I'm higher than my supplier!

Re: Auto-grading decade-old Hacker News discussions with hindsight

#258
post #252

Earlier quoted context omitted.

Reddit's current structure very much produces an echo chamber with only one main prevailing view. If everyone used an extension like this I would expect it to increase overall diversity of opinion on the site, as things that conflict with the main echo chamber view could still thrive in their own communities rather than getting downvoted with the actual spam.

Hacker News structure is identical though. Topics invite different demographics and downvotes suppress unpopular opinions. The front page shows most up voted stories. It's the same system.

HN's moderation and ranking is better. But there's definitely an echo chamber effect here too.

Re: Auto-grading decade-old Hacker News discussions with hindsight

#259
post #168

Earlier quoted context omitted.

This is IPFS

In my experience from the couple of times I clicked an IPFS link years ago, it loaded for a long time and never actually loaded anything, failing the first "I wish we could serve static content" part. If you make it possible for people to donate bandwidth you might just discover no one wants to.

I think that many are able to toss a almost permanently online raspberry pi in their homes and that's probably enough for sustaining a decently good distributed CAS network that shares small text files.

The wanting to is in my mind harder. How do you convince people that having the network is valuable enough? It's easy to compare it with the web backed by few feuds that offer for the most part really good performance, availability and somewhat good discovery.

Post reply on HN