Live data from Hacker News

Auto-grading decade-old Hacker News discussions with hindsight

karpathy.bearblog.dev

131–140 of 285 posts

Re: Auto-grading decade-old Hacker News discussions with hindsight

#131
post #90

Earlier quoted context omitted.

Perhaps a new category, 'highest risk guess but right the most often'. Those is the high impact predictions.

Prediction markets have pretty much obviated the need for these things. Rather than rely on "was that really a hot take?" you have a market system that rewards those with accurate hot takes. The massive fees and lock-up period discourage low-return bets.

Can’t wait for the brave new world of individuals “match fixing” outcomes on Polymarket.

Re: Auto-grading decade-old Hacker News discussions with hindsight

#132
post #84

Earlier quoted context omitted.

"Boring but right" generally means that this prediction is already priced in to our current understanding of the world though. Anyone can reliably predict "the sun will rise tomorrow", but I'm not giving them high marks for that.

I'm giving them higher marks than the people who say it won't. LLMs have seen huge improvements over the last 3 years. Are you going to make the bet that they will continue to make similarly huge improvements, taking them well past human ability, or do you think they'll plateau? The former is the boring, linear prediction.

LaunchHN: Announcing Twoday, our new YC backed startup coming out of stealth mode.

We’re launching a breakthrough platform that leverages frontier scale artificial intelligence to model, predict, and dynamically orchestrate solar luminance cycles, unlocking the world’s first synthetic second sunrise by Q2 2026. By combining physics informed multimodal models with real time atmospheric optimisation, we’re redefining what’s possible in climate scale AI and opening a new era of programmable daylight.

Re: Auto-grading decade-old Hacker News discussions with hindsight

#133
post #84

Earlier quoted context omitted.

"Boring but right" generally means that this prediction is already priced in to our current understanding of the world though. Anyone can reliably predict "the sun will rise tomorrow", but I'm not giving them high marks for that.

I'm giving them higher marks than the people who say it won't. LLMs have seen huge improvements over the last 3 years. Are you going to make the bet that they will continue to make similarly huge improvements, taking them well past human ability, or do you think they'll plateau? The former is the boring, linear prediction.

LLMs aren't getting better that fast. I think a linear prediction says they'd need quite a while to maybe get "well past human ability", and if you incorporate the increases in training difficulty the timescale stretches wide.

Re: Auto-grading decade-old Hacker News discussions with hindsight

#134

> I was reminded again of my tweets that said "Be good, future LLMs are watching". You can take that in many directions, but here I want to focus on the idea that future LLMs are watching. Everything we do today might be scrutinized in great detail in the future because doing so will be "free". A lot of the ways people behave currently I think make an implicit "security by obscurity" assumption. But if intelligence r…

Given the quality of the judgment I'm not worried, there is no value here. To properly execute this idea rather than to just toss it off without putting in the work to make it valuable is exactly what irritates me about a lot of AI work. You can be 900 times as productive at producing mental popcorn, but if there was value to be had here we're not getting it, just a whiff of it. Sure, fun project. But I don't feel pa…

I think you're missing the actual problem.

I'm not worried about this project but instead harvesting, analyzing all that data and deanonymizing people.

That's exactly what Karparthy is saying. He's not being shy about it. He said "behave because the future panopticon can look into the past". Which makes the panopticon effectively exist now.

  Be good, future LLMs are watching
  ...
  or humans using them might be
That's the problem. Not the accuracy of this toy project, but the idea of monitoring everyone and their entire history.

The idea that we have to behave as if we're being actively watched by the government is literally the setting of 1984 lol. The idea that we have to behave that way now because a future government will use the Panopticon to look into the past is absolutely unhinged. You don't even know what the rules of that world will be!

Did we forget how unhinged the NSA's "harvest now, decrypt later" strategy is? Did we forget those giant data centers that were all the news talked about for a few weeks?

That's not the future I want to create, is it the one you want?

To act as if that future is unavoidable is a failure of *us*

Re: Auto-grading decade-old Hacker News discussions with hindsight

#135
post #97

Earlier quoted context omitted.

I'm giving them higher marks than the people who say it won't. LLMs have seen huge improvements over the last 3 years. Are you going to make the bet that they will continue to make similarly huge improvements, taking them well past human ability, or do you think they'll plateau? The former is the boring, linear prediction.

> Are you going to make the bet that they will continue to make similarly huge improvements Sure yeah why not > taking them well past human ability, At what? They're already better than me at reciting historical facts. You'd need some actual prediction here for me to give you "prescience".

> At what? They're already better than me at reciting historical facts.

I wonder what happens if you ask deepseek about Tiananmen Square…

Edit: my “subtle” point was, we already know LLMs censor history. Trusting them to honestly recite historical facts is how history dies. “The victor writes history” has never been more true. Terrifying.

Re: Auto-grading decade-old Hacker News discussions with hindsight

#136

One thing this really highlights to me is how often the "boring" takes end up being the most accurate. The provocative, high-energy threads are usually the ones that age the worst. If an LLM were acting as a kind of historian revisiting today’s debates with future context, I’d bet it would see the same pattern again and again: the sober, incremental claims quietly hold up, while the hyperconfident ones collapse. Some…

Is this why depressed people often end up making the best predictions?

In personal situations there's clearly a self fulfilling prophecy going on, but when it comes to the external world, the predictions come out pretty accurate.

Re: Auto-grading decade-old Hacker News discussions with hindsight

#138
post #90

Earlier quoted context omitted.

Prediction markets have pretty much obviated the need for these things. Rather than rely on "was that really a hot take?" you have a market system that rewards those with accurate hot takes. The massive fees and lock-up period discourage low-return bets.

Can’t wait for the brave new world of individuals “match fixing” outcomes on Polymarket.

As opposed to the current world of brigading social media threads to make consensus look like it goes your way and then getting journalists scraping by on covering clickbait to cover your brigading as fact?

Re: Auto-grading decade-old Hacker News discussions with hindsight

#139
post #126

Earlier quoted context omitted.

To be clear, you are suggesting “huge improvements” in “every intellectual task”? This is unlikely for the trivial reason that some tasks are roughly saturated. Modest improvements in chess playing ability are likely. Huge improvements probably not. Even more so for arithmetic. We pretty much have that handled. But the more substantive issue is that intellectual tasks are not all interconnected. Getting significantly…

There’s plenty of room to grow for LLMs in terms of chess playing ability considering chess engines have them beat by around 1500 ELO

Sorry, I now realize this thread is about whether LLMs can improve on tasks and not whether AI can. Agreed there’s a lot of headroom for LLMs, less so for AI as a whole.

Re: Auto-grading decade-old Hacker News discussions with hindsight

#140

One thing this really highlights to me is how often the "boring" takes end up being the most accurate. The provocative, high-energy threads are usually the ones that age the worst. If an LLM were acting as a kind of historian revisiting today’s debates with future context, I’d bet it would see the same pattern again and again: the sober, incremental claims quietly hold up, while the hyperconfident ones collapse. Some…

> One thing this really highlights to me is how often the "boring" takes end up being the most accurate.

Would the commenter above mind sharing the method behind of their generalization? Many people would spot check maybe five items -- which is enough for our brains to start to guess at potential patterns -- and stop there.

On HN, when I see a generalization, one of my mental checklist items is to ask "what is this generalization based on?" and "If I were to look at the problem with fresh eyes, what would I conclude?".

Post reply on HN