Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Auto-grading decade-old Hacker News discussions with hindsight

karpathy.bearblog.dev

Post reply on HN
Print view View original

171–180 of 285 posts

Re: Auto-grading decade-old Hacker News discussions with hindsight

#171

Post by acyou »

godelski wrote: post #96

> I was reminded again of my tweets that said "Be good, future LLMs are watching". You can take that in many directions, but here I want to focus on the idea that future LLMs are watching. Everything we do today might be scrutinized in great detail in the future because doing so will be "free". A lot of the ways people behave currently I think make an implicit "security by obscurity" assumption. But if intelligence r…

I call this the "judgement day" scenario. I would be interested if there is some science fiction based on this premise.

If you believe in God of a certain kind, you don't think that being judged for your sins is unacceptable or even good or bad in itself, you consider it inevitable. We have already talked it over for 2000 years, people like the idea.

Re: Auto-grading decade-old Hacker News discussions with hindsight

#173

Post by gitremote »

p-e-w wrote: post #146
bryanrasmussen wrote: post #125

Earlier quoted context omitted.

>The former is the boring, linear prediction. right, because if there is one thing that history shows us again and again is that things that have a period of huge improvements never plateau but instead continue improving to infinity. Improvement to infinity, that is the sober and wise bet!

The prediction that a new technology that is being heavily researched plateaus after just 5 years of development is certainly a daring one. I can’t think of an example from history where that happened.

Neural network research and development existed since the 1980s at least, so at least 40 years. One of the bottlenecks before was not enough compute.

Re: Auto-grading decade-old Hacker News discussions with hindsight

#174

Post by xpe »

scosman wrote: post #60

Anyone have a branch that I can run to target my own comments? I'd love to see where I was right and where I was off base. Seems like a genuinely great way to learn about my own biases.

I appreciate your intent, but this tool needs a lot of work -- maybe an entire redesign -- before it would be suitable for the purpose you seek. See discussion at [1].

Besides, in my experience, only a tiny fraction of HN comments can be interpreted as falsifiable predictions.

Instead I would recommend learning about calibration [2] and ways to improve one's calibration, which will likely lead you into literature reviews of cognitive biases and what we can do about them. Also, jumping into some prediction markets (as long as they don't become too much of a distraction) is good practice.

[1]: https://news.ycombinator.com/item?id=46223959

[2]: https://www.lesswrong.com/w/calibration

Re: Auto-grading decade-old Hacker News discussions with hindsight

#176

Post by dang »

Terretta wrote: post #169
dang wrote: post #76

Earlier quoted context omitted.

I just remember them. Or forget them! The process is simply that moderation is super repetitive, so eventually certain pathways get engraved in one's memory. A lot of the time, though, I can't quite remember one of these patterns and I'm unable to dig up my past comments about it. That's annoying, in that particular way when your brain can feel something's there but is unable to retrieve it.

Well, you're #24 in this article's hall of fame, and the LLM thinks your moderation views stood the test of time. Perhaps it can already retrieve them for you.

There are so many interesting points and patterns that I've just lost track of over the years.

https://hn.algolia.com/?dateRange=all&page=0&prefix=true&que...

Re: Auto-grading decade-old Hacker News discussions with hindsight

#177

Post by DonHopkins »

I'd love to see an "Annie Hall" analysis of hn posts, for incidents where somebody says something about some piece of software or whatever, and the person who created it replies, like Marshall McLuhan stepping out from behind a sign in Annie Hall.

https://www.youtube.com/watch?v=vTSmbMm7MDg

Re: Auto-grading decade-old Hacker News discussions with hindsight

#178

Post by apparent »

> And then when you navigate over to the Hall of Fame, you can find the top commenters of Hacker News in December 2015, sorted by imdb-style score of their grade point average.

Now let's make a Chrome extension that subtly highlights these users' comments when browsing HN.

Re: Auto-grading decade-old Hacker News discussions with hindsight

#179

Post by popinman322 »

It doesn't look like the code anonymizes usernames when sending the thread for grading. This likely induces bias in the grades based on past/current prevailing opinions of certain users. It would be interesting to see the whole thing done again but this time randomly re-assigning usernames, to assess bias, and also with procedurally generated pseudonyms, to see whether the bias can be removed that way.

I'd expect de-biasing would deflate grades for well known users.

It might also be interesting to use a search-grounded model that provides citations for its grading claims. Gemini models have access to this via their API, for example.

Re: Auto-grading decade-old Hacker News discussions with hindsight

#180

Post by pixl97 »

bryanrasmussen wrote: post #125
onraglanroad wrote: post #93

Earlier quoted context omitted.

I'm giving them higher marks than the people who say it won't. LLMs have seen huge improvements over the last 3 years. Are you going to make the bet that they will continue to make similarly huge improvements, taking them well past human ability, or do you think they'll plateau? The former is the boring, linear prediction.

>The former is the boring, linear prediction. right, because if there is one thing that history shows us again and again is that things that have a period of huge improvements never plateau but instead continue improving to infinity. Improvement to infinity, that is the sober and wise bet!

Tiger: humans will never beat tigers because tigers are purpose built killing machines and they are just generalist --40,000BC
Post reply on HN