Earlier quoted context omitted.
Curious, what revolutionary products would you cite, that existed within 4 years of the underlying technology? I feel like within that time frame, you're looking at "AOL chatrooms" for the internet, and I'm not even sure what the killer app was for smart phones - GPS was around before that. Computers do admittedly claim the Apollo program pretty early in, so I'll definitely concede that one - just curious if other re…
Visicalc was released 2 years after the Apple II was released. Visicalc changed the world for the better faster than AI has up to this point.
Evidence of inconsistencies in evaluation process and selection of winners
311–320 of 338 posts
Re: Evidence of inconsistencies in evaluation process and selection of winners
#312Hi all, I'm Nick, Product Manager for Kaggle Benchmarks and one of the co-organizers and judges for this AGI hackathon. First off, I want to set some context on the AGI hackathon. This was co-organized by Kaggle and Google DeepMind, and we had ~20 judges from both organizations. The hackathon concluded on Apr 16 and we had initially anticipated a judging period of 1.5 months (till May 31). However, we ended up extend…
- Grand claims backed by no evidence
- Core designs that make zero sense
- Pointless graphs that show nothing of interest
- Yet endless robustness checks on minor methodological assumptions (especially confidence intervals and t-tests)
For example, on the winning entry, not only is the graph completely wrong (as mentioned by the OP), but the interpretation would be nonsensical even if it was (implying bigger models "get more RL"?). And their own results even show the core dataset is worthless, because all their metrics are near-perfectly correlated. There's no way a serious human reader trying to evaluate "is this benchmark useful" would ever miss this.
I don't mean to pick on them - all the winning entries seem like there was no human effort put into them. And again, there's no way a human who actually attempted to read and understand these would ever think these are good by even the most minimal of standards.
Kaggle is legitimately a really awesome website, as someone who's competed before and won a few contests pre-LLMs. Stuff like this winning devalues the entire product and makes it look like a joke. If almost all entries look like this now, it'd be better to allow for the possibility of no winner.
Re: Evidence of inconsistencies in evaluation process and selection of winners
#313Earlier quoted context omitted.
In Dune we never learned what exactly went down in the butlerian jihad. Perhaps it was worse than idiocracy and the galaxy became monumentally stupid for an eon or two, rather than a bloodthirsty Terminator scenario.
https://en.wikipedia.org/wiki/Dune:_The_Butlerian_Jihad
Re: Evidence of inconsistencies in evaluation process and selection of winners
#314Re: Evidence of inconsistencies in evaluation process and selection of winners
#315Earlier quoted context omitted.
Those algorithms were only called machine learning and it's the opposite. Before the marketers got ahold of it we reserved AI for the actually intelligent, sentient, full strength vision of intelligent systems. We now talk about general AIs and A[Super]Is.
The term AGI was coined almost 30 years ago; is that when the marketers got ahold of it?
I did not claim we coined AGI in response to marketers.
Re: Evidence of inconsistencies in evaluation process and selection of winners
#316Earlier quoted context omitted.
Those algorithms were only called machine learning and it's the opposite. Before the marketers got ahold of it we reserved AI for the actually intelligent, sentient, full strength vision of intelligent systems. We now talk about general AIs and A[Super]Is.
That's not true. AI was widely used to refer to the decision systems and state machines that produced NPC behaviour in video games, and I'm sure many other things than just science fiction
As your respondents point out there's also been the pattern of clever algorithms being classified as "AI" until they were understood. That differentiated those selling snake oil from the serious.
Re: Evidence of inconsistencies in evaluation process and selection of winners
#317Earlier quoted context omitted.
To be the devil's advocate, engineers themselves can be hilariously incompetent too. Engineers have a tendency of assuming that budget is infinite and target audience is other engineers from same specialization. Open-source projects often have this problem where you can have dozens of thousands of man-hours poured into a project without a single end-user opinion taken into account. At some point my manager, who himse…
> Open-source projects often have this problem where you can have dozens of thousands of man-hours poured into a project without a single end-user opinion taken into account. How is this "a problem"? The reason there are dozens of thousands of hours on an open source project is because the end-users are working on it. Some projects exist solely for someone to work on it (that is, the "working on it" is the "use case"…
Re: Evidence of inconsistencies in evaluation process and selection of winners
#318Earlier quoted context omitted.
We appreciate your contributions here, but can you dial back this kind of sentiment in your comments? > Go build and never speak to a human again, if that's what you want. Reject the humanity you despise It's a really unfair stereotype to apply to those you're disagreeing with, and I see you do it repeatedly here; characterizing those you disagree with, or the HN community in general, with terms like anti-humanity an…
>It's a really unfair stereotype to apply to those you're disagreeing with, and I see you do it repeatedly here; characterizing those you disagree with, or the HN community in general, with terms like anti-humanity and anti-human. I searched my comments, because that doesn't seem like something I would do. It seems I've only used the term "anti-human" twice since I've been here, two years ago in reference to utopiani…
Go build and never speak to a human again, if that's what you want. Reject the humanity you despise.
– https://news.ycombinator.com/item?id=48947427
The culture here is so deeply self-sabotaging because it's so deeply afraid to be human. It really gets depressing sometimes.
– https://news.ycombinator.com/item?id=48167992
And yes many people in tech are on the spectrum. I suspect that much of what's dysfunctional with SV and tech culture is the result of raising a generation of socially awkward neurodivergent kids on 4chan and making them millionaires before their frontal lobes have even fully developed. That necessarily bleeds into HN and becomes normalized within the culture as performative cynicism, misanthropy and toxicity.
– https://news.ycombinator.com/item?id=47165924
The fact is the ethos of this forum has become antithetical to what hacker culture and tech have become, which is incurious, misanthropic and aggressively pro AI.
– https://news.ycombinator.com/item?id=47621066
I grin a bit whenever someone gets that infantile on what's supposed to be a board of high-minded intellectual sophisticates. But I'm serious. This forum is meant for human beings to satisfy their intellectual curiosity discussing topics of interest which are usually but don't need to be related to tech or programming. The humanity matters, even if it's damned hard to find sometimes because expressing humanity is a negative social signal here.
– https://news.ycombinator.com/item?id=46973059
And yet if someone so much as cracks a joke they get rapped across the knuckles and lectured about a rule that doesn't actually exist (no humor allowed)? Yes, that's negative. That's a culture of performative misanthropy.
– https://news.ycombinator.com/item?id=46459037
That's the pattern I mean. It was particularly the second one – containing the line “The culture here is so deeply self-sabotaging because it's so deeply afraid to be human” – that has stuck with me, because it was so jarring, and indeed so insulting.
We have countless stories where the primary topic, or much of the discussion, is something profoundly human and warm. People building things with or for their kids, partners or other family members. People telling their stories of overcoming or succumbing to severe illness, with comment threads full of profound expressions of compassion. Individuals being victims of bad fortune or corporate dysfunction, and the community rallying to their aid and in many cases, getting their problem solved. And plenty of cases where stories about wars and other atrocities are given front page time here and the community given extensive opportunity to discuss the topic, with Dan and I spending entire work days plus many hours in the evenings and overnights managing the threads, but with the overwhelming sentiment from HN commenters being in favour of civilians and in opposition to the military aggressors.
The Alt-Right and Dark Enlightenment types have never had any foothold here, and I remember vividly when they tried, just how swiftly and efficiently the HN immune system swarmed to keep them out. It's never been a place where hustle-culture or 996 workdays have any support, and stories about corporations demanding more from their employees are always met with resistance and derision in the comments. Corporate excess and abuse is always derided here. In a site that gets over 10,000 comments on most days I can think of fewer than five regular commenters who are staunch economic conservatives, and even then they hold their views because they sincerely believe they are better for humans.
When I read these characterizations you make about HN, I really just shake my head. The community you're describing is not one I recognize. Yes, sure, in a site that gets 300,000 comments per month, you will always be able to find something that fits the narrative you want to perpetuate. Normally, I just put it down to “I guess this is just what he needs to believe” and get on with my day. But your comments above to echelon crossed the line and meant I could no longer just let it slide.
The question of whether your comments are worse than his or not is moot. Your interaction is the perfect example of two people succumbing to the Negativity Bias or the Notice-Dislike Bias and attacking each other as avatars for the forces they so worry about. Neither of you is right and neither of you is better or worse than the other; it doesn't matter if one of you made a specific comment that happens to be a more egregious guidelines breach than the other; both of you are engaging in the same harmful behaviour and it's equally important for both of you to stop the behaviour and let go of these caricature models of HN, because you're both poisoning the place by perpetuating them.
HN is not ”supposed to be a board of high-minded intellectual sophisticates”; HN is a community of reasonably-educated people discussing the pros and cons of things. The deranged techno-utopian dark-enlightenment zealots may lurk in some corners of the tech world, but they’re not on HN or at YC. Anything promoting race science, genetic supremacy or even most mentions of IQ are flagged and killed here, and serial commenters promoting such ideologies are banned.
I understand you're distressed about the dysfunction and injustice in the world. I am too, as is Dan, and most of the people whose comments I read on HN. I can see signs that echelon is too, even though you – and I – may disagree with the way he expresses that.
The worst of this is that by trying to seem like you're the one who cares about humanity, you end up demonising and “othering” many of the people here, most of whom care about the same things you care about and broadly want the same things you want. Most people here have partners and kids and nieces and nephews and friends and communities they care about and want the best for. It's such a mean-spirited mentality to portray such a large community of people as “misanthropic” or “deeply afraid to be human”, when so much of what we see here is the very opposite of that.
So, sure, go and touch grass, if you need to. And of course you're welcome back. I meant it when I said we appreciate you and what you contribute. But no more of that vicious commentary about this community, please.
(And as a final aside: we don't rap people over the knuckles or lecture anyone about cracking a joke! I can hardly think of any instance at all when I or Dan has chided anyone just for an attempted joke. And we wouldn't, geez; when Dan and I are in a room together, much of what we talk about is old comedy sketches. The guidelines ask us to avoid internet tropes and we always discourage anything lame, and sadly, many attempts at humour, everywhere online, are lame tropes (or worse, meanness masquerading as humor). When a good joke lands on HN it’s a great day. I hope that’s not a surprise to anyone.)
Re: Evidence of inconsistencies in evaluation process and selection of winners
#319Earlier quoted context omitted.
IMO that’s awesome. I like when folks are clever. Just modify the rules next go around. It’s a contest judged by and LLM. Not sure why we would take it that serious.
Eventually LLMs will decide between life and death, heck, they are doing it already. People take the outputs seriously
Re: Evidence of inconsistencies in evaluation process and selection of winners
#320"I think you just need to accept the results of the competition. The winning submissions clearly provide value and had a lot of effort invested in them. I'm not really worried about a few inconsistencies or mistakes if the value is still there. Did you think another submission deserved to win over these?" That comment is gold. Yeah, I'm not worried about hallucinated slop, just accept it was the winner folks.
Sounds like that comment about economic value from earlier this week (yesterday?)