Live data from Hacker News

Please commit more blatant academic fraud

jacobbuckman.com

91–100 of 365 posts

Re: Please commit more blatant academic fraud

#91

Earlier quoted context omitted.

Well, I wonder if this could force people to start publishing their code so others can replicate the results. This would help greatly with the kinds of issues that are being discussed: - It would be much harder to hide makeup results and cherry-picked seeds - Useless research would not be as easy to hide - The real-world impact of the research would be more valued. - Researchers would be forced to recheck their code…

I'm active in the security and privacy field, and some conferences have started "artifact reviews" [1][2], where authors can optionally submit the data and code that go along with their work. It's definitely an improvement, but the devil is in the details. Should these be mandatory? If they're mandatory, then what quality standards should artifacts meet? Who's ensuring those standards are followed? How should proprie…

> where authors can optionally submit the data and code that go along with their work.

So cool. I wish there were more steps in this direction.

> Should these be mandatory?

As an outsider, I would say no. However that should have a detrimental effect on the ability to get published if you are from an academic institution. I can totally see a gray area when it comes to non-academic publishers.

> There's also the question of what code should be made public

Shouldn't it be as much as possible? The way I see it is that the easier to replicate a paper, the more people can focus on its true utility. That is, by abstracting ourselves away over the details of the paper, what does this really contribute? From my point of view, publishing as much as possible maximizes the real-world impact of the research.

Re: Please commit more blatant academic fraud

#92

I ran into these issues in an adjacent field a generation ago. Many candidly spoke to me about having to be a ‘team player’. This is all due to the criteria for survival and graduation. Let’s be honest, not every idea will work and this does not reflect badly on the researcher who came up with it. But the only way to graduate and put food on the table is to keep getting funding through novel papers that have great re…

> this does not reflect badly on the researcher who came up with it. > But the only way to graduate and put food on the table is to keep getting funding through novel papers that have great results. So in the way in which it matters, it does reflect badly on the researcher who came up with it.

Science involves a lot of risks but we don’t reward negative results, lessons learned from methodology problems, or failed reproductions except in rare studies. These issues come up all the time and many resort to tactics in the article to keep their funding / graduation plan alive.

Re: Please commit more blatant academic fraud

#93
post #59

The idea that a field can be reformed by making it worse until it suddenly faces a reckoning and emerges much better is ... I don't know where people keep getting the idea that this might work. It has never worked in any field ever in history. The thing that can happen: fields gradually split into rigorous and non-rigorous camps. Like with evidence-based medicine, or chemistry/alchemy. Depending on the field, either…

Well, I wonder if this could force people to start publishing their code so others can replicate the results. This would help greatly with the kinds of issues that are being discussed: - It would be much harder to hide makeup results and cherry-picked seeds - Useless research would not be as easy to hide - The real-world impact of the research would be more valued. - Researchers would be forced to recheck their code…

These discussions remind me of how there's a years long delay before these ideas first reach boards like HN and then popular magazines and the taxi driver.

Yes, code releasing is a big talking point and most major groups release code. It doesn't solve the subtler problems,like claiming 1% improvements as a solid contribution, unequal hyperparameter tuning, cherry picking datasets, overclaiming the novelty, salami publishing, lacking ablations, not comparing against better methods etc etc. Releasing code is a step but not a panacea. At least it partially protects against fully made up numbers which is not nothing, but it's just scratching the surface. In many cases the code is not so easy to use and the original authors may not be willing to help you with reproducing it, ignoring emails and GitHub issues. If you can't reproduce it, they can always claim something. In one case I was told they rewrote the codebase in a new framework since they generated the results in the paper and there may be small discrepancies due to that. Apparently that was enough change that they were no longer the best in the benchmark comparison. But nobody cares at this point, the paper was published, they are on to their next project. People also have the attitude that it doesn't matter anymore, the field has moved on already, a retraction doesn't change much just hurts the author and the group. The vast majority of papers isn't used anywhere actually in practice, the field is moving so fast that any method can only remain on top for a couple of months.

Also, I don't think AI really is worse at this. It's rather that we are more open about it because we have less to lose and professors don't have an iron grip, due to the existence of industry. In some of academia your whole future depends on whose ass you kiss, it's better in CS. It's a bit like sexism accusations directed at nerds, as if they particularly bad.

The molochian cancer is everywhere. The less you see it may just indicate its more effectively hidden.

Re: Please commit more blatant academic fraud

#94
post #59

The idea that a field can be reformed by making it worse until it suddenly faces a reckoning and emerges much better is ... I don't know where people keep getting the idea that this might work. It has never worked in any field ever in history. The thing that can happen: fields gradually split into rigorous and non-rigorous camps. Like with evidence-based medicine, or chemistry/alchemy. Depending on the field, either…

Well, I wonder if this could force people to start publishing their code so others can replicate the results. This would help greatly with the kinds of issues that are being discussed: - It would be much harder to hide makeup results and cherry-picked seeds - Useless research would not be as easy to hide - The real-world impact of the research would be more valued. - Researchers would be forced to recheck their code…

Publishing the code may not be enough. Part of the problem is that replication of a study isn’t usually rewarded in the academic sphere because of the obsession with novelty within publications. Most researchers want to spend their finite time on something they find interesting and also may help their career. One way may be to create a movement to foster good replication studies that can be published and rewarded.

Re: Please commit more blatant academic fraud

#95

I think we need to rethink the core concept of Universities having publish or perish as a part of being a Professor. The reason why fraud exists is because of the incentive structure and due to the nature of research you have intelligent people that will game the system. So, we have to figure out how to change the incentive structure so that professors don't attempt to perform Academic Fraud or we h ave to figure out…

Keeping test data private is good practice actually. It keeps from hyperparameter optimization on the test set. Yes theoretically the dataset authors might still do it, but it's still better for the field as a whole to keep some test data secret and only use it for evaluation.

Registering before getting data is common, so they can make you agree to some terms. It's not suspicious in itself.

There are no hard and fast rules of thumb. You can look at the group, the people involved, the country, the writing style, the quality and thoroughness of ablation, is there code available etc. But nothing is 100%.

Re: Please commit more blatant academic fraud

#96
post #11

Wow, the amount of depravity in some academic circles is astonishing. The article links [1] the case of a PhD student at the University of Florida who was forced to participate in such a publication collusion ring and was pressured to commit scientific fraud by fabricating results and submitting them to a conference [2], being threatened with physical harm should he decide to go public. This student saw no way out an…

Could someone please screenshot the Instagram post and re-upload it somewhere else for those of us who don't have an account. Thanks.

https://imgur.com/a/tJfeQjg

Re: Please commit more blatant academic fraud

#97
One big problem in our society is that there is a Darwinian selection process occurring which selects for psychopathy. It's only going to get worse.

Psycopaths are attracted to power and are willing to do anything for it so they are more likely to get it. Power comes with money. Money allow psychopaths to have more children. The children are more likely to be psychopaths too so they will also end up in positions of power with more money. As automation increases, the system can afford to support an increasing number of psychopaths and they quickly take up all available positions of power via ruthless means. Psychopaths end up occupying all the important political and judicial positions. Altruistic people who follow the rules can't afford to compete in the market place, they have fewer children, they are discriminated against by psychopaths who recognize that they are different from themselves. Altruists become extinct as they are slowly replaced by machines which make altruism a redundant character trait; not necessary for a 'functioning' society. Now that we have machines to automate everything, the economy will no longer rely on 'exploitees' to do the hard value-adding work.

Then a whole new class of ultra-psychopaths (e.g. serial killers) will become more common and proceed to wipe out the regular psycopaths through violent means. Humans will get progressively worse and end up driving themselves to extinction. There's not going to be any robot uprising; humans will make themselves extinct before that happens.

Re: Please commit more blatant academic fraud

#98

Earlier quoted context omitted.

They are part of the power structure, they're just 'protected'. Most publications don't want to be seen as promoting a narrative that goes 'against science' even though of course it's not. There's no room for nuance in populism.

Protected, or just less noticed.

It's more the fact that academia is granted a greater degree of leniency because it is fundamentally something that is societally required for innovations to take place. Genuine ingenuity requires room to flourish, so in the West at least, the theory is you leave tge Boffins alone to do their wizardry.

The part I don't understand is how extorting the students to fund bloated Administrative processes, sport teams, stadiums, and executive staff ever became a thing, nevermind the incestuous relationship with academic publishing. Almost all great work I've seen came not out of gnashing of teeth and publish or perish, but out of a labor of love or an odd obsession with truly understanding something until you could practically get it across to a 5th grader.

"I can explain how, don't ask me why too many times, still figuring that part out."

Re: Please commit more blatant academic fraud

#99
post #79

Earlier quoted context omitted.

At a lunch for female journalists and scientists, Hunt gave a speech... "It's strange that such a chauvinist monster like me has been asked to speak to women scientists. Let me tell you about my trouble with girls. Three things happen when they are in the lab: you fall in love with them, they fall in love with you, and when you criticise them they cry. Perhaps we should make separate labs for boys and girls? Now, ser…

I’ve noticed a pattern where, wherever someone feels the need to write (after a disparaging story about someone else) “ They were not joking ” or “ They really said this ”, it’s almost always false, and the disparaged person really was joking, or really did not say that, or it was taken wildly out of context, etc.

What's the missing context here? That he used an important public speaking opportunity to actually mock the women he was assigned to help, and we misunderstood it as a serious appeal? That he was being fake offensive because it would be funny or enlightening? This wasn't an opportunistic pun. Humor is rooted in one's worldview.

Re: Please commit more blatant academic fraud

#100
I'm really happy to see this article here. I quit my PhD where I was working on AI partly because of the "mundane day-to-day fraud" that the author and I observed in the field. Once you've read enough papers, you can easily see a kind of recipe to an academic paper. Present a new method, compare against a baseline, discuss the differences in the results found. To me, some researchers just follow this recipe and do not think about the steps critically. You see figures that plot two categories of points that are purportedly following different distributions but the error bars (or even more of the distributions) overlap. But a p-value says they are significant. So that's good enough to publish! I felt a lot of anxiety about sharing my research in this environment and felt pressure to make my work seem like it was the best solution for every problem when in reality there are more nuances.
Post reply on HN