And this is exactly why I turned down a Ph.D. and always thought the community of academia (CS academia in my case) is one big pile of circle-jerk and shitty politics..
Potential organized fraud in ACM/IEEE computer architecture conferences
151–160 of 203 posts
Re: Potential organized fraud in ACM/IEEE computer architecture conferences
#152Earlier quoted context omitted.
Seems like Peer Review is hopelessly broken. Conflicts of interest galore. Science did fine without it until ~40 years ago, and it can do so again. I hope.
The solution is still within peer review, I feel. What ethical alternative is there? Required reproduction of the result could be a step forward, but only for cheap research.
arxiv and letting good stuff bubble up after-the-fact.
Peer review in my field is a random process. It's not corrupt; it's just that reviewers give papers a 30 second skim. I've had one paper really reviewed in my whole career. What gets in and out is just random. NIPS did a nice study on this many years back.
On the flip side, academic discourse gets delayed. People aim for Science/Nature, and then down the line. Journals / conferences aim for the prestige of high reject rates. Research goes out sometimes years too late.
Peer review gives the impression of credibility, without the reality. It should be gone. Tiers of journals should be gone too.
Re: Potential organized fraud in ACM/IEEE computer architecture conferences
#153"Conclusion: 99.99% of us are honest but the dishonest 0.01% can cause serious, repeated damage." Precisely how is this ratio determined? Given the unreproducable results and plagiarism that we've learned about in recent times I believe there is a lot more fraud than can be accounted for by "the dishonest 0.01%."
Maybe the absolutely worst are the 0.01% (or more likely, 1%). But the problem is more broad. There are different ways of corruption, making up word salads and fake results on one hand, or engaging in low effort repetitions with little to no questioning of the prevalent methodologies/theories of du jour, to fixing language in grant proposals/reports to get more money. When all these add up, I think 0.01% is likely th…
It's more than 0.01% honest/hard-working people. You have:
* Grad students who will never make tenure.
* Lifelong adjuncts / post-docs who are staying there because they won't play the game.
* Older faculty from a different era.
* Etc.
It's enough that at this point, I find all research from my alma mater suspect unless I've personally reviewed it or know the PI personally. But there's a pretty large honest portion too.
Re: Potential organized fraud in ACM/IEEE computer architecture conferences
#154Did anyone think this wasn't happening? Science operates as a priesthood more than anything. Universities orginally emerged out of schools for religious instruction, to train Catholic clergy, and I'm not sure how far we have come from there. There is the same two-faced rapprochement between the highest ideals and the lowest behaviours. The high priests just wear jeans instead.
This comment is not very helpful. Many scientists are doing their best to improve the world by publishing their research to the public. They are doing so, at least in CS, at a substantial pay cut compared to industry. You can see the efficacy of modern science by the real and impactful discoveries in machine learning and related fields made over the last 10-15 years.
It's like police. Good ones do good, and bad ones do harm.
You'd expect the good ones to be pro-reform, but there apparently isn't enough pro-reform in academia to have reform.
With that, less and less good will happen, and more and more harm.
Re: Potential organized fraud in ACM/IEEE computer architecture conferences
#155This is really sad for a great community like SIGARCH. They need to clean up; rather, they are sweeping things under the rug. Let's hope other communities can learn from their mistakes.
This is the tip of the iceberg for SIGARCH.
Re: Potential organized fraud in ACM/IEEE computer architecture conferences
#156Earlier quoted context omitted.
You can often guess who the reviewers are as well. The most obvious tell is "also cite paper X" which is their own 99% of the time.
It's even easier to trick the authors into believing you (the reviewer) is someone else. E.g: ask the authors to cite two papers of another researcher (as was mentioned somewhere in the thread as making it easy to guess who the reviewer is), ask to compare to results of that researcher more thoroughly, use some writing style typical of that researcher country of origin, etc. It's especially easy if you are familiar w…
Is a reviewer asking an author to cite their own paper not a massive conflict of interest in itself?
Re: Potential organized fraud in ACM/IEEE computer architecture conferences
#157Earlier quoted context omitted.
In small fields, it's sometimes really not very hard to guess who are the authors, just given the topic and what papers they cite. Even with double blind.
Not even in "small" fields. A lot of the time any new work is just a further development of previous work by the same author. So if you're well read in your field, that "double blind" peer review is not blind at all.
the cure to that is a field where independent people can also contribute to further development of existing work - through making sure that resources used and code are published along with the paper.
Of course, this is hard to do with cell cultures and such, and in the case of large databases or compute-intensive tasks it's not quite feasible. But a surprising amount of what goes on in CS and neighbouring disciplines can work that way once you work past the reticence of individual established authors and set it as a goal for your (sub-)field.
Re: Potential organized fraud in ACM/IEEE computer architecture conferences
#158Not directly related, but the "sign in to medium.com with Google" dialog on the top right corner with my account name on it is really freaking me out.
It's an iFrame hosted on Google servers, so medium does not directly have access to the iFrame contents. But it is still irritating.
It's the other way around that I'm concerned with; Google tracking which Medium posts I read.
Re: Potential organized fraud in ACM/IEEE computer architecture conferences
#159Earlier quoted context omitted.
This comment is not very helpful. Many scientists are doing their best to improve the world by publishing their research to the public. They are doing so, at least in CS, at a substantial pay cut compared to industry. You can see the efficacy of modern science by the real and impactful discoveries in machine learning and related fields made over the last 10-15 years.
It's like police. Good ones do good, and bad ones do harm. You'd expect the good ones to be pro-reform, but there apparently isn't enough pro-reform in academia to have reform. With that, less and less good will happen, and more and more harm.
There’s a lot of room for improvement:
- grad student stipends are negligently low, making it difficult for students to complete their PhDs.
- many journals in CS are not open access despite the authors and reviewers not being compensated for their time. I think pre-print servers and the ability to publish work on your own website has really improved the state of CS research over the past 5 years. In the past, the official policies of many top conferences stated that a paper would not appear in the proceedings if it had been published to your website or a pre-print server. This policy has now been dropped (or is not being enforced) by almost all top CS conferences.
- it can be difficult to resolve a conflict with your advisor especially if they have tenure. Tenured professors enjoy significant faculty protection (though maybe not to the same degree as police). Often the most productive avenue for a student in this case is to quietly switch advisors. This can lead to entrenchment of toxic faculty. At my alma mater I can think of a professor who was notorious for having none of their students graduate - they all either left academia or switched advisors
It’s certainly an imperfect system and I’m sure folks can point out other flaws in the comments. There is a lot of room for reform in these and other areas. For example the lack of transparency in this case makes the entire academic community look bad for no good reason.
However I do think that folks on HN (on average) have an overly negative view of academia, some of which may not be based on an accurate understanding of its good parts. For example peer review, despite its many flaws, turns out to be very valuable. During the COVID-19 pandemic, a lot of spurious science has been published in pre-print form and on Medium, only to then be rebutted and rejected by scientific venues with a robust peer-review process.
Re: Potential organized fraud in ACM/IEEE computer architecture conferences
#160Why don’t we require CS research to be reproducible? It’s a fricken computer program. (or should be!)
We also, in machine learning, never truly examine results for basic statistical significance. It’s often rather “I got this number higher”.
I struggle to have even basic trust of academic CS results, unless it’s backed by a major private company research (Google research) or academics I know and trust well.