Stefan Savage put it best, I think: a paper accepted in a journal is part of a conversation that science is having; it's the start of a debate, not the conclusion. What's important about a paper is whether the ideas in it are validated and get built on by other scientists. Peer review is just a sanity check before that process starts, nothing more.
Well, yes. But what's worrying is that so many people who have seen a university from inside and even have advanced degrees do not know this. Every time some one comes up with grants for reproducing published stuff the silly idea finds enthusiastic support. How come? People should know about the practice of science but don't.
I think us humans are flawed enough where we need a reminder of timescale, e.g. how time tested is a theory, how long did a different belief exist before a new understanding arose?
Einstein for example had public resistance to his theory of relativity in the beginning.
> Einstein for example had public resistance to his theory of relativity in the beginning.
More recently, it took years for the medical community to get over its opposition to the evidence that common peptic ulcers are caused by bacteria (which earned two Aussie physicians the 2005 Nobel Prize in Medicine). [0] A NY Times correspondent (and physician) wrote in 2002, "I’ve never seen the medical community more defensive or more critical of a story."
(Much of the opposition was interest-based, e.g., surgeons who didn't want to lose lucrative fees for operating on ulcer patients who could instead be treated with inexpensive antibiotics without the pain and recovery time of major surgery.)
A funny story from academia is about the cohort of peer reviewers whose primary standard for acceptance is that their own work is cited in the bibliography of the submitted paper.
Peer review often fails to catch cheaters, too, as the infamous Jan Hendrik Schön case demonstrated (from 2000-2001 the Bell Labs researcher published 9 papers in Science and 7 in Nature, all of which made it through peer review, all of which were later retracted on grounds of fraudulent data manipulation). In the long run, the scandal did improve the field, as all publications on microelectronic graphite etc. devices now require electron microscopy proof that the claimed devices actually exist. Note current AI technology allows for data fraud and image manipulation that's much harder to detect than in the past, though.
In a lot of fields, more eyes aren't going to create fewer walls, because (a) the important venues are selecting K winners out of N submissions, and that K is the bottleneck, and (b) in most fields, any real (and some unreal) results will get published somewhere.
> A funny story from academia is about the cohort of peer reviewers whose primary standard for acceptance is that their own work is cited in the bibliography of the submitted paper.
In some circles, the two first pieces of feedback on a paper draft, from a nominal co-author:
1. Cite [my people's various loosely related work].
2. Cite [particular researcher in this niche], who'll probably be a reviewer.
If they are nothing more than the the starting conversation of scientists then we should not put much value in them.
Let’s not play the have it both ways game where scientific papers are given deference and then when there is cheating just say li g that scientific papers are just a conversation starter.
If not using papers as a basis for policy then what? Making a committee of academics from a selection of prestigious institutions every time you have a question that needs answering?
> * Then devalue published papers. Don’t use them as the basis for rewards.*
What about the use of papers as metrics for advancing in academic careers?
Career advancement could be based on market forces (e.g., productive researcher on something valued, if they're not given a sweet enough deal by one university, another university can snap them up). But how do buyers in the market get the information about how valuable a given researcher is? (Word of mouth? CV summary without pubs list? Job talks? Interviews? Twitter?)
We should start framing
some journals as "here's what the authors have found convincing from bleeding edge submissions, please take a look and try to confirm these findings" for other researchers and others as "here's a conservative and comprehensive
list of results which have been solidly reproduced" for professionals in the field who want dependable stuff
> here's a conservative and comprehensive list of results which have been solidly reproduced
This sounds more like textbooks.
Journals are usually intended to be venues for experts to talk to each other, not even "professionals in the field".
There are some journals (sometimes called "translational") that try to bring scientific results into practice, though these are often limited to fields where "practice" is its own huge body of knowledge that doesn't always overlap with "research" in that field (e.g., Medicine).
> Journals are usually intended to be venues for experts to talk to each other, not even "professionals in the field".
This made sense when communication was slow and the back and forth of discourse could take weeks to months. But publishing your work in a way that thousands of other readers can consume it is easier now that it's been ever, so it stands to reason that sharing the results of scientific works may yield better outcomes if it takes on a different form.
There's a whole Internet to post important results on without waiting for peer review. If you're an established practitioner in your field making claims that are straightforward to follow, other practitioners will probably read you.
People keep saying peer review is just a sanity check but no one seems to really grasp it. I get it because internet discourse has overblown the validity of peer reviewed articles to an insane degree.
Peer review is just a sanity check. Authors and scientists and studies and papers are important or not independent of the peer review process.
Dr. Christopher Essex in a recent interview [https://www.youtube.com/watch?v=jpjpBWxvamA] highlighted that it's called "peer review" and not "expert review" - which I think is an often most important detail that should not put anyone on a pedestal; of course the "peer review" system has been hijacked-corrupted to some degree, where there are gatekeepers to getting published in the "most reputable" journals.
I took one journalism class, taught by a crotchety old man, who seemed to be of the opinion that they were all liars and cheats pretty much as far back as you can go. Hearst just weaponized a rot that was always there.
Peer reviews should be done in public and considered an ongoing process rather than a one-time thing. If an expert thinks some questions are worth asking, then the resulting discussion should be available for younger generations to learn from it as well. I think the open source ecosystem has shown the effectiveness of such a system. As more data is made public it becomes easier and more effective to sniff out bad actors as well.
We could also take steps towards helping establish high reputation for certain papers by introducing a mechanism other than citation count. Maybe some kind of stamp of approval.
Yes, this is more or less how the non-profit journal https://elifesciences.org (funded by HHMI, Wellcome, Max Planck, and Wallenberg Foundation) works.
A problematic aspect of non-public peer reviews is waste and political battles. I have witnessed first-hand how big names in a particular field reject articles from incumbents that are perfectly sound just to delay their publication and/or to copy them.
A public review introduces some skin in the game and avoids this kind of behavior, as well as rejections or requests to make changes because of reviewer incompetence. It also avoids the opposite thing, blind acceptance of flawed studies.
> Yes, this is more or less how the non-profit journal https://elifesciences.org (funded by HHMI, Wellcome, Max Planck, and Wallenberg Foundation) works.
Also, all 19 journals published by EGU (European Geophysical Union)
I'd take it even further: make peer review public and open to all members of that community. Imagine a forum-like discussion anybody can anonymously review any submission and all reviews are public.
Becoming a reviewer should still be invite-only and the system should keep track of the reviewer's identity behind the scenes to monitor for abuse, of course. The review can include coarse-grained reputation signals like "has reviewed 100+ papers in the last 5 years".
It might be worth embargoing reviews with a fixed time delay before making them public to prevent bandwagon effects and disincentivise review plagiarism tho. The reviewer's identity should be deanonymised too after something along the same time scale as when the paper author's identity is revealed.
Consensus review is on the way out. I don't know what replaces it, perhaps a Github-like system? Whatever it is, hopefully it includes a focus on replication.
There are two kinds of peer review in academic science - at the point of publication, and the point of funding. While quantities of ink have been spilled on the former, the latter gets far less attention - though as you might guess, this is a much more contentious issue because millions of dollars of funding may be on the line. A good discussion is here:
"Is there hard evidence that the grant peer review system performs significantly better than random?"
In short, 'freedom of research direction', aka 'blue skies research'[1] is steadily becoming a thing of the past, as grant managers and politicians and academic administration teams increasingly take the view that they're the ones who should be directing what kinds of research are done, rather than the academic researchers themselves. This is enforced by a grant system that narrowly defines how the funds can be spent, meaning that your average academic researcher has been transformed into a corporate drone following orders from the executive floor.
This transformation of academic research began around the same time Bayh-Dole legislation granted exclusive licensing of university patents developed with taxpayer funds to private interests (1980).
Based on personal experience, I have a very different opinion than Daniel.
I do not think peer review adds much value over self-publishing. Consumers still need to read and verify the work themselves. Bad stuff gets published in peer review so often you still need to verify the integrity of everything yourself. For a simple hypothetical, say peer review is "good quality" 80% of the time, and self-published is good quality 50% of the time. These are made up numbers, but I am saying "80% is too low for the pain of peer review to provide much value". As a consumer I find 0 value in peer review (I can use google and read what I want, being published in peer review is an annoying paywall if anything).
I believe the majority of comments in peer review are not based on technical accuracy (what an outsider may think peer review is about, verifying if something is right or wrong), but tend to be more clearly opinions. So from a writer standpoint for people who say "peer review improves my work", that does not jive with my experience.
There are so many other negatives with peer review in academia (people bean counting pubs, paywalled, the club issue Daniel mentions), I just don't think it adds much of any value. If everyone decided tomorrow "I am just going to publish stuff on ArXiv" (or whatever preprint server), the world would not be worse off. I think we would be better off actually.
Right now in my field (cryptography and security) the number of papers is exploding. The real battle at this point is not only finding time to read the papers, it's even finding time to learn about them. Unfortunately self-publishing (AKA preprints) just means we're producing a lot of papers nobody has time to read, many of them with obvious flaws and bad presentation that could easily be fixed. Peer-review is one rating system that helps to filter some signal from the noise.
Good luck keeping up with and filtering through that deluge of mostly shit research. The ArXiV feeds of even semi-popular fields like computer security is mostly full of garbage that you would never see in a good conference.
What I disagree with in the article is the _I never make mistakes_ attitude; it could be worse, but I still think it's good to discuss. The author writes that because they are "serious", they can essentially rebut all criticism "easily". We are all human and even excellent scientists make honest mistakes. Strong theorists can sometimes make "hard" math mistakes. In the best cases, peer review gives us some assurance that we at least did not make obvious mistakes that can be relatively easily spotted by other specialized researchers.
I think the _never make mistakes_ attitude is dangerous, because it means that researchers need to be very cautious when admitting honest mistakes and their own intellectual fallibility in order to not lose face.
The statements made in the original post are so foreign to me. It sounds like the author is digging really deep to try and say something nice about the peer review process. To be fair, it might be true for CS where they are stuck in the weird trap of publishing in conferences, but in other circles everything goes into journals, and in this case the peer-review processes is definitely not there to "help the authors".
The peer review is there to help the journal maintain its reputation by preventing the publication of sub-standard stuff. Period. Sub-standard can mean uninteresting, incomplete, poorly written, or whatever the journal is aiming for. It is not there to safe guard the integrity of the literature against erroneous results...it's purely self-interest on the part of the journal.
In reality, a rejected paper will just be submitted elsewhere until it is eventually accepted. The authors cannot afford to spend 1-2 years worth of work on a project then have nothing to show for it, just because a reviewer didn't "get it". So authors will keep submitting it (hopefully with some improvements based on past reviewer comments, but maybe not) until it "gets through" somewhere, and eventually nearly everything gets the seal of "peer review".
> There is SO much more I could write on this subject, but I'm trying to stay on point (-:
Isn't this almost identical to the general admissions process, too?
It seems strange to me that we built institutions on the idea of filtering in/out applicants based on relatively arbitrary criteria, and then express shock/surprise when the reward systems inside that institution are.. basically the same?
There are parallels everywhere, e.g. scientists feeling they must get positive 'groundbreaking discovery!' news reporting about their publications, not just actually doing impactful work, in the same way good grades aren't enough and you need some other impactful story to tell in order to be accepted to many schools.
All of it can be traced back to money, money, money.
> It seems strange to me that we built institutions on the idea of filtering in/out applicants based on relatively arbitrary criteria, and then express shock/surprise when the reward systems inside that institution are.. basically the same?
I think it is impossible to use anything other than noisy signals in many of these processes. I'm not bothered by the noise. But what I am bothered by is pretending the noise doesn't exist or trying to convince people it is a feature and not a bug. Why not just admit the process is noisy and that we're just doing our best? These processes may still be frustrating, but they will be less so and it leaves the door open to fixing issues if they are actually fixable.
With the way we publish science, I find it very odd. We are doing essentially the same thing we have done for centuries with only a few minor tweaks. But unlike the past we don't have many of the same constraints; such as, printing, lack of color, distribution/storage, communication (which is arguably the main purpose of papers), video, and so on. Why are we stuck in the 18th century with a sprinkling of 21st century (plots/graphics and arxiv)? We can track data, reproduction efforts, failures, challenges, and all that stuff. Is not the point of science the pursuit of knowledge? If so, it seems like we're being pretty inefficient. I think there's just something funny about how we publish papers on things like computer graphics and video but constrain our communication methods to a piece of paper.
True for a particular dominant but antiquated and rapidly aging out model of peer review. Peer review as practiced at PeerJ, eLife, F1000, etc. is collaborative, productive, and maintains integrity in a visible way.
Peer review is not inherently terrible. Exploitative rent seeking publishers that commoditize academic careers and outputs, and hold knowledge to ransom, are the problem.
Please, everyone, stop publishing in journals that do harm. Think about what the impact of where you publish a paper is and align your choices with your values.
Right, it's a low bar, and it's meant to be a low bar, and that's fine.
It's not no bar. Peer review adds some credibility to a paper. And the venue does as well. It's just less credibility than the popular imagination assumes.
This had a refreshing degree of realism. Naturally, any perfection-minded person will be offended by that.
A decent journal will stop using reviewers who do a lousy job, and complaints by authors about their paper's reviewers ought to be listened to and not dismissed as sour grapes.
> A decent journal will stop using reviewers who do a lousy job, and complaints by authors about their paper's reviewers ought to be listened to and not dismissed as sour grapes.
As a data point in ML conferences, I had a reviewer give me a strong reject with only a few sentences. First was a generic "not novel" with no further explanation. The rest was a complaint that my paper had serious flaws and pointed to an intentionally redacted link and a broken cross-reference. We reported them to the AC and the other two reviewers were borderline and weak accept, but both had low confidence (of course the inane reviewer was high confidence). The result was that the reviewer's rebuttal was much more angry and the weak accept said "authors addressed my concerns, but I've decided to lower my score."
I tell this story because at a certain point it is not a problem with bad reviewers, it is a problem with bad management. The most important thing that a system can have is self correcting behavior. A system that gives a bad faith actor a slap on the wrist and allows them to then escalate their bad behavior and influence others is not a well functioning system. Granted, this is just one example, but I've seen many others (such as a theory paper being rejected for lack of experiments). I've started collecting author tweets I find about these kinds of situations, but I think there's a lot more that aren't easily conveyed on social media.
I do really want to promote discussion of these things because I think we should always be improving our systems. You're right that nothing is every perfect, but we can still recognize flaws without being consumed with the impossible goal of perfectionism.
was shitcanned by a Microsoft reviewer (if you're the one and you're reading this: GFY)
First of all, he ignored it for six months, and it wasn't until I begged the editor that he got it moving. Then he gave one round of critique, and when I addressed it, he said, "the patent law changed" and closed the process. He knew nothing about patents.
As I said, the editors should not defer to these douchebags. Fortunately I don't need pubs for my career, but some people do.
Funny enough, none of my first author works have been "published" but since they are on arxiv I have a competitive citation count and h-index (most cites are from my first author works). Only group upset at me is my grad school. It's extremely frustrating because I have all indications of doing good work and clearly the community agrees. I think the funniest part of it all is that I've offered to show them my reviews and so we can discuss if I am doing something wrong or "just having bad luck." But no one wants to even entertain the idea that the system is dysfunctional.
I will never understand why in CS we use a conference system as our indicator of merit. It is an antagonistic zero sum game. There is no recourse for giving a bad faith review and you are incentivized to reject works. It's easier to reject and thus less time consuming -- every paper has flaws and limitations, just point at them and ignore context--, you marginally increase the chance of your own paper getting in, and venues use acceptance rates as the main measure to indicate their level of prestige (if I cared enough I'd just get LLMs to spam papers to abuse this). Every part of the system that does something good and beneficial to the scientific community relies entirely on people acting in good faith and against other incentives they have, favoring the purity of science. That clearly doesn't scale and clearly sets a stage for bad actors to overwhelm. I think we only continue this because they've successfully convinced good faith actors that there is no other way and that it isn't as bad as it is. I just wonder how much money and time is lost due to all of this (before we even account for the ridiculousness of charging for what arxiv does for free. The paper is given to them for free as well as the reviewing is done for free, and much of the organizing committee and all that is also free labor. Something this important shouldn't have so many similarities to a scam that seeks to extract money from the government and universities).
I am also in a conference-centric CS subfield, but I have published quite a lot in journals as well (both because of multidisciplinary collaborations and because my country has coarse-grained metric-based evaluation systems were conferences don't count much, so at least a part of my work has to go to journals to appease the system).
In my experience, journal reviewing is much worse, and especially much more corrupt, than conference reviewing.
At conferences you get bad actors with almost no oversight or accountability, true. But at least they typically don't have an axe to grind, because accept/reject decisions tend to be binary (there is no "revise and resubmit", at most optional recommendations for the final version that aren't effectively enforced).
At journals, the "revise and resubmit" option gives reviewers and editors a lot of leverage to push papers their way, and I have very often seen reviewers tacitly and subtly hint that the paper would easily be accepted if the authors included three or four references to papers by Author X (which are irrelevant to the subject matter, but of course, Author X will be the reviewer or a friend). Sometimes it's clear that the reviewers didn't read the paper, their sole interest on it is that you include their citations, that's the reason why they accepted to review and all they look at. Editors could in theory prevent this, but often they just echo reviewer comments, and sometimes they are complicit or even the originators of the corruption (it's also typical that the editor asks to cite a bunch of papers from the same journal, even if they're not related, because that inflates impact factor). In any of these cases, since the author is in a position of weakness and there's always the career of some PhD student or other down the line, one doesn't typically dare to complain unless the case is really blatant (which is typically not, because they know how to drive the point subtly). It's easier to go along with it, add the citations (knowing that they will lead to acceptance) and move on.
This has happened to me very often and I'm not talking about shady special issues in journals that print papers like sausages, I'm talking about the typical well-regarded journals from major traditional publishers. In conferences I've gotten all sorts of unfair rejections, of course, but at least the papers I've published accurately reflect my views, I can stand behind them. In journals, maybe half of my papers have this material that doesn't make sense and was added to appease a corrupt reviewer.
I find that many CS authors who haven't had to publish in journals have a "grass is always greener" mentality, and expect that if we moved to journals we would find a fairer review process... and if at some point we do so, they will receive a blow of reality (not saying it's your case, of course. There can also be people who have published in journals and disagree with me due to different experiences! And there are some journals that don't tend to engage in that kind of corruption, only that there are not many of them).
Yeah I don't submit to journals often but have had similar experiences. They have also asked for addition of significant experiments with significant compute. Like they wanted us to try our technique on an additional 3-5 networks (which are all of the same general architecture we originally did) and it is very clear that one of the reviewers was an author of one of those (but adding the other two to "mask" themselves). It would have doubled the project's compute and they weren't happy when we responded that we simply don't have the budget and aren't convinced the experiments would be meaningful but we'd be happy to add them to our table for comparison. All three suggested that the paper did not have much value but the paper already had >200 citations via arxiv by this time too... (several works built on ours and even got published...)
It's a really weird political game and I just think we need to move to a system where we either have reviewing seen as a collaborative/allied effort (as opposed to adversarial) or we rethink the whole system entirely and let reviews happen naturally (i.e. submit to OR and allow comments). It's just very clear that a system can't scale if it relies on all members acting in good faith. Especially when there's high amounts of competition.
Edit: one thing I find interesting is I contextualize different than other reviewers. Like I see a experiment section where they have a node of 2080Tis and think "okay, students or small lab. Compute bound, so do the experiments they performed optimize under those conditions?" Whereas I think most reviewers don't contextualize in this manner and so I see many act as if every lab has access to many A100 nodes. I think this matters because the small labs can still do essential work, but we just need to recognize the signal may not be as strong. Their workload is probably higher due to lack of compute too (and I subsequently tend to see deeper analysis and less reliance on letting the numbers do all the talking). I don't think the GPU poor "aren't contributing" because they can't, I think they can't because gatekeeping. It's insane to expect academics to compete with big tech in a one-to-one comparison. If all you care about is benchmarks then compute always wins.
No I just meant how much could it have changed since 2014 if it was also 2014 or soon after it, seems like a weird thing for a reviewer to say without further qualification
I'd have to go ask my dad about how all this works specifically lol
This is a great point about differentiating between filler and seminal papers. And I actually think the word filler is spot on for two reasons.
Firstly, this reminds me of a blog post about what it means to get a PhD [1]. The problem with that view is that the circle of human knowledge is drawn as a solid, while in reality it is very holey. So seminal papers push the border outward, while filler papers fill in the holes. Both valid and necessary, but general public only cares about advancing the borders.
Also, filler papers are needed not only for the sake of completeness, but scientists need papers to demonstrate productivity. E.g. graduate students needs papers on their CV so they can prove to their next employer that they are productive and capable of serious research. So these filler papers also "fill" the role of "filling" up people's resume. The public seem to think that professors publish papers only to announce to the world about their great discoveries, but 90% of the time its filler papers just reporting on business as usual type progress.
This was what a few academics hypothesized and set out to prove in what is now known as the Grievance Studies Affair [1]. Not only did they get multiple hoax papers published, they got recognized for “excellent scholarship” [2].
That grievance studies thing was idiotic, and really ought to be dismissed and forgotten. Reasons:
1. The hoax authors didn't submit gibberish or anything, like with Sokal - they submitted coherent papers that they considered absurd, but which they expected the journals to find reasonable. As such, they only thing they can even pretend to have proven is that they and the journals disagree about what's reasonable.
2. IIRC, every one of the initial hoax papers was flat-out rejected! If the authors had any backbone they could have stopped right there, and published a paper called "These grievance journals aren't as bad as we expected." :D But they didn't - they set out to iterate on each paper until a journal accepted it. And if your experimental methodology is "keep trying different things until X happens", you can hardly pretend to have proved something when X happens.
I mean, I agree with the premise the hoax authors were trying to prove. But their "experiment" was silly, and I think it's dishonest of them to pretend it proved something. Meh. Meh!
Excellent summary. I’d also mention the lack of a control group. We’re supposed to conclude that it’s easier to (eventually) get fraudulent results published in so-called ‘grievance’ fields than it is in (say) physics. But the hoax authors never made any effort to publish fraudulent results in those fields, so we can’t know if the percentage of papers that were eventually accepted would have been lower or not.
We were typing at the same time - see my other comment (sibling to yours) linking to Thiel arguing things are likely worse in hard sciences, and encouraging people to find ways to expose it.
Even so, it was a good demonstration of the point above: the peer review process in journals is not designed to keep out garbage, and is incentivized to apply less scrutiny when the subject is politically convenient.
Peter Thiel in this recent talk argues by inference that this problem is likely deeper and more pervasive in hard sciences than in the humanities. https://youtu.be/lO0KH-TgvbM?t=22m20s
Any evaluation metric is subject to political influences. Whatever the exact mechanism, it's people who are evaluating the papers, and people have political views.
That's my issue with it, really - I think it was widely presented and received as having "demonstrated" something, but if you look closely, by reasonable standards it didn't.
Actually if you dig up the authors' initial article about it, IIRC at the end there's a line like "so what does any of this actually prove? Well, we'll leave that for you to decide." If the authors can't say what their experiment proved, I think it's reasonable to conclude it didn't prove anything :D
>It was later reported that the manuscript included plagiarized sections from Chapter 12 of Adolf Hitler's Mein Kampf (My Struggle), in which Hitler describes why the Nazi Party is needed and what it requires of its members. The authors replaced Hitler's references to "National Socialism" with "feminism" and "Jews" with "privilege".
Isn't this a pretty good example of how crude the hoax was? Of course some things that Hitler said won't be awful if you literally change the key words in the sentences. Imagine if I played this trick the other way round. I take something innocuous that a politician has said and replace random words or phrases with 'National Socialism' and 'Jews'. Suddenly what they're saying seems really controversial! And this shows...what?
Do you happen to have the passage in question at hand? Mein Kampf is a fairly long book, and it seems likely to me that there are passages of it which – with key words changed! – would not express anything particularly awful. But it is hard to judge without knowing what the modified passage actually said.
Edit: What the hoaxers actually say suggests that the passages were altered quite substantially: "The last two thirds of this paper is based upon a rewriting of roughly 3600 words of Chapter 12 of Volume 1 of Mein Kampf, by Adolf Hitler, though it diverges significantly from the original.". And you can read the paper here: https://newdiscourses.com/feminist-mein-kampf/#solidarity
Sorry, that was a good faith mistake on my part, and I edited it out before you made this comment. I got confused because the link also states (correctly) that it was rejected from Feminist Theory.
Twice in two comments foldr quoted from a source. Both of those quotes are literally right next to another sentence in the source which is exactly the opposite of what foldr asserted.
The Affila wikipedia page section titled “Grievance studies affair” is four sentences long. The sentence right before foldrs quote, which is the first in the section is this.
>In October 2018, it was revealed that the journal had accepted for publication a hoax article entitled "Our Struggle Is My Struggle: Solidarity Feminism as an Intersectional Reply to Neoliberal and Choice Feminism."
The New Discourses link does say the paper was rejected from a specific journal in the first footnote which is . The second footnote says the article was accepted. Both footnotes quoted here:
> Peer reviewed and rejected by Feminist Theory
> Accepted by Affilia , August 21, 2018
At this point I am no longer sorry for not assuming good faith from someone arguing that a reformulation of kompf could be “not that bad”.
If I was intentionally lying, it seems unlikely that I’d have chosen to link to a page which clearly shows that my statement was false. It was just a careless mistake that I corrected as soon as I noticed it. Relax. I haven’t personally attacked you in this discussion. You’re not making your own point any clearer here by harping on this.
I don’t accept the characterization of being thorough as harping, or that anyone would need to be unrelaxed for that. Is that ironic with me being told to assume good faith?
Rules of thumb by definition don't 'scale' (is this the right word?): they're true on the whole most of the time.
The point of 'assume good faith' is that it's very easy to convince yourself that semi-anonymous people on the internet are liars. Most of the time they're not. It's very easy to inadvertently make factual errors. Published books that have been carefully reviewed are full of them. What chance do random morons ranting on the internet stand?
> Imagine if I played this trick the other way round. I take something innocuous that a politician has said and replace random words or phrases with 'National Socialism' and 'Jews'. Suddenly what they're saying seems really controversial
I am not sure this is obvious or even true. Can you provide one example? "Gas all the <X>"...what? capitalists? murderers? rapists? pedophiles? It doesn't even make sense for them, what else is there?
inb4, no you don't want to gas these people, you want to fix them if possible or jail them otherwise.
“We must fight against the influence of special interests in American politics.”
“We must fight against the influence of Jews in American politics.”
You may or may not agree with the first statement, but it’s hardly in the same category as the second (which is deplorable). It seems to me trivially easy to construct many more such examples.
The hoax paper replaced references to Jews with references to privilege. It’s hardly surprising that rants against an abstract concept are less offensive than anti-Semitic rants.
> “We must fight against the influence of special interests in American politics.”
> “We must fight against the influence of Jews in American politics.”
To be fair, some people do use the former language to mean the special interests of "Jews." No all, but dog whistling does exist and dog whistling is explicitly about covert language. I don't think the special interest groups reference is a great example because it is a common phrase that is used to mean a lot of different things and isn't uncommon in various groups using covert language.
It's a phrase that pretty much every major American politician has used. Of course there are dog whistlers and conspiracy theorists, but I think the less problematic usages predominate. Not that the phrase really means anything as far as I can tell, but that's a separate issue.
I do see what you mean though. There's something inherently gross about juxtaposing two sentences like that. I'm not doing it to suggest any equivalence between them. The point is exactly the opposite (and I would have thought an uncontroversial one): that switching out the major vocabulary items in a sentence can take it from being innocuous to offensive or vice versa.
> Not that the phrase really means anything as far as I can tell, but that's a separate issue.
Actually I think that's explicitly the issue and a main part of my point. In fact, dog whistling or other type of coded language typically depend on ambiguous language. It's literally because language works like an autoencoder. There's what you have in your head that's encoded into what you say and then decoded. The coded language comes through a learned/tuned decoding.
My point is that such phrasing isn't inherently innocuous. Vague language is always inherently dubious. The juxtaposition just makes it more obvious in this case, but it's always true. It may not always be dog whistling but in the least dubious case it lets people fill in whatever they want. That's why I'm saying it isn't a good example.
Hmm. I think it's a fine example because 'special interests' is very rarely used as a dogwhistle for 'Jews'. Can you even point to an example where it is?
I'm not sure why you would be so caught up on "Jews" specifically when there are plenty of other targets that can be used. But yes, that term has been used to refer to Jews the same way one might say "Hollywood types" or "Bankers." I can't think of a specific example, but I think that's a bit high of a bar as I don't have an eidetic memory and it's not like I've been keeping a log of every time some mentions special interest group. Like most people I only remember the broad concepts and general notions. If you're aware of a way to easily Google or search this information, I'd be quite interested to know, because that's a very useful task. In the mean time, I'm sure you could find some if you dug around more around the Ukraine war, with both US right wing and Russian news platforms being the most likely place to find these. There's a lot of anti-semetic conspiracies revolving around Ukraine (with it even being the Russian's main initial propaganda) so that's why I suggest looking there.
Well, we disagree over the usage of 'special interests', and as you say, it's difficult to gather objective data to support or refute either view. It's obviously a vague phrase that could be (and occasionally has been) used to dogwhistle many things, but that doesn't mean that a sentence attacking "special interests" is inherently as offensive as a sentence attacking a specific ethnic or religious group. That is the only point I'm making here, which I suspect you don't even disagree with. Even in the case of a dogwhistle, the whole point of a dogwhistle is that it is less immediately and unambiguously offensive than a straightforward statement of a prejudiced viewpoint.
By the way I'm not 'caught up' on 'Jews'. It's just that 'Jews' is one of the words that the authors of the hoax article replaced.
From what I have read so far, they showed that if you completely fabricate the data and follow the advice of peer reviewers, you can end up at a published paper.
Thank you, Captain Obvious.
Had these valiant warriors heard of Diederik Stapel, they would have known that fraudsters already made this point. Extensively.
These fighters for academic rigour seem to have lost sight of their own academic rigour in this "experiment".
Peer review is insufficient to catch fraudulent submissions. This is not a surprise as much as a misunderstanding of what peer review is.
To be clear: to show something, the authors should have shown that the ideas were obviously false, not that fraudulent data that supports findings would go undetected. Einstein's famous works were considered patently absurd at the time. The only way to substantiate or refute such ideas is with data. An experiment that abuses this to "show" anything is, itself, patently absurd.
The peer review system is definitely biased. It's ok for many articles, but there are outliers. For instance really original theories or experimental data. The publication may be stopped to avoid putting the journal's reputation at risk or because the reviewers have personal reasons to not support the article (it would shade their own pet theory, invalidate their research project, they didn't understood it, etc.).
The problem with publishing elsewhere is that many people unable to objectively evaluate the validity of the theory juge its value by the reputation of the journal. Also the other journal will most probably have less visibility. There is thus a higher chance that the other theory will not be reported in reviews.
There are many assumptions in the above comment that I can't agree with. But my experience is with physics, not computer science.
Maybe a better descriptor would be that peer review is a reputation-based system.
The peers that will review your work likely know about the paper you submitted already, because they work on related work themselves and sat through your conference presentations. Most of them want you to publish your work and will provide a good/non-adversarial review of a paper.
Sometimes though, your paper hits too close to home for them, then they will try not to get it published or will slow walk the review so that their own work can come out before yours or at the same time.
On top of that, you have journal editors who can see everything about the process and can decide to ignore a good/bad review to fit their ideas about the paper itself and to fit the overall vision they have for the journal for the coming publication schedule.
> Sometimes though, your paper hits too close to home for them, then they will try not to get it published or will slow walk the review so that their own work can come out before yours or at the same time.
How often does this actually happen? Can't say I've heard of people doing this.
> It has come to my attention that academics are now using generative AI (Chat GPT or whatever) to conduct their peer reviews.
> [..]
> If ever there were a compelling argument to totally abandon the intellectually dishonest notion of “blind” peer review – a system everyone knows is broken, rarely fully “blind” and frequently not “blind” at all, but allowing unscrupulous or mediocre yet established scholars to sabotage promising work – then this finally is it: accountability and transparency in the face of the robot takeover. Every person who reviews another person’s scholarship must be willing to sign their name to their own evaluation, to stand by it and assert it was not the work of the machines.
It is far too early to say there’s some ethics convention for not using ML models to write peer reviews.
If you used a machine to write your peer review, you’ve staked your reputation on its output being correct (in the same way that you stake your reputation on not producing bullshit peer reviews, which is to say… eh, probably not a make-or-break thing but it is in the mix). So you need to check the output. That’s the skill we as a society value.
We don’t employ scientists for their literature skills, but for their ability to build and evaluate theories and data, that sort of thing. And hey, less brainpower remembering grammar rules means more for the study of science. This is good.
Some people will torpedo their careers by putting too much faith in the ML model, but the way to avoid being them is to apply the same level of diligence to the model’s output as any other tool.
I’m more worried that ML models will enable some continued silliness. There’s something odd going on if people type their actual arguments into an ML model, then it produces extra necessary filler text to get into a journal, then we use ML models to review that text and hope to distill it to the actual arguments, and then the general public again uses ML models to do the same. It seems like we’d just be better off sharing the prompts or something, hahaha.
I'm outside this world but what always strikes me is the risk/reward gamble that cheaters are taking.
By the time they are caught cheating, they have invested dozens of years if not decades into a career that is now pretty much dead. Is there a life-after-cheating story in the relevant field? I can't imagine much of one. Part-time lecturer/tutor in a fourth rate school, perhaps.
Of course, that presumes a moment in time where they begin cheating, risking it all. If they were cheating all along, from age 12 onward, maybe stopping is the problem.
This is a great characterization of why peer review is great - it makes honest scientists better. A lot of progress is driven by "jumps" from influential papers, and we want those to be as good as they can. It may not stop frauds, but the frauds weren't going to help us anyway. I think fraudulent research mostly hurts by distracting honest scientists and new scientists, not by convincing them of something untrue.
Peer Review seems like a system designed to encourage group think.
The findings of Copernicus that the Earth rotated around the sun instead of vice versa would not pass peer review. Nor would Louis Pasteur's Germ Theory.
Yeah I agree. The only good system is one where I judge the validity of the work. Especially if its something I don't know about because then I have no baises.
You may think it’s a joke. But it’s pretty much the core idea of the Enlightenment (that everyone can think for themselves and don’t need priests to tell them what to think). The motto of the Royal Society is “Nullius in verba”, “take nobody’s word for it”.
Physicists have been using preprint servers for decades, which means anyone can put a paper on the internet for everyone in the field to read and evaluate. So this idea that Copernicus would be suppressed under the current system is absurd. Einstein and Bohr's ideas passed peer review, as radical as they were. Every physicist is hoping for data that contradict our current theories. It was a huge disappointment when the only result of the LHC was to confirm the standard model.
For all the flaws of academic publishing, we are still in a much better place than when church dogma was the gatekeeper of knowledge.
Thanks, I wasn't aware of the level of peer review of physics journals in the early 20th century. Note that Annalen der Physik was publishing 90-95% of submissions, so there was still some review process. I also wonder how much of that had to do with the average quality of a researcher working in physics back then, since physics was still a specialized pursuit of a very small number of people. We have orders of magnitude more people working in science today, so it might not be feasible for journals to publish 90-95% of submissions without a significant drop in quality. I think the solution is for more fields to adopt the model that most of math/CS/physics already follows. Put everything up on preprint servers and allow people to cite preprints. That way the community can accept ideas and build on them regardless of whether a specific paper makes it into published journals.
If people would actually _measure_ the outcomes of peer review instead of talking about it, I think it would meet a swift end.
Empirically, I find no improvement in reproducibility between arxiv and journals. The costs are incredibly high too.
Like many things in our world peer review is a short lived extrapolation which doesn't resemble it's origins but is regarded as immutable gospel. It matters most if what you need is the respect of academics.
He says peer review is for the authors, but I think science being peer reviewed is one of the key points used to convince the public to trust scientific results.
If someone thinks that a scientific paper is true because it passed peer review, they need to change their mental model of how science works. Peer review ensures that a given paper meets a minimum standard of quality. Trust of scientific results emerges gradually as the broader field forms a consensus based on dozens or hundreds of papers.
where things seem to really go wrong with peer review is when entities outside a scientific community want to use published research to set public policy, decide where to invest, or make high-stakes hiring decisions.
doing this actually requires a costly investment in deep understanding of the research and the literature around it, to know if it is sound and high-impact. but nobody wants to make this investment, and they've mostly convinced themselves that just free-riding off peer review is good enough.
of course it is not, flawed papers make it through peer review all the time. also this outside use of peer review introduces extremely strong distorting incentives -- an even hugely greater desire to be published in specific prestigious outlets -- in the face of which peer review is not really adequate to catch misconduct.
I think the ideal form of peer-reviewed journal is unfortunately logically impossible: its contents would be completely open access, costing nothing for scientists who want to build on the results. But it would require an extremely expensive subscription to find out who has published in it, the money going to fund extremely thorough peer review, so that outside decision-makers can't try to free ride and distort the incentives.
I think this is not even really a problem with the peer review system, but a problem with the (mostly, nonexistence of) an in depth science journalism field. Peer review just means it is ok to stick in a scientific journal and have peers read it… people with advanced bullshit detectors and nuance parsers in a particular domain.
A paper is a brick. A policy is a building. It is not a problem in the brick manufacturing field, if people keep trying to build houses without any mortar.
(Just to be explicit, I think you are right on the money and just wanted to elaborate/rant).
> Peer review just means it is ok to stick in a scientific journal and have peers read it… people with advanced bullshit detectors and nuance parsers in a particular domain.
That's not what peer review means. If that were what it meant, it would have no purpose - the people you describe don't benefit from the paper they're reading having been through peer review. They already understand the domain.
> The solution to this problem relies on a simple observation – in my field, I am completely indifferent to whether a paper has been “peer-reviewed” for the basic reason that I consider myself a “peer”. I do not think it extremely hubristic to say that I am reasonably capable of evaluating whether a paper in my field is worth reading, and then if so, of judging its merits. The opinions of other people in the field are of course important, but in no way does the fact that two or three nameless people thought a paper worth publishing influence my opinion of it.
The only goal that peer review serves is letting people who aren't peers and don't understand what the paper is saying believe that they can tell whether the paper is good or not. They still can't, but because of the peer review system, they feel that they can.
It isn’t pointless, researchers have finite time and attention, the point of peer review is to keep out the really bad papers so that they don’t waste their time on them.
> I think this is not even really a problem with the peer review system, but a problem with the (mostly, nonexistence of) an in depth science journalism field. Peer review just means it is ok to stick in a scientific journal and have peers read it… people with advanced bullshit detectors and nuance parsers in a particular domain.
I don't agree with this. There is deception in its name: peer review. Even "preprints" (things on arxiv and not accepted in journals or conferences) are still read and evaluated by peers. LK-99 is a great example of this, and that's not an uncommon process. The problem is that venues perpetuate the idea that being published within them is a mark of validity. But, like you suggest, it isn't. And "peer review" doesn't mean that it is just okay for peers to read it because we're reading each others works long before they are released in venues. Unfortunately most people (including researchers) don't have advanced bullshit detectors or nuance parsers. These are often fairly narrow and there's not that strong of alignment between the subject matter that a reviewer is reviewing and their niche domain expertise. I agree journalism and politicians need to do better, but so do we in the scientific community. There's more nuance. It is definitely a problem with the brick manufacturer if the marketing team implies you don't need mortar or even says "generally you don't need mortar."
> entities outside a scientific community want to use published research to set public policy
I don't think it's limited to entities outside a scientific community, because many of the people who want to use published research to set public policy are themselves scientists (often the same scientists who published the research).
> where things seem to really go wrong with peer review is when entities outside a scientific community want to use published research to set public policy, decide where to invest, or make high-stakes hiring decisions
OK, but those are the only use cases for peer review.
In fields such as Math, Physics, EE, CS, and AI, "peer review" is already being replaced by open debate online. Cutting-edge research in those fields is now routinely posted first on repositories like arXiv, with supporting code and data made public first on sites like Github. The work is reviewed first online, in the open, in a variety of forums, including X.com (formerly Twitter). Anyone with something to contribute can participate, regardless of pedigree. It's so much better than the outdated system of "peer review," which actually isn't that old.[a]
The question is: Who benefits the most from preserving the dying "peer review" system? Who loses the most from the transition to open debate? Short answer: Publishing houses and conference organizations. Neither wants the status quo to change.
If you are well known in your field, you don't need peer review. People will read your preprints, invite you to conferences, and follow you on your preferred platforms anyway. If you are in a well-known project, people will pay attention to your work, even if you are not famous yourself.
For everyone else, there is peer review. Attention economy is unforgiving and benefits the elite. With peer review, everyone's work gets at least a minimum level of attention. Which may then lead to more attention if the work is worth it.
Every serious proposal for replacing the current peer review practices must have the same feature. When someone submits new work, there must be people who have to review it. Fully voluntary reviews don't work.
Peer review still occurs outside of journals, it’s simply less formalistic.
One of the biggest advantage of peer review is it allows obvious corrections from a 3rd party to occur without wasting everyone in the fields time. Gatekeeping isn’t the point, increasing the signal to noise ratio is.
This is what I was trying to say a couple weeks ago and kept getting down voted into oblivion. History had shown anything that depends on human actors will be manipulated.
What is the alternative? Channeling Chuchill, peer review is the worst form of journal review except for all those other forms that have been tried from time to time.
As an associate editor and frequent reviewer, I can say with confidence that peer review filters out at least 90% of the garbage that journals receive as submissions. It also provides useful feedback to researchers who cannot otherwise get it. People submit papers knowing they will get rejected because they want to read the referee reports.
In my field, academics simply do not trust a result until it has been replicated and scrutinized by other researchers. If that doesn't happen, it's usually because the result was not interesting enough for anyone to care. If the scrutiny reveals problems, that undermines the credibility of the authors, which is ultimately more important than whether they get the pub.
A final comment: Peer-reviewed journals differ dramatically in their quality. If you don't know what the good journals are, you should expect that your reading will include a bunch of crap that no serious person in the field would pay attention to. Why these crap journals even exist is a mystery to me.
And by charging the authors to publish the paper in the first place. Although I think a lot of the money both ways comes from universities and other organisations that pay for publication costs and have deals with publishers.
Thank god for CS's culture of pre-prints on arXiv.
Yes, that's got to be part of the explanation. What's not clear to me is why anyone gets professional credit (e.g., tenure) for publishing in a journal that is obviously bad.
The answer to your question is just part of the whole sad story...
Promotion committees (hopefully!) know which journals or publications are low quality. If they are unfamiliar, they generally don't have time (or inclination) to verify the paper(s) or publication itself. Especially true when it's a slightly different sub-field than their own. So it often comes down to a simple count of pubs. That's usually all the credit they need!
His use of "he" or "his" research was a bit dated looking... More natural to say "they" presented a paper and "their" paper didn't pass muster or whatever
> peer review may perpetuate some biases and prevent researchers from putting into question some fundamental questions (“we decided that this is the right way, if you question it, you are a loony”).
reply