How to Paper Review Computer Science Technical Depth
Toxic Peer Review in Computer science and What to Do about It…
On peer review and some of its most significant failures
Scientific Peer Review affects all informatics researchers and academics in their daily life. More often than we really desire to admit, the process causes severe problems and wrong decisions are existence made.
PEER REVIEW EXAMPLE
Reviewer one: Good paper — take
Reviewer two: Newspaper OK — weak accept
Reviewer 3: Newspaper not and so great , might exist useful after changes— weak turn downMeta Reviewer:
THIS PAPER IS BROKEN BEYOND ALL REASON — Reject!
All scientists face such situations, eventually. They hurt every time. And then, is this organization broken? Why the heck do we fifty-fifty play this game?! Information technology'due south time to look at all the juicy details of peer review, its weaknesses, and some really toxic examples! (Alarm: Heavy triggers ahead!)
In this post, I want to explain the scientific peer review system in reckoner science and its pitfalls. Also, I remember it is a good opportunity to betrayal how the arrangement is under constant attack and to report a couple of cases of scientific misconduct that I experienced myself in this arrangement.
Why do we need Peer Review?
Peer review is a mechanism that developed over many years and its aim is to ensure quality in publications. All scientists are prone to error. It is in your all-time involvement to have someone checking your work independently to preclude yourself from publishing erroneous studies. Imagine how a serious mistake in your work could backlash! Once it is published, everybody will run across information technology! So one could say, it's a prophylactic belt for scientists.
For these reasons, many scientists only consider peer-reviewed piece of work every bit a scientifically grounded observation. Therefore peer review is office of many parts of science. I consider these three different kinds of peer review in here in particular:
- Peer review at conferences
- Peer review in journals
- Review of thesis piece of work, e.g. Ph.D. theses
How does the typical conference peer review work?
The almost frequent form of peer review is the i of scientific conferences. This is typically as well the first experience of reviews to scientists in their career. Depending on the field, conferences may vary from shallow feedback to an elaborate competitive process. In all cases, the review is performed under fourth dimension pressure with a stock-still deadline. As all scientists are decorated people this constraint is not without effect. In some occasions up to 30% of reviews are submitted after the deadline — often by then-chosen emergency reviewers within 24 hours. Obviously such reviews are more prone to errors than a total review that may take weeks to complete. As a outcome, peer review is oft perceived as a stochastic decision.
Types of Briefing Reviews
In order to ensure articulate and likewise disquisitional remarks, most conferences utilize a blinded review procedure. In a then-called single-blind review, the identity of the reviewer is concealed such that negative feedback does not result in acts of vengeance. Notwithstanding, some authors try to re-identify their reviewers by the used language and terms. I can simply advise confronting this as it will but outcome in ungrounded accusations.
Some — often highly reputed conferences — endeavor to reduce potential bias due to personal dislike even further. In these cases, a double-bullheaded review is used. Hither, the authors remove their names and all data from the paper that would allow to identify them. Thus, the reviewer also does not know whose work they are actually rating and is unable to favour certain groups of personal liking over other authors.
Peer Review is a Random Procedure
Reviewer decisions suffer from a certain degree of randomness. They may be affected past a certain diction in the paper or question the general idea. Therefore, a single reviewer is regarded equally unreliable. About conferences choose between two to four reviewers and consider their majority vote on the work under consideration.
In top-tier conferences, even this is not regarded equally trustworthy. Hither a so-called meta reviewer is introduced into the procedure. The meta's task is to arbitrate situations in which the reviewers do not find consensus in the first round of reviews. In these cases, the meta tin guide the discussion and assist the reviewers to find an appropriate evaluation of the paper. Unfortunately, like in the example in the beginning of this article, some metas regard themselves equally some kind of infallible deity that grants wisdom to the rest of the globe. Every bit such, typically all meta decisions are also discipline to control by a higher case such equally the technical program chairs. Even so this option besides bears the take a chance of introducing another cistron of randomness, due east.g. if vi technical program chairs have to manually cheque more than 1500 newspaper decisions.
In order to reduce randomness further, some scientists suggested to utilize even more reviewers—like xxx to fifty. In these cases, obviously a full review would not be possible, but the feedback would simply be "thumbs up" or "thumbs down". Obviously, this would yield much better decisions as we know from boosting theory. Yet, this was never implemented at a existent conference, as the large number of decisions would most probable lead to the trouble that the reviewers will judge the papers merely on the titles or the abstracts and not read the complete submissions.
Pros & Cons of Conference Reviews
Generally, conferences merely employ a one-step or two-step process. Hence, there is no chance of in-depth discussion with reviewers. Even the near elaborate conference review processes endure from this trouble.
Therefore, you should not get demotivated by negative or harsh reviews. A lot of the work is washed by volunteers under time pressure level. Also, the reviewers are prone to fault. Try to empathize the reasons for their critique and utilize it to improve your work.
Some conferences aim at loftier rejection rates as a measure to publish only "the best" newspaper. As the stakes are high, reviews tin often become very harsh. Likewise, expectations on summit-ranked conference papers are very high. Fifty-fifty though peer review — in detail on conferences — is a stochastic process, the ameliorate papers are more probable to survive the peer review process.
Still, a decline is not necessarily a sign of poor work, due to the above-mentioned random factors. There are numerous examples of good papers that were rejected at conferences in their beginning submission (e.g. nnU-Net, Deep Activeness Learning, SupER Benchmark, and many more).
Why do we notwithstanding publish with scientific journals?
A pregnant disadvantage of journal publications is that peer review typically takes much longer. One circular of reviews may have between three to half dozen months. Still, submitting to journals is of loftier value as the peer review process is much more elaborate and focuses much more on in-depth arguments. In many journals, the discussion between the authors and reviewers become back and forth several times. Every bit a consequence, reviewers pay attending to the details of the submitted work. They know that their cess will be presented to the authors and they get a chance to accommodate their work and argumentation accordingly. If their arguments are not elaborate plenty, their critique volition be very easy to counter. Hence, arguments like „not good enough" or "not novel plenty" are not being used. Instead, a list of prior piece of work that already addresses the issue and was not covered by the authors needs to exist presented.
As authors know that they have to debate with the reviewer, some authors accept the trend to talk over all observations of the reviewer abroad. This is, still, not the purpose of peer review. Each circular should better the paper. Then authors should not autumn for common mistakes such every bit taking the critique personal or deeming the reviewer equally stupid. Reviewers are representative readers of your newspaper. Other persons reading the newspaper will take like thoughts.
Therefore, if you have to provide boosted explanations to your reviewer, you want to add them to the paper also. A discussion section is the right identify to do and so every bit other readers will have similar questions. Such a department can too be a safety belt, if your paper goes viral and is broadly misunderstood in a general public. In such cases, y'all can bespeak at the give-and-take section which should already counter foreseeable critique and misunderstandings. Also, you want to make sure that your argumentation is correct. Any errors will be spotted past the reviewer and you risk rejection of your paper.
A reject at a journal means that you lot cannot submit it once more unless you completely rework the paper. (If your newspaper gets rejected, y'all probably want to do that anyway.)
I recommend to try your chances at high-ranking journals and venues. A desk turn down will maximally take 2 weeks and it will give yous the feeling that you lot tried at least. Publishers such as Nature even have the instrument of a presubmission inquiry in which but the abstract will be evaluated within one or two weeks.
Should we still write Ph.D. theses?
A third kind of review that I desire to include in this article is the review of thesis work such as Ph.D. theses. Some supervisors asking accepted peer-reviewed publications for the successful completion of their Ph.D. Many universities even permit cumulative works, that is, a fix of peer-reviewed articles that is submitted instead of a thesis. Nevertheless, this process has quite a few bug.
As the thesis is associated to a formal degree, cumulative works accept to provide evidence that the set of submitted works is really based mainly on the contributions of the submitting person. Hence, authors are oftentimes required to ask their co-authors to sign off the use of a paper as part of a thesis. In near cases, in that location is no abuse and the forms help to spot situations in which the same paper is attempted to exist used for the graduation of two different persons.
However, the problem could ascend that someone who actually did not write the work on their own could endeavor to obtain a degree using a long list of co-author contributions. As a result, some academy withal require a the writing of a formal thesis. The list of arguments for and against this is long. Some consider it every bit a waste of resources. For others it is the only means to demonstrate true academic ability.
In whatsoever case, the resulting thesis will exist subject area to evaluation past independent reviewers. Apparently, they volition assess the work to the best of their ability. To my experience, it helps to show that the piece of work has resulted in high-ranking peer-reviewed work demonstrated either past impact or citation factors of the venue or the number of citations of the work itself. Therefore, pursuing peer-reviewed publications during the scope of a Ph.D. program is highly advisable.
Pitfalls in Peer Review
For the above reasons, the pressure to publish in peer-reviewed venues may be strong. Therefore, authors that experienced previous rejections might be looking for lower-ranked venues in order to have their work accepted quickly. While this option seems similar a quick mode out, this path may also lead to problem.
Predatory journals annunciate fast publication with total peer review. Frequently this is even done with a reference to a prior publication of the authors. Unfortunately, they just publish any contribution without peer review and charge the publication or conference fee. Acceptance of a paper in such a periodical may upshot in serious damage to the authors' reputation, even if the submitting author did non know about the nature of the journal or if a co-author was added to the authors list without his or her noesis. Public lists of such journals help to avoid such simulated peer review.
What works for journals can obviously also be applied to conferences. Predatory conferences are also hard to spot. Oftentimes they are at popular tourist locations such that submission and travel get even more attractive. One way to identify such conferences is to look for like events occurring at the same date and location. You might be able to find several "Earth Conferences on [insert topic here]" that are organised simultaneously sharing even the same keynote speakers. In society to avoid such scam events, yous may want to consult your colleagues or briefing rankings such as guide2research.com.
Lately, Video Journals have also emerged. While the peer review procedure seems rigorous, the publication and the production of the video by the journal might be rather expensive. Therefore, authors are well-brash to report potential cost of publication before submission to whatsoever venue.
Toxic behaviour in peer review
Plain misconduct is more often than not very rare in peer review. Toxic behaviour is institute much more than often. The examples reported here range from unpleasant, but common, to extreme cases. Note that it took my entire career to collect these examples. Scientific fraud is extremely rare. Yet, ane cannot deny its being.
A very mutual toxic behaviour is to exist overly negative to other works. Even during anonymous peer-review, you will not be presented work of your own institution. Equally many conferences set a hard threshold on the credence score, whatsoever negative review that you create volition increment the probability of your own grouping's papers to be accepted. Likewise you may spot similar work to your own and y'all may want to filibuster its publication on purpose. Hence, a common strategy of some reviewers is to find easy reasons to reject a paper. Statements like "not novel", "not articulate enough", and other but coarsely grounded reasons are typically found in such rejects. Therefore, yous want to make sure that you are enlightened of all relevant literature and compare information technology to your ain work — in the best case — to counter claims of lacking novelty. Besides highlight the strong points of your method or research equally much equally possible such that claims of defective clarity can be avoided. If you piece of work in machine learning, you want to avoid the 7 Sins of Machine Learning, as they are like shooting fish in a barrel indications for a fast rejection.
Also some reviewers try to increase their own citation statistics. Yous volition spot such reviews giving a mixed review in combination with the demand to cite several papers that are all written by the same grouping of authors. While using such behaviour to increase one own's commendation count is already questionable, this procedure is problematic for some other reason: The identity of the reviewer is easy to estimate and may therefore hamper the anonymity of the peer review process. Hence, performing such reviews is not advisable.
With the increasing number of submissions in the past years, the number of peer reviews is also increasing. Therefore, some reviewers who run out of time don't practice the assessment themselves. If they happen to head a grouping, they sometimes ask a fellow member of their team to perform the review for them. If they still check the review for validity and even if the profitable person agrees to help in a confidential fashion, the process is even so decumbent to error as the confidential peer review fabric has to exist transferred to a third party. Hence, the procedure is legally questionable, in detail at venues that exercise not have clear legal terms. Fortunately, this trend was already identified several years ago and counter measures such every bit enabling peer review also for inferior researchers have been taken. This increases the number of possible reviewers and decreases the work load for every reviewer.
Editors and meta reviewers have a lot of power. With great power comes — who would have guessed — dandy responsibleness. Information technology is tempting to push a decision towards a sure direction because you know that it is correct. You are the person in charge and there is no way how your conclusion ever could be changed. It will only relieve the authors, the reviewers, and yourself a lot of time, right? Well, no. Y'all might be missing an argument or an important observation that could tip the decision. Therefore, catastrophe a controversial review on a paper prematurely comes at the risk of a incorrect determination, in particular if you lot have to overturn the majority of the reviewers as in the instance in the beginning. Imagine how you would feel every bit an author not being able to reply to controversial critique.
A terminal questionable exercise happens when meta reviewers and editors assign papers only to their own students. This enables the forming of cliques and increases the likelihood of receiving like dependent reviews. Therefore, reviewers for a sure paper should ideally originate from dissimilar institutions globe-wide.
Clear Instances of Misconduct
Cases of scientific misconduct are rare and may have astringent consequences, if people are caught. In particular, plagiarism of other works are piece of cake to spot and will be discovered sooner or later. Since the emergence of plagiarism checkers, this procedure can even be performed automatically and many journals and conferences already use them or plan to employ them in the most future. Therefore copy & paste from other works is ill-advised.
In order to bulldoze their ain citation count, some authors appoint in so-called newspaper mills. Typically, journals of low rank or even predatory journals are used for these purposes. Also, some authors form cliques that engage in heavy mutual commendation. While some of such networks may merely emerge form the construction of a sure field, clearly abusive instances are too known. A contempo discovery of a paper mill establish more than than 400 papers plagiarised including copied figures.
Every now and so falsified results appear in research. Astringent cases even make headlines in general news. Making a significant discovery is of course temping. Nevertheless, the menses of fame will be simply very short. Therefore information technology lies in the interest of all scientists to exist very careful near ones own observations. The will to scientific discovery may exist strong, withal the consequences of fabrication or even only carelessness will exist discovered as soon as the experiments cannot exist reproduced past others. In particular for motorcar learning inquiry, this might easily emerge by simply not paying enough attention. Such tampered results are probable non to be discovered by the peer review procedure and therefore are a significant threat to any researcher.
Under the tip of the iceberg
The above cases are probably well known to all scientists. Yet, I too encountered other instances of misconduct that might be non as well known. The cases reported in the following happened and have been reported. Hence, I suggest anybody to refrain from such practises. Yet, you may find some of these incidents shocking. Please note that I accept been agile in many different scientific communities. Therefore, you volition not be able to approximate in which community the respective incident happened and I too advise confronting fifty-fifty attempting so. I volition also not answer questions on who was actually involved in these incidents.
Peer review is of a stochastic nature. We know from probability theory, the more frequently we effort, the higher the odds volition be to accept a newspaper finally published. Even so, this procedure takes time because concurrent submission of papers is non immune. Still, I have seen attempts of groups that understand the nature of the peer review process very well. In a item case, the authors inverse the title, abstruse, and keywords and submitted the exact same paper twice. On this particular conference, the reviewer selection was based on keywords. Therefore, the probability of getting the same reviewer twice with a disjointed set of keywords was modest. With two submissions, you double the chance of the paper to be accepted. In the case that both submissions get accustomed, you but withdraw 1. In this particular case, however, the program chairs looked at all papers sharing the same set of authors such that the attempted fraud was detected. The affair was escalated up to the lath of this specific scientific community and was dealt according to the community's standards.
Sometimes a paper seems somewhat familiar and by accident you may even cease up reviewing your own newspaper or a paper from your grouping. Manifestly such incidents have to exist reported to the program chairs and they will accept care of this result. Also, authors sometimes change afterward the acceptance of the paper. Generally, papers get finished in the last few hours to minutes of the submission borderline. Therefore, there must exist opportunities to correct for last-minute mistakes. Nonetheless, what should never occur is that a reviewer suddenly appears as co-author of a submission later on acceptance of the paper. This incident was likewise reported and dealt with by the respective board of the community.
The nigh shocking incident that I ever encountered in peer review, even so, is a result of a combination of several of the above-mentioned practices, a actually unfortunate series of events. In this case, a young researcher submitted a paper to a highly esteemed, double-bullheaded conference in the field. The paper was rejected by a very close margin. Several of the comments helped to improve the paper. Half dozen months afterward, the paper was submitted again to a well-reputed single-blinded conference. This fourth dimension, yet, the paper was immediately rejected due to plagiarism. The new reviewers considered the paper as plagiarized and reported this finding to the conference chairs who started an investigation confronting the submitting author. Afterward careful consideration, the investigation found out that the anonymous, double-blinded, first version of the newspaper was discovered on a public file server of another very well known group in the field. Plainly, the paper was uploaded to the file server past one of the anonymous peer reviewers of the first conference. It seemed that a senior researcher delegated the review to one of his subordinates. What did not occur to him was that the group's file server was not protected confronting search engines. Therefore, Google started indexing the content of the file server. As a result, the 2nd conference reviewer found the paper when he was looking for relevant research in the state-of-the-art. He identified the paper equally plagiarism as the authors were different from the owners of the file server. After the investigation concluded, the submitting author was constitute as not guilty of plagiarism. Furthermore, this complicated concatenation of events did not lead to any measures against the group that uploaded the anonymous newspaper to their own public file server. In the end, the submitting author left academia and never graduated from their Ph.D.
How tin can I counter such problems?
If you spot pregnant toxic behaviour, you lot can report it. Conferences and journals offering confidential comments to the deciding example. If you doubtable a reviewer to have crossed the line, report your suspicion to the associate editor or meta reviewer. In any case, you should grant the do good of the doubt, equally the recommendation to cite just papers from a certain group may simply exist related to factual grounds. Always scrutinize your own suspicions.
Use ArXiv ! Once you published your ideas in your and your co-authors names, ideas cannot go lost in bearding peer review anymore. This is an excellent style to protect your intellectual property. Publishing pre-prints on ArXiv is by now accepted by all major car learning and pattern recognition conferences. Manifestly, you want to make certain that all your co-authors agree with this and potential invention disclosures are already submitted. Also brand certain that you update the ArXiv submission with the reference to the last publication such that other researchers find the peer-reviewed and published newspaper.
ArXiv is not a scientifically peer-reviewed source. If the paper is just on ArXiv, it is probably preliminary piece of work. It may still comprise potential errors and may be updated until its final publication. As well, don't report ArXiv publications equally relevant prior fine art equally reviewer, in item if the newspaper appeared there after the submission borderline.
Non all counter measures accept to exist taken by yourself. Conferences and journals now heavily rely on plagiarism checks. In many conferences and journals, reviewers now get the paper and the plagiarism written report for the assessment of the work. Also meta reviews and reviewer assignments are manually checked past the program commission. Often reviewers and metas get assigned confidential scores that rate their trustfulness.
All misconduct is handled in secrecy. Nosotros do this in lodge to not damage science equally a whole. Scientists should by and large be perceived as knowledgeable and trustworthy. Remember: Misconduct is very rare, mistakes are more frequent. Always grant the do good of the uncertainty.
Unfortunately, there is no general and clear „code of deport" that would clarify what type of misconduct is associated to what kind of penalisation. Cases are dealt with on an private basis. Therefore, consequences can also vary considerably from community to customs and from land to state. Be aware of the rules that employ to you, follow them, and appoint against toxic behaviour in peer review, for example past increasing your colleagues sensation on the topic. Don't commit acts of vigilante justice and accusations on social media!
If you spot misconduct, every scientific institution has an arbitration trunk that will have intendance of the issue appropriately. In some communities, the public accusation of misconduct may be considered as misconduct itself as it amercement the reputation of scientific discipline as a whole and is in violation of granting the benefit of the uncertainty. Remember: Peer review processes are complex, stakes are high, and certain behaviour is very easily misinterpreted. Always doubtable that everybody is acting on their best intentions simply be prepared for the worst-case.
And as a concluding hint: Don't travel to the conference that rejected your piece of work and distribute your paper on your own. If you lot have ever encountered a person doing this in front of a conference venue, yous volition immediately realise why this is non advisable.
Take-home messages
Reading this text, it became probably articulate that peer review has many problems. Yet, we are not aware of any arrangement that works ameliorate. Also, nosotros take seen that there is no single version of peer review. Every scientific community is building their own standards at their own needs. In addition, many peer review systems undergo constant change in order to find better solutions. This also prevents continued exploitation by players in the field who but know the process well. If you lot are in accuse of adjusting a peer review process, brand sure that none of the involved persons ends upwards with too much power. By and large, all contributors aim at interim on a fair and responsible footing. Yet, abuse of ability may be tempting and while being difficult to spot.
ArXiv is probably one of the best counter measures against unfair peer review every bit information technology takes the force per unit area of the need of early publication. Yet, an ArXiv submission cannot exist regarded as a full scientific piece if it has not undergone peer review successfully at some point in time.
Toxic behaviour in peer review more often than not only reduces the quality of the result of the peer review process. Its aim is to get an unfair reward over other groups. Furthermore, toxic peer reviews are likely to evoke more toxicity in other scientists. Therefore, one must refrain from using toxic practices at all times, even if you were a victim of such behaviour yourself.
Some behaviour may seem as misconduct, but it may also be just an unfortunate series of events. Therefore, always grant the do good of the doubt. If you suspect misconduct contact your arbitration body. Do not go public immediately as it may very well harm your own career! So, e'er treat others as you want to be treated yourself.
If you liked this essay, y'all tin can notice more essays here, more educational material on Machine Learning hither, or have a look at my Deep Learning Lecture. I would also appreciate a follow on YouTube, Twitter, Facebook, or LinkedIn in case you desire to be informed about more essays, videos, and research in the future. This article is released under the Creative Eatables iv.0 Attribution License and can be reprinted and modified if referenced.
Source: https://towardsdatascience.com/toxic-peer-review-in-computer-science-and-what-to-do-about-it-562ed5608310
0 Response to "How to Paper Review Computer Science Technical Depth"
Post a Comment