Commentary
Open Access

How to review a paper on medical education

Richard Hays[1], Barbara Jennings[2], Trevor Gibbs[3]

Institution: 1. James Cook University and University of Tasmania, 2. University of East Anglia, 3. Association of Medical Education in Europe
Corresponding Author: Prof Richard Hays ([email protected])
Categories: Scholarship/Publishing
Published Date: 29/07/2019

Abstract

There has been a substantial increase in the number of medical and health professional education manuscripts being submitted to an increasing number of journals in this field.  More reviews and more reviewers are needed to facilitate discussion of both relevance and quality of those manuscripts.  MedEdPublish relies on readers and Review Panel members to contribute to this process, thereby helping to maintain standards in medical and health professional education publishing.  This article provides guidance that is most relevant to reviewers and potential authors for MedEdPublish, but may be relevant to publishing in other medical and health professional journals.

Keywords: medical education; publishing; guidance for reviewers.

Introduction

One of the major challenges facing academic publishing is that manuscript reviews are increasingly difficult to source.  Although the number of academic journals in medical education has increased over the last 15 years or so, the number of manuscripts submitted has increased even more.  Accordingly, the demand for peer review of manuscripts has also increased dramatically.  In response, many journals now reject a substantial proportion of manuscripts without the benefit of peer review.  Anecdotally, editors often invite 6-7 reviewers to receive 1-2 reviews.  Those known to do reviews are being asked more frequently.  For example, these authors often receive at least one review request each day, which is very difficult to fit into busy lives.

This situation threatens one of the underlying principles of academic endeavor because peer review is regarded as the most effective way to maintain academic standards.  While MedEdPublish is a post-publication peer review online journal, with a high proportion of submitted manuscripts published relatively rapidly without the potential for some causes of publication bias (Hays, 2016), the journal depends upon reviews of published articles from both our Review Panel members and the general readership.  The discussion threads that follow amongst our community of practice maintain both standards and relevance.  More reviewers and more reviews are needed to allow the journal to continue to meet its objectives.  

This article aims to assist two audiences. The first is the members of our Review Panel, because feedback has suggested that some reviewers feel uncomfortable attaching their names to open reviews that may be critical.  The second audience is the general readership, because MedEdPublish genuinely wants more readers to write reviews and contribute to discussions about ideas and research findings.  Writing reviews may also be a valuable learning experience.  Reviewing is, arguably, the most effective way of keeping abreast of developments and research in our community of practice, although it can be resource intensive. The review process for MedEdPublish is a little different to reviewing for more traditional medical journals, so here is some guidance.  Although the guidance is particularly suited to MedEdPublish, it is likely to be suitable for other medical and health professional education journals.  Further, the content may be of interest to potential authors, as it describes features that may receive better reviews.

Performing the review

The time required is about 30-60 minutes per manuscript.  If the topic is in your area of expertise, writing a review may be quicker, although it may be tempting to offer too much advice, or perhaps even re-write the paper!   The potential for conflict of interest should, if anything, be over-declared, allowing the editors and readers to decide.  The medical education community is small, so any links through institutions, committees and research interests are possible conflicts.  If not in your area of expertise, a ‘generalist’ view is still often very useful, but this should be stated in the review.  Similarly, reviews by the general readership, particularly students and recent graduates, are welcome, but this background should be stated to provide the context of your perspective.  For less experienced reviewers, a systematic approach to the analysis of a paper is helpful. One evidence-based-medicine textbook that has excellent checklists for the appraisal of research papers (not just medical education) is ‘How to Read a Paper’ (Greenhalgh, 2010).  

It is worth asking the question: is there a difference between reviews and comments?  The answer is yes, although there may be some overlap.  Comments may be as short as a few words or a sentence and about only a single issue, whereas reviews should be a more detailed, comprehensive and structured analysis, with a combination of positive feedback and suggestions for improvement. Different kinds of papers may warrant different responses.  For example, personal opinion manuscripts, editorials and commentaries may better suit comments rather than reviews. Reviews are welcome on all categories of manuscripts, but are essential for those presenting research, evaluation or new educational methods, where data are analysed and interpreted. 

Step 1. Read the manuscript through from beginning to end, then pause to consider it as a whole

Scope.  Is the manuscript about medical education?  Sometimes authors submit manuscripts that are more about health care or healthcare education of the public.  While these may be interesting, they may fit better into other journals.  Increasingly, some people submit out-of-scope manuscripts to either gain a publication falsely or to demonstrate that peer review is flawed.  The editors will probably forward only those that are within scope, but sometimes this is a difficult judgement.  Does this appear to be an original contribution, or have you read previously something either similar or based on the same data?  ‘Salami slicing’ refers to the dividing-up of data from a large study into two or more smaller studies, with same question and population described, without transparency, in more than one publication. This is not uncommon, particularly for studies involving large surveys.  Reviewers should alert editors if this is suspected.  Fraudulent research with fabricated data is rare in medical education, perhaps because this field is less likely to be a battlefield of grants, high citation index journals and institutional reputations, but reviewers should be mindful of the possibility.  Is the manuscript reasonably current?  Sometimes a paper is written or based on data collected several years ago, which may be of limited value now.  If particular expertise may be helpful for highly specialized or rather obscure topics, suggest this in the review.

Ethics approval.  This is essential for any manuscripts that report data collected from patients or learners, indeed any situation where there could be perceived to be a power imbalance or potential for harm to participants.  Ethics approval details (Ethics Committee name and approval number) should be provided.  Some medical education evaluations are exempt from ethics approval processes because they are linked to quality assurance projects, but these approvals may not cover external publication.  Evidence of exemption should be provided, along with a statement that the principles of the Declaration of Helsinki were followed (World Medical Association, 2013).

Writing skills.  Does the text make sense?  Is there a clear finding or message?  Ideally, a manuscript ‘tells a story’, albeit in a concise, structured manner.  There should be a logical flow of ideas.  Is the grammar, language and syntax correct?  These can be particular challenges for authors writing in a second or third language, but are not restricted to this group.  Auto-correct functions are often used but these can be ‘false friends’ that do not always get things right.  Is the length appropriate for the message? The art of clear, concise writing may not come naturally, as novice authors often write as they would speak.  This can result in a discursive, lengthy manuscript that does not hold a reader’s interest.  Longer manuscripts are harder to review and often attract fewer reviews.  Some journals publish word limits for certain kinds of manuscripts.  Qualitative research reports often require more space to present results and interpretation.  Reviewers should not correct pedantically minor errors of spelling or grammar, but add a statement that there are several errors that should be fixed. 

Authorship.  Is it clear what each author contributed?  Conventions vary, but ideally all authors should have contributed to most parts of the underlying project and resulting paper.  Contributions should be stated.

Step 2. Read the manuscript again, but this time more strategically and focus on sections

Comment where possible on each section and take notes.

Title.  Does the title encapsulate the message? Some readers read further only if the title fits their interests, so these few words are important. For research and evaluation manuscripts, titles should define the study design.  Better titles help readers choose which abstracts to read.

Abstract.  Does the abstract summarise adequately the manuscript?  Many readers will not read complete articles, so it is important that the purpose, methods and outcomes are clear.  Structured abstracts make this easier for research papers. 

Introduction/background.  Does the introduction/background section provide sufficient context to ‘set the scene’? This section should explain where the author began the journey, why this study is necessary and why this manuscript should be read.

Methods.  This section is likely to be present only in reports of research or evaluation papers, where it is essential.  Are the methods clear, grounded in theory and appropriate for the question? Quantitative research includes many different methods.  The two most common are quasi-experimental designs (where attempts are made to reduce the number of dependent variables) and non-parametric approaches (because interventions are often applied to small populations that do not reflect a ‘normal’ curve).  If in any doubt, seek or recommend opinion from a statistician.  Qualitative research also includes many methods, but is very useful for seeking understanding of complex issues.  There should be sufficient separation between researchers and the information gathering or analysis to minimize the potential for perceived bias.  If in any doubt, seek or recommend an opinion from a qualitative researcher.  The term ‘mixed methods’ is currently popular, but this is a rather lazy term that needs to be justified by explanation of why each method was chosen to address each part of the usually complex question.  Manuscripts are often mostly descriptive, without complex analyses, but are still of value in medical education as similar initiatives are often applied in very different contexts.  

Results.  Are the results clearly presented, at least initially free of interpretation?  The facts should speak for themselves.  Ideally, tables and figures should summarise and augment the text, but the text should clearly describe results without the tables and figures.  There should not be more than three tables or figures, which should summarise the key findings.  If more space is required to provide a more complete view of results, these can be added as an appendix.  

Interpretation and Discussion.  These sections are usually separate for quantitative studies and often combined for qualitative studies.  Are the findings supported by the results?  Over-interpretation is not uncommon, particularly for novice researchers.  Not all quantitative designs can clearly attribute causation.  Evaluations of educational activities are often low level (satisfaction) and short term (before and just after), where impacts can be measured more easily, but may not be worth reporting.  Demonstrating longer term impact on practice is harder, but more valuable.  Ideally, evaluation of impacts on health care practices and outcomes are attempted, but this can be very difficult in education research.  While quantitative studies may produce results that may be applicable in other contexts, qualitative research is best at identifying influencing factors and enhancing understanding, but cannot attribute causation or association or be extrapolated easily to other contexts.  Are there implications for other education programs?  What research questions have been addressed and what further questions have been identified?  Are recommendations for change in practice supported by the analysis?  Descriptive manuscripts are just descriptions without much analysis or interpretation, but can still inform or inspire others.  Manuscripts reporting ‘negative’ findings may be very important, as others can learn from knowing what did not work.

Conclusion/summary.  All manuscripts should end with a ‘so what’ statement.  For research and evaluation reports, this could be a single paragraph that summarises why the study was done, the findings and what they might mean.  This can be similar to the abstract, only shorter and less focused on methods.  This is not the right place for discussion or justification of results and their implications.

References.  Are the references current, comprehensive and accurate? It is worthwhile checking briefly some references at random by pasting them into a search engine.  While ‘seminal’ references may be older, most references should be from the last 5 years or so and authors should not self-cite too often.  Citations of original research papers are better than other papers that cite (and perhaps misinterpret) original research.  Researchers sometimes observe that their research is cited to support something different from what the research found!  Are any key references missing?  Literature searches do not always identify every relevant paper; much depends on the searcher and the search engine.

Limitations.  Are limitations adequately presented and discussed?  Small studies may not have sufficient scale, sampling may have been restricted or unrepresentative, or specific contextual factors may affect validity; these should be mentioned.

Step 3. Write the review

Most journals provide a framework with a series of text boxes to complete.  MedEdPublish does not, allowing more freedom.  It is safer to write the review in a word processing program, save it and then cut and paste into boxes, as directly typed text may be lost if there is a software glitch.  Reviews should be polite, constructive and helpful.  They are visible to all readers, as is the name of the reviewer.  Because of the openness, it is best to think of the review as advice to a colleague.  For each section, think of things that have been done well and things that could be done to make this a better manuscript if revised.  The ‘feedback sandwich’ concept is relevant here (Cantillon & Sargeant, 2008).  The aim is to help authors and readers learn from the experience of publishing.  Sometimes reviewers and authors correspond within the MedEdPublish website, creating a discussion thread that all readers can observe, enjoy and learn from. It is important to state what changes would earn a higher star rating, because the same reviewer is often asked, and should agree, to review a revision.

Step 4. Provide the star rating

MedEdPublish and some other journals have a star rating.  This is an overall, ‘global’ rating that takes everything into account, including relevance, interest and practicality, as well as methodological issues.  The MedEdPublish website describes the attributes for each level and this should be followed.

Summary

This article provides a guidance for conducting reviews of medical education manuscripts, with a focus on reports of research, evaluations and innovations.  More concise comments, rather than formal reviews, may be more appropriate for other types of articles.  Both reviews and comments can generate interactive discussion threads that contribute to learning.  With the growth in submitted manuscripts and medical education journals, many more reviewers and reviews are needed to support the medical education community of practice.  For authors, there may also be key messages for how to write a better manuscript on medical and health professions education.

Take Home Messages

  • Reviewing medical and health professional education manuscripts is similar to reviewing in other fields, but there are some differences.
  • Reviewing is an excellent way of keeping abreast of developments.
  • The open discussion threads can provide valuable learning for authors, reviewers and readers.
  • Potential authors may gain some tips for writing better papers.

Notes On Contributors

Richard Hays is the Editor of MedEdPublish. ORCID: https://orcid.org/0000-0002-3875-3134

Barbara Jennings is an Associate Editor of MedEdPublish. ORCID: https://orcid.org/0000-0003-3792-9182

Trevor Gibbs is an Associate Editor of MedEdPublish. ORCID: https://orcid.org/0000-0002-1776-6518

Acknowledgements

The authors would like to thank the editorial team at MedEdPublish for their helpful advice in preparaing this manuscript.

Bibliography/References

Cantillon P, Sargeant J. (2008) 'Giving feedback in clinical settings',  British Medical Journal, 337. https://doi.org/10.1136/bmj.a1961 

Greenhalgh, T. (2010). How to Read a Paper: The Basics of Evidence-Based Medicine (4th ed.). Chichester, West Sussex, UK; Hoboken, NJ: Wiley-Blackwell.

Hays R, B. (2016). 'A new outlet for medical education scholarship', MedEdPublish, 5, [1], 1. https://doi.org/10.15694/mep.2016.000003

World Medical Association. (2013) The Declaration of Helsinki. Available at: https://www.wma.net/policies-post/wma-declaration-of-helsinki-ethical-principles-for-medical-research-involving-human-subjects/

Appendices

None.

Declarations

There are some conflicts of interest:
The authors are all Editors of MedEdPublish.
This has been published under Creative Commons "CC BY-SA 4.0" (https://creativecommons.org/licenses/by-sa/4.0/)

Ethics Statement

Ethics approval was not required because this article reflects the experiences and opinions of the authors.

External Funding

This article has not had any External Funding

Reviews

Please Login or Register an Account before submitting a Review

Daniel Coyle - (17/08/2019)
/
This article is a fresh take on the importance of reviewing medical education papers in medical literature. The abstract delivers an excellent and straight forward overview of the papers main information.

The introduction contains concise background information for reviewers looking to improve or awaken their research reviewing skills. It provides great context for the main body of the paper. The major objective of the article is clear in the "Take Home Messages" section.

The step by step guide outlined is very useful. The inclusion of ethics in the reviewing process is particularly relevant and important in the medical world today. The main point the authors make is that reviewers should try to target every aspect of their articles.

With an ever-growing surplus of medical education papers I will certainly be implementing the above mentioned tips and suggestions in future reviewing activity.
Olle ten Cate - (13/08/2019) Panel Member Icon
/
• This is in interesting contribution to the field, reminding about something that the Dutch call "Droste-effect": a Droste cocoa box showing a nurse carrying a tray with hot chocolate and a Droste cocoa box showing a nurse carrying a tray with hot chocolate and a Droste cocoa box showing... etcetera. This is a paper to be reviewed instructs potential reviewers how to review a paper. It merges a journal article with instructions for reviewers for this same journal in which it is published soliciting post publication reviews.

• For inexperienced reviewers is it a good guideline, but to be used flexibly, as everyone develops their own routines in reviewing. I personally like to print the paper and directly jot comments in the margins while doing the initial read (reflecting how a reader would experience the text), at variance with the authors' recommendation.

• What I missed in this paper is how to respond to the first question that arises when one receives a review request, that is: Accept or decline? That leads to the consideration What is a reasonable number of papers to review? Rule of thumb I like is: review at least the same number of peer reviewed papers one published per year, times the average number of reviews a manuscript has (usually around 3), divided by the average number of co-authors of one's own papers. So, if you publish 4 papers per year, on average with 4 co-authors , a reasonable number of reviews to do per year is (4*3)/4 = 3 papers to review per year at least, i.e. every 3-4 months one review, to keep the system as a whole sustainable.
• "Reviewers should not correct pedantically minor errors of spelling or grammar, but add a statement that there are several errors that should be fixed" - Hmm. I'm not sure. In fact those small suggestions are easily processed, and authors with less language proficiency are very pleased to be pointed at them. This touches a more fundamental question: is a review just meant to support an editorial decisions (then authors of this paper are right) or also to support the manuscript authors in revisions. I think both.

• A smaller comment regards the expected sections of a manuscript to be reviewed. I would add that the Introduction should should end with a research question or equivalent.

Two comments I have specifically regard reviewing for MedEpPublish.

• Most journals help reviewers and responding authors by providing line numbering. Reviewing for MedEdPublish has the disadvantage that there are no line numbers to refer to in comments, not even page numbers

• Not clear what to do if the reviewer judges that the paper should not be published. I recently reacted to a journal "I would be very concerned if this manuscript would be published". Fortunately the current manuscript is great and only requires minor revisions. I hope it will happen! Despite the fact that I did not follow all instructions for reviewers for this journal.

Balakrishnan(Kichu) Nair - (08/08/2019) Panel Member Icon
/
I found this article very useful. To check its utility , I just used this step by step process, to review an article for another journal and am glad to report it worked really well.

As the authors mention, peer review is a challenging process because of the increasing number of journals and the demand on reviewers. Traditionally journals had a single or double blinded peer reviewing process with its own issues. .So the open process is very transparent and timely . This will avoid the power imbalance and lack of transparency

This clear and brief paper will help new reviewers to get involved in the journal .

Possible Conflict of Interest:

I am member of the reviewing panel , but have no conflict of interest

Janie Dade Smith - (08/08/2019)
/
I found this paper really useful in preparing a paper for publication and recommend it to others. It is clearly written, informative and helpful.
Possible Conflict of Interest:

Richard Hays is a colleague of mine