Practical tips and/or guidelines
Open Access

Practical Considerations for Online Open Book Examinations in Remote Settings [Version 2]

Hui Meng Er[1], Vishna Devi Nadarajah[1], Pei Se Wong[1], Nilesh Kumar Mitra[1], Zabibah Ibrahim[1]

Institution: 1. International Medical University, Malaysia
Corresponding Author: Prof Hui Meng Er ([email protected])
Categories: Assessment
Published Date: 19/03/2021


The COVID-19 outbreak has led to lockdown of cities and restricted access to university campuses, and hence face-to-face delivery of education has been disrupted worldwide. In order to continue teaching, learning and assessment activities, academic institutions have embarked on online delivery and assessments using technology. Online open book examination is one of the tools considered during the crisis period to ensure that students’ progression in the academic programmes and graduation are not delayed. Its use is supported by literature evidences that show promotion of critical thinking and problem solving skills amongst students. The positive findings from our previous study on the impact of open book examinations on student performance and learning approach have encouraged us to implement online open book examinations in various health professional programmes in our institution during the COVID-19 pandemic. While some successful progress has been made, there are areas that need further exploration to provide detailed insights on the practice and effectiveness of remote online open book examinations. The objective of this paper is to share the practical tips for implementing online open book examinations remotely, in order to ensure the validity, reliability and fairness of the examinations.

Keywords: Online examination; Open book examination; Remote; Health professional programmes

Author Revision Notes

We would like to express our sincere appreciation to the reviewers for the valuable comments. We have revised the manuscript and the amendments are as below:
1. The abstract has been revised to summarise the main points in the paper.
2. The IMU faculty guide on online open book assessment is made available online for the readers. It consists of information about the principles and considerations for remote online open book examinations.
3. The IMU students guides on online open book assessment and online invigilation are made available online for the readers. They provide guidance to the students on preparation for online open book examinations as well as online invigilation.
4. A section on limitations has been added in the paper to discuss the areas that require further studies, particular to assure the quality of online open book examinations. A few studies are currently being undertaken at our institution to gain better understanding on the quality, reliability and validity of remote online open book examinations.


The COVID-19 pandemic has resulted in lockdown of cities globally as public health measures to mitigate the spread of the coronavirus. Technology has rapidly been deployed by institutions to enable continuation of education delivery. This has accelerated the buy-in of online learning among the educators and students. Meanwhile, much attention has also been focussed on online assessments, particularly the platforms, tools and formats that can serve the assessment purpose and address its utility, i.e. validity, reliability, educational impact, acceptability and cost (Van der Vleuten, 1996).

While closed book examinations promote students’ test preparation and deep learning (Block, 2012; Durning, Dong, Ratcliffe, Schuwirth et al., 2016), open book examinations enhance their critical thinking and creative problem solving skills in an environment that simulates the real working scenario (Johanns, Dinkens and Moore, 2017). In addition, open book assessments help students to develop competency in knowledge management (Rowlands and Forsythe, 2006). Despite the availability of various online proctoring tools in the market, many have cautioned against their use in view of data privacy and confidentiality (Thompson, 2020). Besides, some students may not have access to webcam that is required for online proctoring. These limit the conduct of closed book examinations remotely. Therefore, online open book examinations have been adopted by many academic institutions during this unprecedented period when campus access is restricted. The objective of this paper is to discuss the practical considerations for implementation of online open book examinations, based on our experience in conducting these examinations in various health professional programmes at our institution. 

Institutional Context and Programme Needs Analysis

The decision to convert conventional examinations to online open book examinations should be made based on the institutional context and educational programme needs. These include the availability of a secure online assessment platform, administrative and logistic capacity of the institution, academic calendar, student assessment load as well as post-lockdown modifications to campus activities. As the COVID-19 situation continues to evolve, there is much uncertainty over the lockdown duration. Moreover, it is anticipated that conventional teaching, learning and assessment activities need to be modified post-lockdown in order to ensure physical distancing in the campus. Postponing the examinations to after lockdown is lifted will inevitably lead to increased workload of the examination administrators and logistic burden in scheduling assessment activities that may exceed the university capacity (Figure 1). Examinations may need to be conducted in smaller groups and hence higher number of examination venues, extended examination schedules and quarantine procedures are necessary. It will also likely to delay students’ graduation and entry into the healthcare workforce. Assessment overload on the students is another consequence that could contribute to stress among the students. These implications have to be deliberated realistically against the challenges faced in converting to online open book examinations within a short period of time.

Figure 1: Managing the impact of examination postponement and post-lockdown modifications to campus activities

Figure 1 is based on the “Coronavirus Flattening the Curve PowerPoint Template” from The blue curve represents the consequence of postponement of examinations, while the green curve represents the effect of intervention through conduct of online open book examinations during and post-lockdown. 

Faculty Readiness

In open book examinations, students are allowed to refer to textbooks, online resources, or reference materials during the examinations (Ramamurthy, Er, Nadarajah and Pook, 2016). Faculty training is necessary to ensure that the examinations are designed to test higher order thinking skills, e.g. applying, analysing, evaluating, and creating with resources (Brightwell, Daniel and Stewart, 2004). Other skills can also be tested, such as locating information, using computer applications and conducting internet research using real world scenarios (Johanns, Dinkens and Moore, 2017). These skills are indeed essential for healthcare professionals. However, some faculty may face challenges in preparing these types of questions within a short time, particularly in situations where clinical scenario or application based teaching is limited. To support this transition, university guidelines, webinars and personalised online faculty mentoring clinics for setting online open book examinations are highly recommended. An example of faculty guide on online open book assessment is available online for the readers (IMU, 2020a). The guide describes the principles and considerations for remote online open book examinations. On the other hand, it is important to recognise the limitations in assessing clinical and procedural skills using online open book examinations. Creativity makes it possible to assess some of these competencies, for example, by getting the students to demonstrate the skills via live streaming in anticipation that some modifications may be required.

Students’ Readiness

Students have been reported to spend less time to prepare for open book examinations compared to closed book examinations due to misconception that the answers could be found in books (Durning, Don, Ratcliffe, Schuwirth et al., 2016). As a consequence of inadequate preparation, some spent more time looking for answers rather than producing a quality answer (Block, 2012). These findings have highlighted the importance of communicating to the students what online open book examinations are about and familiarising them with higher order thinking questions during teaching and formative assessments. Besides, the students should be informed in advance of the rules and regulations for online open book examinations including the consequences of academic misconduct. An example of student guide on online open book assessment is available online for the readers (IMU, 2020b). It guides the students on preparation for online open book examinations. Additionally, students are also provided with information about the preparation and process for online examinations with online invigilation (IMU, 2020c).

Online open book examination relies on students’ computer hardware, software and internet connection during the examination. Evidences have shown that students could experience a greater cognitive load during online examinations compared to traditional paper-based examinations, as a result of having to deal with navigating technology in addition to answering the examination questions (Cramp, 2019). This could lead to stress and anxiety. Hence, practice examinations are crucial for students to test their login to the online assessment platform, familiarise with the features of the platform and have practice opportunities with questions that require them to draw diagrams or plot graphs, either using computer software or by hand, followed by uploading to the answers for submission. Technical support should be available before and during the examinations. For example, if a student is disconnected from the online assessment platform during the examination due to internet connection problems, the technical support personnel can check the examination time remaining and the logs. Based on these reports, the examination coordinator can advise on the extra time allocation if needed. In situations where students face intermittent internet access, the asynchronous online mode should be considered, for example, by allowing the students to download the examination questions, work on the answers offline (without having to depend on internet connection) and submit the answers within the specified duration.

Quality Assurance

It is important that the decision to convert conventional examinations to online open book examinations is made in compliance with the institutional governance in order to uphold the credibility and integrity of the academic programmes. With programme and course learning outcomes being the cornerstone, the design of online open book examinations should be guided by the assessment blueprint. Vetting of examination questions are particularly critical to ensure the validity, reliability, fairness and integrity of the examinations. Besides verifying the content, construct and concurrent validities, standard setting and psychometric analysis should continue to be practised for quality assurance. In addition, policies and standard operating procedures are required for the entire spectrum of work processes from preparing examination questions, vetting, uploading questions to the online assessment platform, authenticating test candidates, test taking, marking, to results processing, reporting and publishing. Not only will these ensure the examination efficiency, they safeguard the examination security which is a key concern since most of these activities are conducted online remotely. Besides the professional support staff of the examination office, technical staff from the Information Technology (IT) and E-Learning departments will also be involved in setting up the online examinations and providing technical assistance remotely during the examinations. Hence, the roles and responsibilities among the stakeholders should be clearly defined. Timely feedback from the stakeholders is crucial for continuous improvement.

Adaptability and Agility

In preparation for the COVID-19 crisis, universities would have developed various business continuity plans (BCPs) to minimise impact to operations and ensure that teaching and learning activities remain viable, taking into consideration of personal and community safety. BCPs in this context will normally include plans to transition to online delivery and assessments, which are justified by transition risk assessment, needs prioritisation, resource readiness, communication strategies while maintaining academic standards and governance. Most of these BCPs are prepared at the institutional level, but they need to be contextualised to individual programme requirements to enable autonomy according to the varying context and communities they operate in. On the other hand, decisions related to public health and safety are usually made at the national level, resulting in varying external factors that are beyond the institution’s control. Nevertheless, the roll out of BCPs can be challenging in a constantly evolving situation like the COVID-19 pandemic. Frequent changes of plans are unavoidable and these can affect all stakeholders including the faculty, professional support staff and students. Reflecting on our experience, it can be concluded that while BCPs are useful guides, adaptability and agility among the stakeholders are crucial in a crisis situation. Factors that help to reduce the stress level of the stakeholders include data accessibility to guide decision making, preparedness of faculty for online delivery and assessments, availability of faculty development and mentoring, readiness and upskilling of professional support staff, as well as open and transparent communication and collaboration among all staff in a collegial manner with the common goal of ensuring positive student experiences during the challenging time.


The measures discussed in this paper have helped our institution to be better prepared for remote online open book examinations and ensured continuity of assessments during the COVID-19 pandemic period. Nevertheless, some faculty members argue that this format may not be appropriate for all learning outcomes. It could be more suitable for senior students who have more exposure to real world scenarios compared to the students in the earlier academic years. The potential of misalignment between curriculum delivery and assessments as well as access to resources during online open book examinations could affect the student performance. Debates remain on whether the same standard setting practice for closed book examinations can be applied for open book examinations. A study is currently being undertaken to explore the faculty perception on these areas. More evidences are necessary to support the validity and reliability of remote online open book examinations, under invigilated and non-invigilated environments. The impact of remote online open book assessment on students’ learning and performance is being investigated. 


Online open book examination is a viable and relevant option for health professions education at times of uncertainty. The COVID-19 pandemic has increased the utility of this tool among the higher education institutions. However, it is important to have a systematic and collaborative approach for its implementation based on good practices, standards and evidences. Continuous studies on its validity, reliability and impact on learners should be carried out. The findings will help health professional educators to improve its implementation in order to ensure that the graduates are competent and work ready in an increasingly challenging healthcare environment.

Take Home Messages

  • The decision to convert conventional examinations to online open book examinations should be made based on the institutional context and educational programme needs.
  • Faculty training and support for setting online open book examinations are highly recommended.
  • Students should be familiarised with the conduct of online open book examinations through practice examinations.
  • Quality assurance is important to ensure the validity, reliability and fairness of online open book examinations.
  • Adaptability and agility among the stakeholders of online open book examinations are crucial in a crisis situation.

Notes On Contributors

Professor Hui Meng Er is the Acting Dean of Teaching and Learning at the International Medical University, Malaysia. 


Professor Vishna Devi Nadarajah is the Pro Vice Chancellor, Education and Institutional Development, at the International Medical University, Malaysia. 


Dr Pei Se Wong is the Associate Dean of Teaching and Learning at the International Medical University, Malaysia. 


Nilesh Kumar Mitra is the Acting Director of Learning Resources at the International Medical University, Malaysia. 


Ms Zabibah Ibrahim is the Assistant Manager of E-Learning at the International Medical University, Malaysia. 



The authors would like to acknowledge that Figure 1 is based on the “Coronavirus Flattening the Curve PowerPoint Template” from

The authors would also like to thank the staff of the Departments of E-Learning, IT and Examination Unit at the International Medical University for the useful feedback on the conduct of online open book examinations. 


Block, R. M. (2012) ‘A discussion of the effect of open-book and closed-book exams on student achievement in an introductory statistics course’, PRIMUS, 22 (3), pp. 228-238.

Brightwell, R., Daniel, J-H. and Stewart, A. (2004) ‘Evaluation: Is an open book examination easier?’, Bioscience Education, 3(1), pp.1–10.

Cramp, J., Medlin, J. F., Lake, P. and Sharp, C. (2019) ‘Lessons learned from implementing remotely invigilated online exams’, Journal of University Teaching & Learning Practice, 16(1), Article 10. Available at: (Accessed 15 May 2020).

Durning, S., Dong, T., Ratcliffe, T., Schuwirth, L. et al. (2016) ‘Comparing open-book and closed-book examinations: A systematic review’, Academic Medicine, 91, pp. 583-599.

International Medical University (IMU) (2020a) ‘IMU faculty guide on online open book assessment’. Available at: (Accessed: 23 Nov 2020).

International Medical University (IMU) (2020b) ‘IMU student guide on online open book assessment’. Available at: (Accessed: 23 Nov 2020).

International Medical University (IMU) (2020c) ‘IMU student guide on online examination with online invigilation’. Available at: (Accessed: 23 Nov 2020).

Johanns, B., Dinkens, A. and Moore, J. (2017) ‘A systematic review comparing open-book and closed-book examinations: Evaluating effects on development of critical thinking’, Nurse Education in Practice, 27, pp. 89-94.

Ramamurthy, S., Er, H. M., Nadarajah, V. D. and Pook, P. C. K. (2016) ‘Study on the impact of open and closed book formative examinations on pharmacy students’ performance, perception, and learning approach’, Currents in Pharmacy Teaching and Learning, 8, pp. 364-374.

Rowlands, J. E. and Forsythe, D. (2006) ‘Open-book professional accountancy examinations’, South African Journal of Higher Education, 20(5), pp. 703–717.

Thompson, M. (2020) UCSB Faculty association issues letter advising against the use of ProctorU testing services. Available at: (Accessed: 15 May 2020).

Van Der Vleuten, C. P. M. (1996) ‘The assessment of professional competence: Developments, research and practical implications’, Advances in Health Sciences Education, 1, pp. 41-67.




There are no conflicts of interest.
This has been published under Creative Commons "CC BY-SA 4.0" (

Ethics Statement

The practical tips have been developed based on our experience in implementing online open book examinations. Ethics approval is not required.

External Funding

This article has not had any External Funding


Please Login or Register an Account before submitting a Review

Gominda Ponnamperuma - (19/04/2021) Panel Member Icon
This article provides useful practical information for any institute that plans to implement online learning. Out of all the sections, I particularly like the sections on quality assurance and student readiness. The open book assessment guides for faculty would also be quite useful for anyone planning to conduct open-book assessment. If the authors plan to revise this article further, I could offer the following suggestions.
1. Since the objectives in the abstract and the main article slightly differ, it would be good to repeat the objective stated in the abstract (in exactly the same way) in the main article. This would ensure that both objectives contain 'validity, reliability and fairness'.
2. The authors could develop a summary table that meshes the tips in relation to each of the sections (e.g. faculty readiness, student readiness, quality assurance, etc.) in rows against the assessment properties (e.g. validity, reliability, fairness) in columns.
Ken Masters - (19/03/2021) Panel Member Icon
I am pleased to see that the authors have made some attempt at addressing the issues raised in response to the first version, but the changes are not great. Although a Limitations section has been added, many of the other changes appear to be minor. For example, in the Abstract, apart from the addition of a single sentence (“While some successful progress…”), there appears to have been no change made, and it remains a lead-in to the paper rather than a summary of the paper.

Almost the entire rest of the paper is identical with only minor alterations and insertions.

So, while the paper’s value has been somewhat increased, I'm afraid it is rather a missed opportunity.

Possible Conflict of Interest:

For transparency, I am an Associate Editor of MedEdPublish.