Individuals on the path towards becoming a physician in the United States spend a great deal of time in educational settings and must pass many standardized high-stakes examinations.  These examinations may have tremendous implications for their futures, including getting into college (SAT or ACT), medical school (MCAT), as well as residency training programs (USMLE Steps 1 and 2). After they complete residency and/or fellowship training, most physicians typically seek board certification, as hospitals have increasingly implemented board certification as a required element in the credentialing process (Cassel and Reuben, 2011; Freed, Dunham and Gebremariam, 2013) and physicians without board certification may have a slightly lower earning potential (Gray et al., 2012). Board certification has also been linked to improved knowledge of current medical standards through continuous learning and generally providing better patient care (Lipner, Hess and Phillips, 2013). Additionally, failing and needing to retake board certification examinations can impact a physician’s career (Atsawarungruangkit, 2015). As such, board certification examinations are high-stakes examinations.

The American Board of Internal Medicine (ABIM) has the largest number of board-certified physicians in the United States with over 250,000 members (American Board of Medical Specialties, 2017) with over 7,000 physicians having taken the American Board of Internal Medicine Certifying Examination (ABIM-CE) for the first time in 2016 (American Board of Internal Medicine, 2017). As internal medicine residency programs are assessed on the board-exam success rate of their graduates, both by applicants and by the Accreditation Council for Graduate Medical Education, which requires programs to have at least an 80% certification examination first-time pass rate over a three-year period (Accreditation Council for Graduate Medical Education, 2017), internal medicine residency program directors must be invested in the board-exam success rate of their graduates. The best methods to prepare for the ABIM-CE are unclear but have included taking advantage of residency program educational opportunities, using internal and external board-review courses (Flannery, 2014), and most recently, using online board-review question banks.

In 2014, NEJM Group partnered with Area9 Learning (now Area9 Lyceum) to develop and launch the NEJM Knowledge+ Internal Medicine Board Review (NEJM Knowledge+) product, which supports the board certification preparation and lifelong learning for internal medicine physicians. This product features a comprehensive question bank that NEJM editors developed on the basis of an augmented ABIM blueprint. The editors wrote clinical scenarios and learning objectives as the foundation of three types of questions: (a) board-style multiple-choice questions, (b) related, shorter multiple-choice questions, and (c) related fill-in-the-blank questions (see Figure 1). Both types of multiple-choice questions include detailed feedback and citations/further resources. This comprehensive question bank uniquely incorporates adaptive-learning technology, which uses an algorithm “to observe how an individual learns and then delivers questions that are targeted to address any gaps in knowledge” (McMahon and Drazen, 2014, p. 1648). Specifically, through this algorithm, NEJM Knowledge+ takes into account the learner’s answer to a question, self-reported confidence on a four-point scale (1– “I know it”, 2 – “I think so”, 3 – “not sure”, and 4 – “no idea”), the learner’s overall confidence in the topic, the time the learner takes to indicate an answer, and how others have answered in order to determine the optimal sequencing of additional questions. In addition, the algorithm uses spaced repetition of related items mixed in with new questions until it determines that the learner has likely mastered all of the underlying learning objectives in each specialty module. Additionally, a separate “recharge” function fields question variations on items that the learner has already seen and may have forgotten by using spaced repetition based on Ebbinghaus curves and the algorithm described above. At launch, NEJM Knowledge+ included 2,477 learning objectives, 1,577 case-based questions, and 3,147 total items, including all the variants. The content has been maintained through both proactive review and a “challenge” function that allows learners to suggest improvements. New questions have been added for a current total of 2,787 learning objectives, 1,771 case-based questions, and 4,935 total items. Further, these questions are used for the basis of the two full-length practice exams that learners can access, which complements the other features such as progress reports to evaluate and measure performance.

The aims of this exploratory study were to measure the users’ ratings of NEJM Knowledge+ features and to address the relationship between the use of NEJM Knowledge+ and passing the ABIM-CE. Our primary hypotheses were that the user ratings of the features would be high and that the use of NEJM Knowledge+ would have a beneficial impact on passing the ABIM-CE.

NEJM Group developed a series of five surveys (two in 2015, one in 2016, and two in 2017), using a cross-sectional survey design intended to measure satisfaction and board passage rate of NEJM Knowledge+ users. These surveys each included over 30 questions that consisted of multiple-choice, rank-order, rating, scaled, and open-ended questions. The surveys were electronically distributed anonymously to a total of 4,400 users. Distribution occurred during the respective years and collection occurred between three and four weeks after the distribution date. Minor incentives were offered to those completing the survey in the form of Amazon gift cards in the amount of $10 in 2015, $25 in 2016, and $25 in 2017.

Twelve questions that were identical across the five surveys were analyzed for this study (see Table 1). Ten questions (questions 1-7 and 9-11) were used to develop a basic profile of the resident, fellow, and physician users that responded via descriptive statistics. Seven questions (questions 2, 5, and 8-12) were used to conduct an advanced analysis using a subgroup of responding users from the United States who primarily used NEJM Knowledge+ to prepare for an initial ABIM-CE between 2014 and 2016. The advanced analysis first consisted of examining the ratings and rankings of NEJM Knowledge+ and comparing those who passed and those who did not pass the ABIM-CE on their first attempt via conducting independent-samples t-tests using IBM SPSS 24 (Armonk, New York) and a Mann–Whitney U test using a Mann-Whitney U test calculator (http://www.socscistatistics.com/tests/mannwhitney/default2.aspx). We also examined the ABIM-CE pass rates comparing the identified subgroup of the responding users to the average published ABIM-CE national pass rates between 2014 and 2016 for first-time test takers (https://www.abim.org/~/media/ABIM%20Public/Files/pdf/statistics-data/certification-pass-rates.pdf) by conducting a z-score calculation using a z-score calculator (http://www.socscistatistics.com/tests/ztest/Default2.aspx).

Basic Profile of the Resident, Fellow, and Physician Users

A total of 836 of 4,400 (19%) users responded (2015 survey responses = 267; 2016 survey responses = 180; and 2017 survey responses = 389). Respondents were 31.2% residents/fellows (n=261) and 68.8% physicians in practice (n=575). Of the respondents, over 40 countries were represented, with the three most frequent consisting of the United States (n=651, 77.9%), Canada (n=34, 4.1%), and Australia (n=27, 3.2%). The most frequent factor that influenced the respondents’ use of NEJM Knowledge+ was the NEJM Group brand (n=224, 26.8%). Additionally, respondents most frequently completed/almost completed the program (n=293, 35%) or were using it regularly (once or twice per week) (n= 217, 26%).

The most frequent reasons for using the product included preparing for an initial ABIM certification exam (n=256, 30.6%), lifelong learning (n=219, 26.2%), or preparing for an ABIM recertification exam (n=179, 21.4%). The difficulty of the questions in NEJM Knowledge+ compared with the difficulty of the questions on the exam were rated as a lot harder (n=12, 1.4%), somewhat harder (n=88, 10.5%), just about as difficult (n=250, 29.9%), somewhat easier (n=100, 12.0%), or a lot easier (n=20, 2.4%). The majority of respondents rated NEJM Knowledge+ as either about as helpful in preparing them for the exam (n=146, 17.5%) or substantially more helpful in preparing them for the exam (n=139, 16.6%) than the best of the other resources they had used.

Respondents rated NEJM Knowledge+’s quality of content most frequently as either excellent or good (n=776, 92.8%), relevance to practice most frequently as either excellent or good (n=750, 89.7%), were most frequently very satisfied with the five product components (average percent very satisfied=74.4%), and were most often very satisfied with the ten product features (average percent very satisfied=55.5%).   

Advanced Analysis – Identified Subgroup of Responding Users

Of the 836 respondents, there were 153 (18.3%) from the United States who used NEJM Knowledge+ primarily to prepare for an initial ABIM-CE between 2014 and 2016.

As illustrated in Table 2, the average ratings of the quality of NEJM Knowledge+’s content and its relevance to practice amongst users in this subgroup were high. Although respondents who passed the ABIM-CE on their first attempt rated the quality and relevance of the content higher than those who did not pass, these differences did not reach statistical significance. 

As illustrated in Table 3, users in this subgroup of respondents rated the five components of NEJM Knowledge+ highly on a scale of 1 (very dissatisfied) to 4 (very satisfied). Those who passed on their first attempt rated the components higher than those who did not pass; however, the only difference between these groups’ ratings that reached statistical significance was in the component “detailed feedback” (p=0.002), as those who passed rated it 3.68, on average, compared with those who did not pass, who rated it 3.00, on average. 

As illustrated in Table 4, users in this subgroup of respondents also rated the ten features of NEJM Knowledge+ highly. Though there were differences between the ratings for those who passed and those who did not pass on their first attempt, the differences were not statistically significant.

The ranking of the ten features of NEJM Knowledge+ according to their value by users in this subgroup of respondents from highest (1) to lowest (10) indicated that the top three ranked features were practice exams (3.57, on average), adaptive delivery of questions (3.75, on average), and reports on progress, performance, and metacognition (4.36, on average) (see Figure 2). There were no significant differences between the rankings of those who passed on their first attempt and those who did not pass (p=NS).

Finally, during this time period (between 2014 and 2016), there was a significantly higher proportion of users in this subgroup of respondents who reported passing the ABIM-CE on their first attempt than the national pass rate average (95% vs 89%, z=2.6397, and p=0.0083).   

A physician’s time is divided amongst several areas, including clinical care, documentation through electronic medical records, and staying up to date with medical knowledge and procedures. Specifically, Leafloor et al. (2015) highlighted that internal medicine residents spend an average of 41.8% of their time on patient care activities, compared to an average of 13.8% of their time spent on educational activities. Methods to comprehensively, effectively, and efficiently prepare for high-stakes examinations are essential and can be highly beneficial. Advances in technology have led to the development of products, such as online question banks and e-learning examination-preparation products, that can potentially provide physicians with an advantage in their exam preparation.

This study highlights that the NEJM Knowledge+ product can potentially be useful for learners and that specific features, including adaptive delivery of questions, practice exams, and reports on progress, performance, and metacognition, could aid in enhancing learning efficiency and knowledge retention. These results are consistent with other research findings in the literature. While the effectiveness of adaptive e-learning environments for both health professionals and health professions students is unclear (Fontaine et al., 2017), the possibilities of adaptive learning technologies in higher education is being explored as there is much agreement regarding its beneficial value (Liu et al., 2017) when applied thoughtfully and in a manner consistent with learning needs. Additionally, retrieval practice by taking periodic practice exams or quizzes and reviewing the results appears to enhance overall medical knowledge retention and may potentially provide effective direction for continued study and preparation (Wagner et al., 2016). Further, the application of metacognition allows learners to effectively think critically, direct their own learning, and self-assess, which are important skills in lifelong learning and increasingly emphasized by health-professional organizations and accreditation bodies (Medina, Castleberry and Persky, 2017).

In addition, this study highlights that the use of NEJM Knowledge+ may have been a contributing factor for users in the identified subgroup of respondents on passing the ABIM-CE. As such, program directors, faculty members, and educators can benefit by being aware of the potential educational value of e-learning examination-preparation products to provide another avenue of support for their residents and junior colleagues in successfully attaining initial board certification.

Finally, the design of this exploratory study had many strengths, including the anonymity of the users who responded to the survey, numerical data utilized for statistical analysis providing researcher objectivity, and the ability to develop an identified subgroup of respondents for advanced analysis. There were some limitations, including a low response rate, which is a common problem in surveys of health care professionals and affects the generalization of these results. Additional limitations included that other factors related to user preparation for the ABIM-CE were not examined in this study and users self-reporting their ABIM-CE result. Future research includes examining which specific adaptive learning features and user metrics (e.g., time of login, duration of session use, duration of product use, etc.) related to NEJM Knowledge+ ultimately increase overall knowledge retention.

This exploratory study highlighted that an e-learning examination-preparation product featuring adaptive delivery of questions, effective feedback, and analysis of overall metacognition can potentially be an effective tool for first-time test takers preparing for initial certifying examinations. Thus, the ability for residents, fellows, and physicians to utilize products such as NEJM Knowledge+ as learning tools in this high-stakes-examination era can potentially be of high educational value. Finally, further research should be undertaken regarding e-learning examination-preparation products in other medical and allied-health specialties.