There have been an increasing number of studies on medical professionalism since the 1990s(Smith, 2005). Traditionally, professionals such as physicians and attorneys were considered professionals based on the social processes through which the professions were established. However, it has been criticised that physicians are not sufficiently fulfilling the social contracts(Hafferty, 2009), and physicians are consequently expected to cultivate higher levels of professionalism.

There are some definitions of professionalism. Stern and Arnold(Stern, 2006) stated that “professionalism is demonstrated through a foundation of clinical competence, communication skills, and ethical and legal understanding, upon which is built the aspiration to and wise application of the principles of professionalism”. The Medical Professionalism Project also defined three fundamental principles and ten professional responsibilities of professionalism as the Physician Charter (ABIMFoundation et al., 2002). A comparison between the concepts of professionalism by Arnold and Stern(Stern, 2006) and the Physician Charter(ABIMFoundation et al., 2002)  results in many common and divergent aspects. Therefore, the development and assessment of programmes that foster medical professionalism first requires defining medical professionalism and then measuring it.

The most commonly used instrument to measure professionalism is that created by Arnold et al.(Arnold et al., 1998) based on the definitions of professionalism given by the American Board of Internal Medicine (ABIM) in 1995, and several studies that revised and used this instrument. However, the results of factor analysis showed that there were some items with higher factor loading among several factors, and a reliability assessment showed that some subscale items had a coefficient alpha of less than 0.6. As Arnold et al.(Arnold et al., 1998) suggested the necessity of adding questionnaire items, this scale lacks reliability and validity.

The concept of professionalism has changed over time, from how the ABIM defined the concept of professionalism in 1995 to that published in the Physician Charter in 2002. The original six elements of professionalism presented in 1995 are no longer adequate for describing the concept of professionalism. Many other scales and instruments for measuring and evaluating professionalism of physicians have been developed, but there are many that have not been tested for construct validity.

This study seeks to develop a medical professionalism assessment scale for evaluating how medical professionalism is taught and fostered in students. Of the presumed steps to acquire medical professionalism, this study defines the stage before starting clinical practice as Level 1 and the stage at the time of graduation as Level 2, and develops two scales matched to the students' level of readiness. Development of these scales will contribute to objective evaluation of the outcomes of education in medical professionalism.

The following procedures were undertaken for developing an assessment scale of medical professionalism. First, the concepts of medical professionalism is extracted. Second, an item pool will be created based on the concepts extracted as a source for the scale items. Third, a pre-test will be conducted to develop the Level 1 scale for assessment readiness in pre-clinical students. Fourth, the main survey for developing the Level 1 scale for assessment of readiness in pre-clinical students will be conducted. Fifth, the Level 2 scale adapted for assessing the readiness of students at the time of graduation will be developed.

Step 1: Extracting the concepts of medical professionalism

A close investigation was performed on various documents (Table 1) describing medical professionalism of physicians, nurses and physiotherapists as presented or published by associations of each profession in the four countries of Japan, the United States, United Kingdom and Canada. Content corresponding to medical professionalism were coded and abstraction and specification of concepts were repeated until 90 codes in 21 categories, 7 areas and 3 fields on the concept of medical professionalism  were extracted (Table 2).

Step 2: Investigating the concepts of medical professionalism and creating an item pool

Based on the concepts of medical professionalism, a working group of seven individuals, including the author made a close investigation of the draft items and created an item pool composed of 259 items.

Step 3: Developing the Level 1 Scale: Pre-test

Study design: A cross-sectional study with a collective survey using anonymous, self-administered questionnaires were conducted in April – June 2016.

Subjects: Subjects were first-year students of two universities located in Hokkaido, the northernmost part of Japan. The survey was conducted on 714 first-year students in eight departments (pharmacy, dentistry, nursing, clinical social welfare, clinical psychology, physical therapy, occupational therapy, speech-language-hearing therapy and dental hygiene) of University A, and 110 first-year students of the department of medicine of University B.

Survey items: As demographic characteristics, students’ academic faculty and department, year, and gender were surveyed. Thirty-five items were selected from the item pool as items to assess medical professionalism. The response options were as follows: Agree’ (5 points), ‘Agree to a certain extent’ (4 points), ‘Neither’ (3 points), ‘Disagree to a certain extent’ (2 points), and ‘Disagree’ (1 point).

Analysis: Exploratory factor analysis was performed to confirm the factor structure of the concepts of medical professionalism. The method for factor extraction consisted of the principal factor method using the promax rotation. The Kaiser-Meyer-Olkin (KMO) test and the Bartlett's test of sphericity were used to examine the appropriateness of factor analysis. A result of the former of ≥0.8 and the latter of p<0.05 was considered to define approapriateness.

Step 4: Developing the Level 1 Scale: Main Survey

Study design: A cross-sectional study by collective survey using an anonymous, self-administered questionnaire was conducted in September 2016.

Subjects: 131 fourth-year students in the medical department and 40 students in the department of pharmacy of University C located near Tokyo.

Survey items: As demographic characteristics, students’ academic faculty and department, year, and gender were surveyed. Twelve items were excluded from 35 items used in pre-test, and 12 items were newly added from the item pool, so that 35 items were used in the main survey.

The KiSS-18 is an 18-item scale for measuring social skills in young adults developed in Japan with confirmed reliability and validity (Kikuchi, 2004; Niitsuma et al., 2012), The KiSS-18 evaluates skills in communication and developing personal relationships, and is used to assess criterion-related validity of the present medical professionalism scale. Each question on the KiSS-18 can be answered according to a following five-point Likert scale.

Analysis: Exploratory factor analysis was performed basically the same as that of pre-test. The Pearson correlation coefficient between the KiSS-18 and total score for items that compose the factors were calculated to assess criterion-related validity of the scale. As the KiSS-18 scale assess and measures social skills, the correlations with the medical professionalism scale were as follows: Moderate correlation (0.5) or above was expected with subscale items related to communication skills and interpersonal skills, whereas a mild positive correlation (0.3 or greater but less than 0.5) was expected with other subscale items.

The Cronbach’s coefficient alpha for each subscale item and all items were calculated to assess reliability of the scale.

Step 5: Developing the Level 2 scale

Study design: A cross-sectional study of a collective survey using anonymous, self-administered questionnaires was conducted in March – September 2016.

Subjects: Subjects were 352 junior residents immediately before starting their residencies in 35 postgraduate education hospitals affiliated with University B (i.e., immediately after graduation), and 83 fourth-year students in the nursing department of University C. Questionnaires were either collected and mailed back by the person in charge, or were mailed by individual residents.

Survey items: Gender was surveyed as a demographic characteristic. Level 2 items of the medical professionalism assessment scale were selected among the item pool. Fifty items were selected. Twenty-two of the 50 items were re-selected among items used in the Level 1 scale.

The Japanese version of the P-MEX (Professionalism Mini-Evaluation Exercise) (Tsugawa et al., 2011; Tsugawa et al., 2009), often used as a scale to assess professionalism in residents, were used to test criterion-related validity with the Level 2 scale. The P-MEX is composed of 4 factors and 24 items, among which the second factor, “Reflective skills” (5 items) was used as skills assumed to be largely acquired at the time of graduation. The P-MEX is used to assess others, not for self-assessment, the options for answers were revised to five-point Likert scale.

Analysis: The same methods as those used to for the main survey for the development for the Level 1 scale were used.

SPSS Statistics 22.0 (IBM) was used for the analysis of both surveys, and p<0.05 was set statistically significant.

Ethical considerations

Anonymous, self-administered questionnaires were used for the survey. Subjects were explained orally and in writing of the purpose, methods, ethical considerations and publication of the study results, and the survey was conducted on those who gave consent. Subjects were clearly explained the ethical considerations, including the following: (1)Data that allow identification of individuals would be deleted before use in analysis; (2)Subjects would not be subjected to losses or disadvantages regardless of their decision to cooperate or not cooperate in the study.

This study was approved by the Ethical Board of the Sapporo Medical University (27-2-58).

Level 1 Scale: Pre-test

The response rate was 97.5% (803/824). Respondents were represented in largest numbers of the departments of pharmacy, nursing, medicine, physical therapy and dentistry, in that order. There were more female respondents (60.4%) than males (Table 3).

An exploratory factor analysis was performed to test construct validity. Eleven items that did not meet the criteria presented in the methods chapter were excluded from the factor analysis, and 5 factors composed of 24 items were extracted.

Level 1 Scale: Main survey

The response rate was 90.6% (155/171). There were slightly more male (58.3%) than females (41.7%) (Table 3). The mean KiSS-18 score ± standard deviation was 62.65±10.44. There are studies that have assessed university students in medical domains to score around 60 points (Fujino et al. 2005; Kudou et al. 2007 ; Yamamoto et al. 2013), the social skills of the subjects of analysis can be estimated to be largely standard.

Thirty items in 7 factors were extracted byfactor analysis (Table 4). The KMO statistic was 0.839 and the Bartlett's test of sphericity was significant (p<0.001, χ²=1384.8, df=435), demonstrating the appropriateness of the results of the factor analysis. The 7 factors were named as follows: “Building interpersonal relationships” (Factor 1), “Planned learning”  (Factor 2), “Interest in community health” (Factor 3), “Reflective practice” (Factor 4), “Knowledge and skills” (Factor 5), "Ethical and social responsibility” (Factor 6) and “Self-management” (Factor 7).

The Pearson product-moment correlation coefficient between the scores for the 7 factors and the KiSS-18 was 0.23-0.76. The correlation was strongest with Factor 1, “Building interpersonal relationships” (r=0.76), whereas the correlation was weakest with Factor 5, “Knowledge and skills” (r=0.23) and Factor 6, "Ethical and social responsibility” (r=0.23). The correlation coefficient between the total scores for the 30 items and the KiSS-18 was 0.71 (Table 5).

The coefficient alpha of the 7 factors was 0.63-0.86, and the coefficient alpha for all 30 items was 0.90.

Level 2 scale

The response rate was 55.2% (240/435). There were slightly more females (51.7%) compared to males (48.3%) (Table 3). The score distribution of “Reflective skills” of the P-MEX was 15.33±2.12.

Thirty-one items from 8 factors were extracted by factor analysis (Table 6). The KMO statistic was 0.886 and the Bartlett's test of sphericity was significant (p<0.001, χ2=3794.1, df=465); thus, it demonstrated the appropriateness of the the factor analysis results.  The 8 factors were named as follows: “Providing safe, quality care” (Factor 1), “Providing patient-centred care” (Factor 2), “Planned learning” (Factor 3), “Collaborative practice” (Factor 4), “Building interpersonal relationships” (Factor 5), “Interest in community health (Factor 6), “Ethical and social responsibility” (Factor 7) and “Reflective practice” (Factor 8).

There was a moderate correlation between ”Reflective skills,” Factor 2 of the P-MEX and the Level 2 medical professionalism assessment scale, at 0.28-0.59 (Table 7). The correlation coefficient with Factor 8, “Reflective practice” for which the details of the assessment items are similar, was particularly high at 0.59.

The coefficient alpha of the Level 2 scale was in the range of 0.71 to 0.87, and the α coefficient for all 31 items was 0.93.

Construct validity of the medical professionalism assessment scale

A factor analysis resulted in the extraction of 30 items in 7 factors from the Level 1 scale and of 31 items in 8 factors of the Level 2 scale. The results of Kaiser–Meyer–Olkin test and Bartlett's test of sphericity were good overall.

We will attempt to compare the concepts that compose the ABIM scale(Arnold et al., 1998) and the P-MEX scale(Cruess et al., 2006) to investigate the construct validity of the Level 2 scale. The ABIM scale is composed of the 3 factors of “Altruism,” “Respect for others,” “Excellence” and “Honour and Integrity.” The details of “altruism” and “respect for others” are related to patient-centred care and collaboration with professionals in other healthcare professionals. They correspond to the 2nd factor “Providing patient-centred care” and the 4th factor “Collaborative practice” of our Level 2 scale. “Excellence” also corresponds to the 2nd factor “Providing patient-centred care” and the 4th factor “Collaborative practice.” “Honour and integrity” corresponds to the 2nd, 7th and 8th factors, which are respectively, “Providing patient-centred care,” "Ethical and social responsibility” and “Reflective practice.”

The P-MEX(Cruess et al., 2006) has 24 items in the 4 factors of “Doctor-patient relationship skills,” “Reflective skills,” “Time management” and “Interprofessional relationship skills.” Comparing this to Level 2 scales in this study, “Doctor-patient relationship skills” correspond to “Providing patient-centred care” (Factor 2), “Reflective skills” corresponds to “Reflective practice” (Factor 8), “Time management” corresponds to “planned learning” (Factor 3) and “Interprofessional relationship skills” corresponds to “Collaborative practice” and “Building interpersonal relationships” (Factors 4 and 5). As such, the components of the ABIM and P-MEX scales are encompassed by the Level 2 scale of this study, and were also found to be consistent with the concepts of medical professionalism displayed in Table 2. These observations demonstrate that the Level 2 scale is a scale that can comprehensively assess medical professionalism.

Our Level 2 scale contains the concepts of “Providing safe, quality care” (Factor 1) and “Interest in community health” (Factor 6) which are not included in the ABIM and P-MEX scales. Perhaps there is some debate as to whether Factor 6, “Interest in community health,” is an element of professionalism that all healthcare professionals should have. However, in recent years, days in hospital are being cut dramatically in acute care hospitals especially in Japan, and with this change, healthcare professionals in acute care are also required to adjust the treatment and care they give to one with consideration for the patient’s life and environment following discharge. This suggests that it is very important for “Interest in community health” to be included in the scale for measuring medical professionalism. As such, the Level 2 scale of this study regards the concepts of medical professional more broadly and comprehensively than existing scales of medical professionalism, and is a novel scale that is well adapted to change in the healthcare environment.

On the other hand, for the characteristics of the Level 1 scale 18 items in the field “Personality development and social skills (Category: 11a, 11b, 11c, 11d)” have been selected, which suggests the importance of cultivating professionalism in this field among students before they start their clinical practice. There are no scale items that correspond to the field “Patient-centred care(Category: 21a, 21b, 21c)” in the Level 1 scale, but there are 9 items in the Level 2 scale, which illustrates that this is an aspect of professionalism that is fostered in clinical practice. As such, the scales at the two levels are adapted to the readiness of students, as the Level 1 scale measures professionalism of students before clinical practice and the Level 2 scale measures medical professionalism at graduation.

The criterion-related validity of the medical professionalism assessment scale

To investigate the criterion-related validity of our medical professionalism assessment tools, we examined the correlation between the Level 1 scale and the KiSS-18, which measures social skills. The correlation was the strongest (r=0.76) with the 1st factor, “Building interpersonal relationships,” which is the most similar to the details of the KiSS-18, suggesting that the respondents were reacting and answering appropriately to the items and thereby securing criterion-related validity.

We examined the correlation between eight Level 2 subscale items and "Reflective Skills" of the P-MEX. The correlation was the strongest (r=0.59) with the 8th factor of "Reflective practice". This also suggested that the respondents were reacting appropriately to the items, thereby again securing criterion-related validity.

Reliability of the medical professionalism assessment scale

The reliability coefficient for the 8 subscales in the Level 2 scale lies in the range of  0.71–0.87, ensuring high reliability. The reliability coefficient α of seven subscales in the Level 1 scale was above 0.6 for all subscale, which indicates that a certain degree of reliability was ensured. However, it might be necessary to reinvestigate the possibility of revising some of the scale items as the reliability coefficient α of three subscale items was below 0.7.

Limitations and future challenges in the medical professionalism assessment scale

The first of the limitations and tasks of this scale, we are yet to verify the cross-cultural validity of the scales. Second, there is room for improvement in the Level 1 scale to increase reliability, and this may require reinvestigation with possible revision of the scale. Third, test-retest reliability should be examined. Forth, this is a self-assessment tool and does not directly measure the medical professionalism abilities acquired by students. The predictive validity should be examined by the longitudinal study.

We developed two medical professionalism assessment scales for students to self-evaluate the levels of medical professionalism that they have acquired. The scales were based on concepts of medical professionalism extracted from various materials on medical professionals published or presented by various professional associations. Factor analysis resulted in 30 items from 7 factors for Level 1 (before clinical practice) and 31 items from 8 factors for Level 2 (at the time of graduation) assessments. Construct validity, criterion-related validity and reliability were generally confirmed, and we will study these issues further to enable their use in evaluation of medical education.