New education method or tool
Open Access

Developing the Comprehensive Medical Professionalism Assessment Scale

Takeshi Yamamoto[1], Akito Kawaguchi[2], Yoshinori Otsuka[3]

Institution: 1. Sapporo Medical University, 2. Hokkaido University of Science, 3. Sapporo International University
Corresponding Author: Mr Takeshi Yamamoto (t-yamamoto@sapmed.ac.jp)
Categories: Assessment, Professionalism/Ethics, Students/Trainees, Teaching and Learning, Curriculum Evaluation/Quality Assurance/Accreditation
Published Date: 18/04/2019

Abstract

Background: There has been an increasing number of articles that have studied the topic of medical professionalism. The aims of this study were to develop a scale of medical professionalism.

Methods: The concepts of medical professionalism as defined by associations and groups of physicians, nurses and physiotherapists were investigated. The surveys using self-administered questionnaires were conducted on students and junior residents. An exploratory factor analysis, calculation of the coefficient alpha, and correlation coefficients with other scales were performed.

Results: A factor analysis resulted in the extraction of 30 items in 7 factors as items of the Level 1 scale for pre-clinical level students. The correlation coefficients between the scores for the 7 factors and KiSS-18 were in the range of 0.23 - 0.76. The coefficient alpha of all 30 items was 0.90.  A factor analysis resulted in the extraction of 31 items in 8 factors from the Level 2 scale for students at the time of graduation. The correlation coefficient between the scores for the 8 factor and “Reflective skills” of the P-MEX was 0.31 - 0.59. The coefficient alpha for all 30 items was 0.93.

Conclusion: Construct validity, criterion-related validity and reliability were generally confirmed for the two scales.

 

Keywords: professionalism; assessment scale; factor analysis; questionnaire survey; validity; reliability

Introduction

There have been an increasing number of studies on medical professionalism since the 1990s(Smith, 2005). Traditionally, professionals such as physicians and attorneys were considered professionals based on the social processes through which the professions were established. However, it has been criticised that physicians are not sufficiently fulfilling the social contracts(Hafferty, 2009), and physicians are consequently expected to cultivate higher levels of professionalism.

 

There are some definitions of professionalism. Stern and Arnold(Stern, 2006) stated that “professionalism is demonstrated through a foundation of clinical competence, communication skills, and ethical and legal understanding, upon which is built the aspiration to and wise application of the principles of professionalism”. The Medical Professionalism Project also defined three fundamental principles and ten professional responsibilities of professionalism as the Physician Charter (ABIMFoundation et al., 2002). A comparison between the concepts of professionalism by Arnold and Stern(Stern, 2006) and the Physician Charter(ABIMFoundation et al., 2002)  results in many common and divergent aspects. Therefore, the development and assessment of programmes that foster medical professionalism first requires defining medical professionalism and then measuring it.

 

The most commonly used instrument to measure professionalism is that created by Arnold et al.(Arnold et al., 1998) based on the definitions of professionalism given by the American Board of Internal Medicine (ABIM) in 1995, and several studies that revised and used this instrument. However, the results of factor analysis showed that there were some items with higher factor loading among several factors, and a reliability assessment showed that some subscale items had a coefficient alpha of less than 0.6. As Arnold et al.(Arnold et al., 1998) suggested the necessity of adding questionnaire items, this scale lacks reliability and validity.

 

The concept of professionalism has changed over time, from how the ABIM defined the concept of professionalism in 1995 to that published in the Physician Charter in 2002. The original six elements of professionalism presented in 1995 are no longer adequate for describing the concept of professionalism. Many other scales and instruments for measuring and evaluating professionalism of physicians have been developed, but there are many that have not been tested for construct validity.

 

This study seeks to develop a medical professionalism assessment scale for evaluating how medical professionalism is taught and fostered in students. Of the presumed steps to acquire medical professionalism, this study defines the stage before starting clinical practice as Level 1 and the stage at the time of graduation as Level 2, and develops two scales matched to the students' level of readiness. Development of these scales will contribute to objective evaluation of the outcomes of education in medical professionalism.

 

Methods

The following procedures were undertaken for developing an assessment scale of medical professionalism. First, the concepts of medical professionalism is extracted. Second, an item pool will be created based on the concepts extracted as a source for the scale items. Third, a pre-test will be conducted to develop the Level 1 scale for assessment readiness in pre-clinical students. Fourth, the main survey for developing the Level 1 scale for assessment of readiness in pre-clinical students will be conducted. Fifth, the Level 2 scale adapted for assessing the readiness of students at the time of graduation will be developed.

Step 1: Extracting the concepts of medical professionalism

A close investigation was performed on various documents (Table 1) describing medical professionalism of physicians, nurses and physiotherapists as presented or published by associations of each profession in the four countries of Japan, the United States, United Kingdom and Canada. Content corresponding to medical professionalism were coded and abstraction and specification of concepts were repeated until 90 codes in 21 categories, 7 areas and 3 fields on the concept of medical professionalism  were extracted (Table 2).

Table 1. Referrences for constructing of the concept of medical professionalism

No.

Profession

Name

Person or Group, Nation

Year of publication

1

Doctor

Medical professionalism in the new millennium: a physician charter

ABIM: American Board of Internal Medicine, .ACP-ASIM: American College of Physiciasn-American Society of Internal Medicine, EFIM: European Federation of Internal Medicine

2002

2

Doctor

Good medical practice

General Medical Council, U. K.

2013

3

Doctor

CanMEDS 2015 physician competency framework

The Royal College of Physicians and Surgeons of Canada, Canada

2015

4

Doctor

Professional ethics guidelines of the doctor

Japan Medical Association (Japan)

2008

5

Doctor

A draft proposal of goals for professionalism

Japan Society for Medical Education, Japan

2015

6

Doctor

International code of medical ethics

World Medical Association

2006

7

Doctor

Measuring Medical Professionalism

Stern, D. T. Arnold, L.

2005

8

Nurse

The ICN code of ethics for nurses

ICN: International Council of Nurses

2012

9

Nurse

The ANA nursing code of ethics

ANA: American Nurses Association, U. S.

2015

10

Nurse

The code of ethics for nurses

Japan Nursing Association, Japan

2003

11

Nurse

A model for professionalism

Miller, B. K

1988

12

Physical
therapist

Professionalism: Physical therapy core values

APTA:  American Physical Therapy Association, U. S.

2012

13 Occupational therapist Code of professional values and behaviour CSP: Chartered Society of Physiotherapy, U. K. 2011

Table 2. Medical professionalism concepts extracted from various products

Area       

Field      

Cateogory

Personality development, knowledge and skills as the basis for medical professionalism(1)

 

 

 

 

Personality development and social skills(11)

 

 

 

Character, integrity, empathy(11a)

Self-management (11b)

Communication skills (11c)

Continued education and career development (11d)

Reflective practice (11e)

Practice based on high skill levels and knowledge (12)

Practice based on high skills and knowledge (12a)

Establishing autonomy (13a)

Interactions between the patient, the medical profession and other health professionals(2)

 

 

 

 

Patient-centred care (21)

Patient-centred approach (21a)

Supporting patient autonomy (21b)

Understanding and interacting with the patient as a whole person (21c)

Collaborative practice (22)

 

Collaboration with professionals in other fields (22a)

Collaboration within the medical/healthcare profession (22b)

Building an organizational environment for care(23)

Preparing the organizational environment for care (23a)

Providing and promoting safe medical care (23b)

Fulfil social responsibilities (3)

 

 

 

 

 

 

Contributing to the community, professional associations and society (31)

 

 

Affiliation and contributions to  professional medical/healthcare associations (31a)

Understanding of and contribution to public health activities in the community (31b)

Contributing to health policies (31c)

Use of mass media and information provision (31d)

Understand the general legal and ethical principles to fulfil social responsibilities(32)

 

Respect legal and ethical principles (32a)

Appropriate handling and protection of personal information (32b)

Fulfilling social responsibilities (32c)

Concept identification numbers in parentheses.

Ninety "code" subordinated to the "category" are omitted due to the lack of space.

Step 2: Investigating the concepts of medical professionalism and creating an item pool

Based on the concepts of medical professionalism, a working group of seven individuals, including the author made a close investigation of the draft items and created an item pool composed of 259 items.

Step 3: Developing the Level 1 Scale: Pre-test

Study design: A cross-sectional study with a collective survey using anonymous, self-administered questionnaires were conducted in April – June 2016.

Subjects: Subjects were first-year students of two universities located in Hokkaido, the northernmost part of Japan. The survey was conducted on 714 first-year students in eight departments (pharmacy, dentistry, nursing, clinical social welfare, clinical psychology, physical therapy, occupational therapy, speech-language-hearing therapy and dental hygiene) of University A, and 110 first-year students of the department of medicine of University B.

Survey items: As demographic characteristics, students’ academic faculty and department, year, and gender were surveyed. Thirty-five items were selected from the item pool as items to assess medical professionalism. The response options were as follows: Agree’ (5 points), ‘Agree to a certain extent’ (4 points), ‘Neither’ (3 points), ‘Disagree to a certain extent’ (2 points), and ‘Disagree’ (1 point).

Analysis: Exploratory factor analysis was performed to confirm the factor structure of the concepts of medical professionalism. The method for factor extraction consisted of the principal factor method using the promax rotation. The Kaiser-Meyer-Olkin (KMO) test and the Bartlett's test of sphericity were used to examine the appropriateness of factor analysis. A result of the former of ≥0.8 and the latter of p<0.05 was considered to define approapriateness.

Step 4: Developing the Level 1 Scale: Main Survey

Study design: A cross-sectional study by collective survey using an anonymous, self-administered questionnaire was conducted in September 2016.

Subjects: 131 fourth-year students in the medical department and 40 students in the department of pharmacy of University C located near Tokyo.

Survey items: As demographic characteristics, students’ academic faculty and department, year, and gender were surveyed. Twelve items were excluded from 35 items used in pre-test, and 12 items were newly added from the item pool, so that 35 items were used in the main survey.

The KiSS-18 is an 18-item scale for measuring social skills in young adults developed in Japan with confirmed reliability and validity (Kikuchi, 2004; Niitsuma et al., 2012), The KiSS-18 evaluates skills in communication and developing personal relationships, and is used to assess criterion-related validity of the present medical professionalism scale. Each question on the KiSS-18 can be answered according to a following five-point Likert scale.

Analysis: Exploratory factor analysis was performed basically the same as that of pre-test. The Pearson correlation coefficient between the KiSS-18 and total score for items that compose the factors were calculated to assess criterion-related validity of the scale. As the KiSS-18 scale assess and measures social skills, the correlations with the medical professionalism scale were as follows: Moderate correlation (0.5) or above was expected with subscale items related to communication skills and interpersonal skills, whereas a mild positive correlation (0.3 or greater but less than 0.5) was expected with other subscale items.

The Cronbach’s coefficient alpha for each subscale item and all items were calculated to assess reliability of the scale.

Step 5: Developing the Level 2 scale

Study design: A cross-sectional study of a collective survey using anonymous, self-administered questionnaires was conducted in March – September 2016.

Subjects: Subjects were 352 junior residents immediately before starting their residencies in 35 postgraduate education hospitals affiliated with University B (i.e., immediately after graduation), and 83 fourth-year students in the nursing department of University C. Questionnaires were either collected and mailed back by the person in charge, or were mailed by individual residents.

Survey items: Gender was surveyed as a demographic characteristic. Level 2 items of the medical professionalism assessment scale were selected among the item pool. Fifty items were selected. Twenty-two of the 50 items were re-selected among items used in the Level 1 scale.

The Japanese version of the P-MEX (Professionalism Mini-Evaluation Exercise) (Tsugawa et al., 2011; Tsugawa et al., 2009), often used as a scale to assess professionalism in residents, were used to test criterion-related validity with the Level 2 scale. The P-MEX is composed of 4 factors and 24 items, among which the second factor, “Reflective skills” (5 items) was used as skills assumed to be largely acquired at the time of graduation. The P-MEX is used to assess others, not for self-assessment, the options for answers were revised to five-point Likert scale.

Analysis: The same methods as those used to for the main survey for the development for the Level 1 scale were used.

SPSS Statistics 22.0 (IBM) was used for the analysis of both surveys, and p<0.05 was set statistically significant.

 

Ethical considerations

Anonymous, self-administered questionnaires were used for the survey. Subjects were explained orally and in writing of the purpose, methods, ethical considerations and publication of the study results, and the survey was conducted on those who gave consent. Subjects were clearly explained the ethical considerations, including the following: (1)Data that allow identification of individuals would be deleted before use in analysis; (2)Subjects would not be subjected to losses or disadvantages regardless of their decision to cooperate or not cooperate in the study.

This study was approved by the Ethical Board of the Sapporo Medical University (27-2-58).

Results

Level 1 Scale: Pre-test

The response rate was 97.5% (803/824). Respondents were represented in largest numbers of the departments of pharmacy, nursing, medicine, physical therapy and dentistry, in that order. There were more female respondents (60.4%) than males (Table 3).

An exploratory factor analysis was performed to test construct validity. Eleven items that did not meet the criteria presented in the methods chapter were excluded from the factor analysis, and 5 factors composed of 24 items were extracted.

Table 3. Demographic characteristics of respondents

Reserch

Variable

Category

n

%

Mean,SD

Level 1:

Pre-test

(n=803)

 

 

 

 

 

 

 

 

 

Disipline

 

 

 

 

 

 

 

 

 

Pharmacy

169

21.0

 

Nursing

114

14.2

 

Medicine

101

12.6

 

Physical therapy

85

10.6

 

Dentistry

83

10.3

 

Clinical psychology

67

8.3

 

Speech-language-hearing therapy

66

8.2

 

Occupational therapy

43

5.4

 

Clinical scial work

41

5.1

 

Dental hygiene

34

4.2

 

Gender

 

Male

318

39.6

 

Female

485

60.4

 

Level 1:

Main survey

(n=144)

 

 

Disipline

 

Medicine

105

72.9

 

Pharmacy

39

27.1

 

Gender

 

Male

84

58.3

 

Female

60

41.7

 

KiSS-18

 

143

 

62.65±10.44

Level 2:

Main survey

(n=237)

 

Disipline

 

Medical intern

163

68.8

 

Nursing

74

31.2

 

Gender

 

Male

114

48.3

 

Female

122

51.7

 

P-MEX Reflective slills 233   15.33±2.12

Level 1 Scale: Main survey

The response rate was 90.6% (155/171). There were slightly more male (58.3%) than females (41.7%) (Table 3). The mean KiSS-18 score ± standard deviation was 62.65±10.44. There are studies that have assessed university students in medical domains to score around 60 points (Fujino et al. 2005; Kudou et al. 2007 ; Yamamoto et al. 2013), the social skills of the subjects of analysis can be estimated to be largely standard.

 

Thirty items in 7 factors were extracted byfactor analysis (Table 4). The KMO statistic was 0.839 and the Bartlett's test of sphericity was significant (p<0.001, χ2=1384.8, df=435), demonstrating the appropriateness of the results of the factor analysis. The 7 factors were named as follows: “Building interpersonal relationships” (Factor 1), “Planned learning”  (Factor 2), “Interest in community health” (Factor 3), “Reflective practice” (Factor 4), “Knowledge and skills” (Factor 5), "Ethical and social responsibility” (Factor 6) and “Self-management” (Factor 7).

Table 4. Results of factor analysis of the level 1 medical professional assesment scale

ID Category Item Factor loadings
Factor1 Factor2 Factor3 Factor4 Factor5 Factor6 Factor7

1

11c

I can continue a conversation while paying attention to the other person's responses.

0.841

0.075

-0.110

-0.083

0.030

-0.093

-0.104

2

11c

I can engage in active listening by demonstrating empathy to the other person.

0.711

-0.221

0.040

-0.095

-0.101

0.176

0.042

3

22a

I know how to facilitate smooth group work.

0.707

-0.007

0.080

0.090

0.069

-0.049

0.052

4

22b

I have built a relationship of mutual aid and cooperation with other students in the school.

0.672

-0.168

-0.150

0.103

0.103

-0.034

0.142

5

11c

I can express my thoughts in a logical manner.

0.56

0.208

0.170

0.019

-0.139

-0.079

-0.100

6

11a

I know how to speak and act according to the situation or according to my role in it.

0.546

0.078

-0.05

-0.082

0.014

0.156

0.106

7

22b

I know how to assert my opinion with respect to others opinions or values.

0.524

0.247

0.040

0.156

0.026

0.013

-0.130

8

11c

I can "break the ice" easily and speak frankly with people who I meet for the first time.

0.518

0.059

0.280

-0.06

-0.024

-0.054

0.064

9

11d

I can identify aspects that I should study.

-0.098

0.741

0.020

0.130

0.090

-0.098

-0.010

10

11d

I know what my present tasks are to bring myself closer to my future goals.

0.038

0.722

0.030

-0.013

-0.013

0.152

-0.010

11

11d

I participate in activities both in and outside of the school to promote my own development.

0.076

0.655

-0.040

0.042

0.015

0.036

-0.139

12

11d

I can create a study plan and follow it.

-0.052

0.608

-0.070

-0.144

0.113

0.184

0.297

13

11d

I have the habit of studying daily whether or not there is a major or minor examination coming up.

-0.017

0.518

0.030

-0.096

0.168

0.096

0.161

14

31b

I am interested in public health activities, welfare, and medical care services in the community in addition to the care offered at hospitals.

-0.038

-0.068

0.950

0.072

0.001

0.002

-0.043

15

31b

I am interested in promoting health and preventing diseases in the community in addition to offering treatment and care for diseases.

0.025

-0.027

0.780

-0.087

0.107

0.124

-0.052

16

31b

I have knowledge regarding professionals working in public health and welfare facilities other than medical facilities, such as hospitals and clinics.

-0.010

-0.019

0.610

0.006

0.121

-0.067

0.154

17

31b

I can imagine the lifestyles of patients living in the community.

0.060

0.209

0.540

0.016

-0.153

-0.015

0.084

18

11a

If I do not know something, I can modestly admit it.

-0.066

0.119

0.030

0.720

-0.146

-0.104

-0.005

19

11e

I am open to criticism or assessment by others.

0.127

0.032

-0.080

0.634

0.204

0.102

-0.111

20

11e

I can accurately estimate my current capacity and limits.

-0.098

0.224

-0.110

0.537

0.123

-0.016

0.104

21

11a

I make every effort to understand others' feelings.

0.078

-0.189

0.110

0.508

-0.039

0.150

0.059

22

11a

I always try to be honest.

-0.111

-0.12

0.06

0.449

-0.120

0.237

0.235

23

12a

I have the skills to operate an academic database to obtain information.

-0.01

0.081

-0.040

0.008

0.644

0.013

0.062

24

12a

I obtain the latest knowledge in medicine and medical care through media such as newspapers or television.

0.003

0.154

0.200

-0.027

0.622

-0.136

-0.018

25

32c

I believe that it is wrong to use others for my benefit.

-0.183

0.138

0.130

0.084

-0.114

0.695

-0.121

26

32a

I believe that cheating in a test or verbal examination is wrong under any circumstances.

0.104

0.232

-0.072

0.020

-0.075

0.570

-0.05

27

32a

I am confident that I will never divulge personal information that I have access to.

0.232

-0.143

-0.054

0.069

0.188

0.426

-0.013

28

11b

I always arrive on time and am prepared to learn at lectures, seminars, or internships.

-0.072

-0.026

0.085

0.019

0.196

-0.059

0.792

29

11b

I always pay attention to my own health.

0.177

0.179

-0.125

0.053

-0.348

-0.123

0.517

30

11b

I have an accurate grasp of my daily schedule.

0.161

-0.051

0.063

0.161

0.064

-0.078

0.447

The 7 factors were named as follows: “Building interpersonal relationships” (Factor 1), “Planned learning”  (Factor 2), “Interest in community health” (Factor 3), “Reflective practice” (Factor 4), “Knowledge and skills” (Factor 5), "Ethical and social responsibility” (Factor 6) and “Self-management” (Factor 7).

The Pearson product-moment correlation coefficient between the scores for the 7 factors and the KiSS-18 was 0.23-0.76. The correlation was strongest with Factor 1, “Building interpersonal relationships” (r=0.76), whereas the correlation was weakest with Factor 5, “Knowledge and skills” (r=0.23) and Factor 6, "Ethical and social responsibility” (r=0.23). The correlation coefficient between the total scores for the 30 items and the KiSS-18 was 0.71 (Table 5).

The coefficient alpha of the 7 factors was 0.63-0.86, and the coefficient alpha for all 30 items was 0.90.

Table 5. Pearson's correlation coeffieients between the sub-scale of Level 1 medical professional evaluation scale and KiSS-18, and Cronbach's alpha coefficients

Sub-scale

Correlation coeffieient with KiSS-18

Cronbach's alpha

Factor 1: Building interpersonal relationships

0.76 ***

0.86

Factor 2: Planned learning

0.52 ***

0.82

Factor 3: Interest in community health

0.44 ***

0.84

Factor 4: Reflective practice

0.46 ***

0.72

Factor 5: Knowledge and skills

0.23 **

0.63

Factor 6: Ethical and social responsibility

0.23 **

0.63

Factor 7: Self-management

0.43 ***

0.64

Total score: 30 items

0.71 ***

0.90

** p<0.01, ***p<0.001

Level 2 scale

The response rate was 55.2% (240/435). There were slightly more females (51.7%) compared to males (48.3%) (Table 3). The score distribution of “Reflective skills” of the P-MEX was 15.33±2.12.

Thirty-one items from 8 factors were extracted by factor analysis (Table 6). The KMO statistic was 0.886 and the Bartlett's test of sphericity was significant (p<0.001, χ2=3794.1, df=465); thus, it demonstrated the appropriateness of the the factor analysis results.  The 8 factors were named as follows: “Providing safe, quality care” (Factor 1), “Providing patient-centred care” (Factor 2), “Planned learning” (Factor 3), “Collaborative practice” (Factor 4), “Building interpersonal relationships” (Factor 5), “Interest in community health (Factor 6), “Ethical and social responsibility” (Factor 7) and “Reflective practice” (Factor 8).

There was a moderate correlation between ”Reflective skills,” Factor 2 of the P-MEX and the Level 2 medical professionalism assessment scale, at 0.28-0.59 (Table 7). The correlation coefficient with Factor 8, “Reflective practice” for which the details of the assessment items are similar, was particularly high at 0.59.

The coefficient alpha of the Level 2 scale was in the range of 0.71 to 0.87, and the α coefficient for all 31 items was 0.93.

Table 6. Results of factor analysis of the level 2 medical professional evaluation scale

 

ID

Category

 

Item

Factor loadings

Factor

1

Factor

2

Factor

3

Factor

4

Factor

5

Factor

6

Factor

7

Factor

8

101

21a

I can plan treatment and care strategies considering the patient's quality of life.

0.965

0.080

0.028

-0.004

-0.031

-0.008

-0.012

-0.205

102

12a

I understand the basic principles of safety management in providing treatment and care.

0.846

-0.237

-0.090

0.045

0.158

0.073

0.139

0.017

103

12a

I know the standard treatments and care methods for common diseases.

0.760

-0.256

0.122

0.000

0.031

-0.068

-0.066

0.167

104

21a

I can explain things to patients accurately in a way that is easy to understand.

0.573

0.283

-0.047

-0.165

-0.008

0.021

-0.125

0.209

105

21b

I can provide the necessary information required by patients for decision making.

0.535

0.318

0.006

0.029

-0.119

0.062

-0.146

0.064

106

21a

I understand the intent and purpose of a second opinion.

0.434

0.292

0.055

-0.017

-0.047

0.063

0.099

-0.170

107

21c

I do my best to gain a deep understanding of the patient's values and background to determine his/her lifestyle.

-0.149

0.868

0.050

-0.036

0.069

0.035

-0.032

-0.062

108

21c

I try to predict patients' feelings or thoughts while interacting with them.

-0.125

0.733

-0.165

0.072

0.208

0.029

0.070

0.031

109

21a

I make efforts to build a relationship of trust with patients.

0.253

0.678

-0.115

-0.129

0.124

-0.115

0.171

-0.056

110

21b

I can interact with patients to encourage them to learn or acquire knowledge on their own.

0.073

0.530

0.136

0.156

-0.159

0.096

-0.106

0.063

13

11d

I have the habit of studying daily whether or not there is a major or minor examination coming up.

-0.026

-0.034

0.832

-0.013

0.037

0.026

-0.033

-0.034

12

11d

I can create a study plan and follow it.

-0.056

-0.173

0.703

-0.121

0.125

0.171

0.016

0.204

28

11b

I always arrive on time and am prepared to learn at lectures, seminars, or internships.

0.218

0.042

0.544

0.074

-0.017

-0.068

0.101

-0.092

30

11b

I have an accurate grasp of my daily schedule.

0.101

0.172

0.423

0.130

-0.005

-0.043

0.075

-0.083

111

22b

I have built a relationship of mutual aid and cooperation with colleagues within my profession.

-0.023

-0.078

-0.110

0.870

0.139

-0.036

0.116

-0.109

112

22b

I can discuss the state of treatment and care with colleagues in the same profession with both more or equal amount of experience as myself.

0.055

-0.048

0.005

0.833

-0.127

0.047

-0.045

0.062

113

22b

I want to actively participate in the guidance or education of my juniors.

-0.081

0.081

0.177

0.526

0.136

-0.146

-0.067

-0.117

114

22a

I understand the knowledge, skills, and specializations of other professionals to consult them appropriately.

0.102

0.052

-0.052

0.509

-0.095

0.183

-0.016

0.139

8

11c

I can "break the ice" easily and speak frankly with people who I meet for the first time.

0.034

0.007

0.108

-0.039

0.740

0.058

0.024

-0.067

1

11c

I can continue a conversation while paying attention to the other person's responses.

-0.006

0.293

-0.136

0.058

0.574

0.074

-0.056

0.076

115

11a

Friends trust and rely on me.

0.029

0.179

0.193

0.194

0.504

-0.168

-0.057

0.021

116

31b

I have knowledge on the medical or public health activities and welfare services being offered in accordance with the characteristics of the community.

0.030

-0.027

0.043

-0.010

-0.011

0.886

0.008

-0.069

117

31b

I know the functions and roles of professionals working in public health, medical, and welfare facilities other than hospitals.

0.042

0.039

0.069

-0.097

0.037

0.810

0.143

-0.066

118

31a

I can describe my specialization or activities to a layperson.

0.020

0.114

-0.056

0.284

0.056

0.402

-0.039

0.077

119

32c

I will not misuse my title as a medical professional for personal gain (e.g., goods offered by pharmaceutical companies).

0.119

-0.119

-0.092

0.110

0.012

-0.052

0.756

0.156

120

32c

I can firmly refuse monetary gifts or any other gift from patients.

-0.069

0.015

0.113

0.126

-0.026

0.106

0.639

-0.033

121

32b

I do not disclose information that I could obtain through internships and other means to family or friends.

-0.103

0.210

0.045

-0.198

-0.014

0.124

0.539

0.019

20

11e

I can accurately estimate my current capacity and limits.

0.047

-0.132

-0.069

-0.070

0.106

0.039

-0.071

0.804

19

11e

I am open to criticism or assessment by others.

-0.062

0.034

0.053

-0.031

-0.116

-0.123

0.176

0.641

122

21a

I report or can report my mistakes or errors to my instructor without hiding them.

-0.064

0.340

0.061

-0.028

-0.152

-0.125

0.261

0.451

123

11e

I can look back on my mistakes to analyze their cause and benefit from them in future.

0.057

0.073

0.139

0.104

0.202

0.000

-0.024

0.429

The numbers less than 100 are the items used in common by Level 1 and Level 2 scale. The numbers 100 more than 100 are the items used in only Level 2 scale.

The 8 factors were named as follows: “Providing safe, quality care” (Factor 1), “Providing patient-centred care” (Factor 2), “Planned learning” (Factor 3), “Collaborative practice” (Factor 4), “Building interpersonal relationships” (Factor 5), “Interest in community health (Factor 6), “Ethical and social responsibility” (Factor 7) and “Reflective practice” (Factor 8).

 

Table 7. Pearson's correlation coeffieients between the sub-scale of Level2 medical professional evaluation scale and P-MEX, and Cronbach's alpha coefficients

Sub-scale

Correlation coeffieient
with Reflective skills of P-MEX

Cronbach's alpha

Factor1: Providing safe, quality care

0.37 ***

0.87

Factor2: Providing patient-centred care

0.47 ***

0.82

Factor3: Planned learning

0.28 ***

0.78

Factor4: Collaborative practice

0.41 ***

0.78

Factor5: Building interpersonal relationships

0.32 ***

0.77

Factor6: Interest in community health

0.36 ***

0.83

Factor7: Ethical and social  responsibility

0.31 ***

0.71

Factor8: Reflective practice

0.59 ***

0.72

Total score: 30 items

0.53 ***

0.93

** p<0.01, ***p<0.001

Discussion

Construct validity of the medical professionalism assessment scale

A factor analysis resulted in the extraction of 30 items in 7 factors from the Level 1 scale and of 31 items in 8 factors of the Level 2 scale. The results of Kaiser–Meyer–Olkin test and Bartlett's test of sphericity were good overall.

 

We will attempt to compare the concepts that compose the ABIM scale(Arnold et al., 1998) and the P-MEX scale(Cruess et al., 2006) to investigate the construct validity of the Level 2 scale. The ABIM scale is composed of the 3 factors of “Altruism,” “Respect for others,” “Excellence” and “Honour and Integrity.” The details of “altruism” and “respect for others” are related to patient-centred care and collaboration with professionals in other healthcare professionals. They correspond to the 2nd factor “Providing patient-centred care” and the 4th factor “Collaborative practice” of our Level 2 scale. “Excellence” also corresponds to the 2nd factor “Providing patient-centred care” and the 4th factor “Collaborative practice.” “Honour and integrity” corresponds to the 2nd, 7th and 8th factors, which are respectively, “Providing patient-centred care,” "Ethical and social responsibility” and “Reflective practice.”

 

The P-MEX(Cruess et al., 2006) has 24 items in the 4 factors of “Doctor-patient relationship skills,” “Reflective skills,” “Time management” and “Interprofessional relationship skills.” Comparing this to Level 2 scales in this study, “Doctor-patient relationship skills” correspond to “Providing patient-centred care” (Factor 2), “Reflective skills” corresponds to “Reflective practice” (Factor 8), “Time management” corresponds to “planned learning” (Factor 3) and “Interprofessional relationship skills” corresponds to “Collaborative practice” and “Building interpersonal relationships” (Factors 4 and 5). As such, the components of the ABIM and P-MEX scales are encompassed by the Level 2 scale of this study, and were also found to be consistent with the concepts of medical professionalism displayed in Table 2. These observations demonstrate that the Level 2 scale is a scale that can comprehensively assess medical professionalism.

 

Our Level 2 scale contains the concepts of “Providing safe, quality care” (Factor 1) and “Interest in community health” (Factor 6) which are not included in the ABIM and P-MEX scales. Perhaps there is some debate as to whether Factor 6, “Interest in community health,” is an element of professionalism that all healthcare professionals should have. However, in recent years, days in hospital are being cut dramatically in acute care hospitals especially in Japan, and with this change, healthcare professionals in acute care are also required to adjust the treatment and care they give to one with consideration for the patient’s life and environment following discharge. This suggests that it is very important for “Interest in community health” to be included in the scale for measuring medical professionalism. As such, the Level 2 scale of this study regards the concepts of medical professional more broadly and comprehensively than existing scales of medical professionalism, and is a novel scale that is well adapted to change in the healthcare environment.

 

On the other hand, for the characteristics of the Level 1 scale 18 items in the field “Personality development and social skills (Category: 11a, 11b, 11c, 11d)” have been selected, which suggests the importance of cultivating professionalism in this field among students before they start their clinical practice. There are no scale items that correspond to the field “Patient-centred care(Category: 21a, 21b, 21c)” in the Level 1 scale, but there are 9 items in the Level 2 scale, which illustrates that this is an aspect of professionalism that is fostered in clinical practice. As such, the scales at the two levels are adapted to the readiness of students, as the Level 1 scale measures professionalism of students before clinical practice and the Level 2 scale measures medical professionalism at graduation.

The criterion-related validity of the medical professionalism assessment scale

To investigate the criterion-related validity of our medical professionalism assessment tools, we examined the correlation between the Level 1 scale and the KiSS-18, which measures social skills. The correlation was the strongest (r=0.76) with the 1st factor, “Building interpersonal relationships,” which is the most similar to the details of the KiSS-18, suggesting that the respondents were reacting and answering appropriately to the items and thereby securing criterion-related validity.

 

We examined the correlation between eight Level 2 subscale items and "Reflective Skills" of the P-MEX. The correlation was the strongest (r=0.59) with the 8th factor of "Reflective practice". This also suggested that the respondents were reacting appropriately to the items, thereby again securing criterion-related validity.

Reliability of the medical professionalism assessment scale

The reliability coefficient for the 8 subscales in the Level 2 scale lies in the range of  0.71–0.87, ensuring high reliability. The reliability coefficient α of seven subscales in the Level 1 scale was above 0.6 for all subscale, which indicates that a certain degree of reliability was ensured. However, it might be necessary to reinvestigate the possibility of revising some of the scale items as the reliability coefficient α of three subscale items was below 0.7.

Limitations and future challenges in the medical professionalism assessment scale

The first of the limitations and tasks of this scale, we are yet to verify the cross-cultural validity of the scales. Second, there is room for improvement in the Level 1 scale to increase reliability, and this may require reinvestigation with possible revision of the scale. Third, test-retest reliability should be examined. Forth, this is a self-assessment tool and does not directly measure the medical professionalism abilities acquired by students. The predictive validity should be examined by the longitudinal study.

Conclusion

We developed two medical professionalism assessment scales for students to self-evaluate the levels of medical professionalism that they have acquired. The scales were based on concepts of medical professionalism extracted from various materials on medical professionals published or presented by various professional associations. Factor analysis resulted in 30 items from 7 factors for Level 1 (before clinical practice) and 31 items from 8 factors for Level 2 (at the time of graduation) assessments. Construct validity, criterion-related validity and reliability were generally confirmed, and we will study these issues further to enable their use in evaluation of medical education.

Take Home Messages

  • The concepts of medical professionalism are changing with changes in the healthcare environment.
  • There are no existing assessment scales of medical professionalism with proven validity and reliability that can be applied for evaluating education.
  • The two new assessment scales developed for medical professionalism can be used at two levels: before starting clinical practice (Level 1) and at the time of graduation (Level 2).
  • The assessment scales warrant further investigations to examine applicability in other cultures and test-retest reliability.

Notes On Contributors

Takeshi Yamamoto is an associate professor of school of health sciences at Sapporo Medical University, Japan. He is a medical sociologist in the field of medical education and medical professionalism. ORCID ID: https://orcid.org/0000-0003-4765-459X

Akito Kawaguchi is a professor of school of health sciences at Hokkaido University of Science, Japan. He is a phycisian with research regarding health science.

Yoshinori Otsuka is a professor of faculty of sports & human at Sapporo International University, Japan. He is a phycisian with research regarding health science.

Acknowledgements

We would like to express our gratitude to all respondents. We appreciate the feedback offered by M. Abe and J. Kameno.

Bibliography/References

ABIM Foundation, American Board of Internal Medicine, ACP-ASIM Foundation, American College of Physicians-American Society of Internal Medicine and European Federation of Internal Medicine (2002) 'Medical professionalism in the new millennium: a physician charter', Annals of Internal Medicine, 136(3), pp. 243-6. https://doi.org/10.7326/0003-4819-136-3-200202050-00012

 

Arnold, E. L., Blank, L. L., Race, K. E. and Cipparrone, N. (1998) 'Can professionalism be measured? The development of a scale for use in the medical environment', Academic Medicine, 73(10), pp. 1119-1121. https://doi.org/10.1097/00001888-199810000-00025

 

Cruess, R., McIlroy, J., Cruess, S., Ginsburg, S. and Steinert, Y. (2006) 'The Professionalism Mini-evaluation Exercise: a preliminary investigation', Academic Medicine, 81(10 Suppl), pp. S74-8. https://doi.org/10.1097/00001888-200610001-00019

 

Hafferty, F. W. (2009) 'Professionalism and the socialization of medical students', in Cruess, R.L., Cruess, R. and Steinert, Y. (eds.) Teaching medical professionalism. New York: Cambridge University Press, pp. 53-70. https://doi.org/10.1017/cbo9780511547348.005

 

Kikuchi, A. (2004) 'Notes on the researchs Using KiSS-18', Bulletin of the Faculty of Social Welfare, Iwate Prefectural University, 6(2), pp. 41-51.

 

Niitsuma, M., Katsuki, T., Sakuma, Y. and Sato, C. (2012) 'The relationship between social skills and early resignation in Japanese novice nurses', Journal of nursing management, 20(5), pp. 659-667. https://doi.org/10.1111/j.1365-2834.2011.01256.x

 

Smith, L. G. (2005) 'Medical professionalism and the generation gap', The American Journal of Medicine, 118(4), pp. 439-442. https://doi.org/10.1016/j.amjmed.2005.01.021

 

Stern, D. T. (ed.) (2006) Measuring medical professionalism. New York: Oxford University Press.

 

Tsugawa, Y., Ohbu, S., Cruess, R., Cruess, S., Okubo, T., et al. (2011) 'Introducing the Professionalism Mini-Evaluation Exercise (P-MEX) in Japan: results from a multicenter, cross-sectional study', Academic Medicine, 86(8), pp. 1026-31. https://doi.org/10.1097/acm.0b013e3182222ba0

 

Tsugawa, Y., Tokuda, Y., Ohbu, S., Okubo, T., Cruess, R., et al. (2009) 'Professionalism Mini-Evaluation Exercise for medical residents in Japan: a pilot study', Medical Education, 43(10), pp. 968-978. https://doi.org/10.1111/j.1365-2923.2009.03437.x

Appendices

None.

Declarations

There are no conflicts of interest.
This has been published under Creative Commons "CC BY-SA 4.0" (https://creativecommons.org/licenses/by-sa/4.0/)

Ethics Statement

This study was approved by the Ethical Board of the Sapporo Medical University. Reference (27-2-58).

External Funding

This study was supported by aid grants from the Grant-in-Aid for Scientific Research (C) (17K08920) from the Ministry of Education, Culture, Sports, Science and Technology, Japan.

Reviews

Please Login or Register an Account before submitting a Review

Julie Williamson - (16/10/2019) Panel Member Icon
/
This article describes a research study that attempts to compile existing professionalism scales into two comprehensive medical professionalism scales--one for pre-clinical students (Level 1) and one for students at graduation (Level 2). The study does an adequate job of describing its methodology but falls somewhat short of its aim by not including source material from enough diverse medical professions across enough cultures and regions to create a truly comprehensive medical professionalism scale. Additionally, the authors are not creating a medical professionalism scale, but a student rating scale of their self-perceived professionalism, which is not the same thing. I would like to see the study repeated with a larger list of source document material, and with experts reviewing the source document material rather than, or in addition to, students offering their self-ratings. I agree with previous reviewers that a qualitative analysis would add depth to the study.
Possible Conflict of Interest:

For transparency, I am an Associate Editor of MedEdPublish. However I have posted this review as a member of the review panel with relevant expertise and so this review represents a personal, not institutional, opinion.

Ken Masters - (01/08/2019) Panel Member Icon
/
The paper proposes a comprehensive medical professionalism assessment scale.

While the idea is good, the paper suffers from several severe weaknesses:
• It draws its initial documents from four countries only (Japan, the United States, United Kingdom and Canada). Apart from the fact that this represents less than 8% of the world’s population, this is an extremely culturally biased sample. (And mentioning a cultural limitation in the Limitations does not, unfortunately, obscure the fact that there a serious problem here).
• The authors have attempted too many professions at once, and have not collected nearly enough documents – physical therapists have one (US) document, and occupational therapists have one (UK) document. Many of the professions polled later are not represented in these documents at all.
• There needs to be a more detailed description of the search methodology, inclusion and exclusion criteria, how the data extraction occurred, (e.g. independent processes among different researchers, resolution of differences, verification of data, whether or not software was used, etc.)
• These three bullets above indicate that the starting position outlined in Tables 1 and 2 are severely compromised.
• Step 2 also needs to be described in far more detail.
• The authors then surveyed students and junior residents only, limiting the input to an extremely inexperienced group.
• From one country only. This makes the rest of the paper, no matter what happens, extremely weak. Although these issues are listed as limitations, they are not merely limitations – they undermine the crux of the paper.
• An exploratory study of this type should also have an associated qualitative study.

The work from the paper may be salvaged, but, in order to do so, I would recommend that authors make the following revisions to their paper:
• Make it clear that this is NOT a professionalism scale, but a student perception and construction of a professionalism scale. This would need to be in the title, and the aim, which is not to develop a scale, but to develop student views or construction of a scale, and this also needs to be clear in the discussion and conclusion.
• Emphasise that this is not a global scale, but local to Japan only. Then, in the limitations, acknowledge that, although it is limited to Japan, only a few universities from Japan were sampled, so there may be generalisation issues even in Japan.
• Also acknowledge that many of the professions were not represented in the documentation – in fact, only Doctors and Nurses have any real representation.
• On this note, though, I don’t quite know what the authors would do about the wide range of professions that they have attempted to evaluate, and the limited documents to support them (especially as they polled students from professions outside those in the documentation.) If they can tease out medical doctor data only (and perhaps Nursing data), then they may confine to that/those profession/s, but, really, any approach they take will carry weaknesses with it.

Minor issue:
• The Methods begin by describing the process in the future tense. As the project has been completed, these should be in the past tense.

Overall, I think the researchers had a good idea – to develop a comprehensive medical professionalism scale. The undertaking, however, is huge, and the authors would need to:
• Draw on material from across the globe (certainly, far more countries than they currently have)
• Poll a much large group of professionals.
It may be better to have stuck with one country only, and aimed at producing that scale for that one country only. Either way, the other methodological issues would need to be addressed, practicing professionals will need to be included in the surveys or the paper re-written as suggested above, as a student view of professionalism.

Possible Conflict of Interest:

For Transparency: I am an Associate Editor of MedEdPublish

Felix Silwimba - (18/04/2019)
/
This is an important study in medical professionalism. My challenge I faced was with the statistics. I would recommend a qualitative study as well that analyses how medical students perceived the professionalism assessment tools. I agree with the authors the analysed tools need to be tested in other regions and cultures of the world