Method
We conducted this observational study during the September 2010 to July 2011 admissions cycle at the University of California, Davis (UCD) School of Medicine (SOM). We invited all applicants who attended an MMI session during this cycle to participate by completing a questionnaire designed to measure personality factors. The UCD institutional review board approved the study protocol.
Applicant Screening
All applicants applied via the American Medical College Application Service (AMCAS). On the basis of application review using criteria that included Medical College Admission Test (MCAT) scores, admissions personnel invited screened applicants to submit a secondary application. Faculty evaluated the completed secondary applications on the basis of MCAT scores, personal statements, extracurricular experiences, and recommendation letters, to select applicants to invite to an MMI session. Each invited applicant self-scheduled his or her MMI session date online; sessions were held from September 2010 through March 2011.
The MMI: Process and Scoring
The MMI consisted of 10 individual timed stations, each lasting 10 minutes. Applicants completed 10 stations consecutively; the starting station and station completion order varied among applicants in nonrandom fashion. Before entering the room at each station, applicants had two minutes to read a brief summary of the station's focus. After entering the room, applicants had eight minutes to address the assigned tasks. Nine stations were designed to facilitate assessment of cognitive and/or noncognitive skills in one or more of the following areas: integrity/ethics, professionalism, interpersonal communication, diversity/cultural awareness, teamwork, ability to handle stress, and problem solving. The 10th station asked applicants to explain their choice to pursue a career in medicine. Content for most stations was adapted from ProFitHR (Advanced Psychometrics for Transitions Inc., Hamilton, Ontario, Canada; http://www.profithr.com/).
Each station was observed by a single trained rater, who had no information regarding applicants' academic or personal backgrounds. Raters from a variety of backgrounds participated, including full-time and volunteer faculty, resident physicians, medical students, nurses, administrative staff, pharmacists, and community members. At some stations, raters interacted with applicants, whereas at others they only observed applicant behavior (e.g., interactions with an actor or another applicant).
At each of the 10 stations, raters scored applicant performance using a four-point scale (range: 0–3 points, with higher scores indicating better performance). For this study, we summed individual station scores to yield a total MMI score for each respondent (range: 0–30). Cronbach alpha for the score in this study was 0.68 (comparable to that observed in other studies), and the average interstation score correlation was 0.18.
Personality Factors: Questionnaire and Scoring
At the end of each MMI session, all applicants who participated attended a group debriefing meeting led by the associate dean of admissions (M.H.). Immediately following the debriefing, and after the associate dean exited the room, a research assistant asked the applicants to complete a paper questionnaire designed to measure their FFM personality factors. Applicants were informed verbally and in writing that questionnaire completion was voluntary and that their participation decision and (if they participated) their responses would not influence acceptance offers, as selection personnel could not access their responses. They were asked to record their AMCAS identification numbers on the form (but no other identifying information) to allow linkage of their responses with other study variables.
The personality questionnaire we employed was the Big Five Inventory (BFI), a validated measure typically completed in less than five minutes. The BFI consists of 44 statements assessing agreeableness (9 items), conscientiousness (9 items), extraversion (8 items), neuroticism (8 items), and openness (10 items). Table 1 summarizes the dispositional tendencies associated with high and low levels of these personality factors and provides example statements from the BFI scales measuring each factor. Respondents indicate their degree of disagreement or agreement with each statement using a five-point Likert scale (1 = strongly disagree to 5 = strongly agree). Sixteen items are reverse coded before scoring. For each respondent, we averaged the scores across all items related to each personality factor to yield a mean factor score (range: 1–5, with higher scores indicating higher levels). Cronbach alphas in this study were as follows: agreeableness, 0.80; conscientiousness, 0.82; extraversion, 0.88; neuroticism, 0.81; and openness, 0.75.
Other Measures
We defined acceptance offers (versus not) as official offers to attend the medical school extended by the dean of admissions after considering all application materials, MMI performance, impressions from small-group meetings with applicants, and selection subcommittee recommendations.
In addition to MMI scores, the admissions office provided us with information extracted from AMCAS applications regarding the following characteristics: age; sex; self-identified race, ethnicity, and disadvantaged status; state of residence; cumulative grade point average (GPA); and MCAT scores. The admissions office provided these data to the research assistant in an electronic spreadsheet that identified applicants only by their AMCAS numbers. The research assistant added BFI data and, to maintain anonymity, replaced AMCAS numbers with unique study identification numbers before sending the data file to the study investigators for analysis.
Data Analysis
We analyzed the data using Stata version 11.2 (Stata Corporation, College Station, Texas). We employed the chi-square test (for categorical variables) and t test (for continuous variables) to compare characteristics of respondents who were offered acceptance with those who were not.
We employed two linear regression models to examine associations between respondent characteristics and total MMI score (the dependent variable). The first model included the following covariates: age (19–21, 22, 23–24, or 25–39 [reference]); female sex (yes/no); any self-identified Hispanic ethnicity (solely Hispanic or multiethnic including Hispanic); any self-identified white race (solely white or multiracial including white); self-identified disadvantaged status (yes/no); California resident (yes/no); cumulative GPA and MCAT Verbal Reasoning, Physical Sciences, and Biological Sciences scores (continuous measures); and, to capture potential changes in the tendency to offer acceptance during the admissions screening cycle, date of MMI participation by quartile (September 3 to October 1, 2010 [reference], October 8 to December 3, 2010, December 10, 2010 to January 28, 2011, or February 4 to March 18, 2011). The second linear regression model included all of the covariates in the first model, plus scores on each of the five personality factors (by quartile, with the lowest quartile scores as reference). To examine the additional contribution of the personality factors to the MMI score, we determined the adjusted model R with and without the personality factors.
We employed three sequential logistic regressions to model medical school acceptance (acceptance offer versus not), the dependent variable. The first model included the same covariates as the first linear regression, the second model added score on each of the five personality factors (by quartile, with the lowest quartile as reference), and the third model added the MMI score. To examine the additional contribution of the personality factors to an offer of acceptance, we calculated the model area under the receiver operating curve (ROC), or discrimination, for the first and second logistic regression models.
In all analyses, we defined statistical significance as a P value < .05.