Results
A total of 96 assessments were completed and 43 registrars were observed.
Inter-rater Reliability
At a scale level, none of the three pairs of observers achieved the desired level of 0.7 for inter-rater reliability (0.575, 0.532 and 0.419). However, ICCs calculated using mean scores for each assessment were more acceptable (0.824, 0.702 and 0.519). The confidence intervals (CIs) for ICCs of individual skills are very large, due to the small sample size, making it difficult to make any meaningful interpretations (see online appendix). However, comparing values across the observers it is possible to identify skills that had particularly low ICCs and those that were generally high. For example, Authority and Assertiveness had low ICCs for all three pairs (−0.111 to 0.455) as did Situational Awareness (SA), Informing the Team (0–0.474). Similarly, Workload Management had generally high ICCs (0.595–0.778), as did Team Building (0.641–0.714).
A review of assessment sheets and discussion notes identified possible sources of variation and solutions to rater error. This is summarised in Table 1 and further examples are given in the online appendix.
Test-retest Reliability
Eighteen participants were observed more than once. Spearman's ρ calculation using all 12 individual skills was 0.26 and 0.70 when using mean scores.
Observability
The observability of each skill is shown in Table 2. While all skills achieved the acceptable level of 50%, four were observed less frequently than the others. These were Supervision & Feedback (77%), Authority and Assertiveness (68%), Outcome Review (76%) and Anticipation (70%).
Description of the Range of Skills
Figure 2 illustrates the mean and dispersion of ratings for each skill. A series of bar charts showing the frequency of ratings for each skill are available in the online appendix. Only one skill, Maintenance of Standards, had a mean <5 (mean 4.8). Other skills with relatively low means included Supervision & Feedback, Authority and Assertiveness, Decision Making (DM)—Selecting & Communicating and Anticipating. The highest mean was calculated for Team Building (mean 6.0). Skills with a larger range of ratings were Maintenance of Standards, Workload Management and SA-Gathering Information, while skills with relatively low distribution of ratings included Authority & Assertiveness, DM-Outcome Review and SA- Informing the Team. Examples of observed good and poor behaviours are shown in Table 3 (a complete list is available from the authors).
(Enlarge Image)
Figure 2.
Mean ratings and dispersion of ratings.
Comparison of Hospitals
Two of the 12 skills, Supervision & Feedback (Skill 3) and SA-Gathering Information (Skill 10), had significantly different distributions of ratings across the four hospitals (p<0.037 and 0.007, respectively) as illustrated in figure 3. Hospital 1 participants performed significantly poorer at the skill, Supervision & Feedback, compared with the other three hospitals. Discussion with observers revealed that Hospital 1 had the highest level of 'shop-floor' consultant presence, so registrars may have relinquished teaching and supervision to the consultant, thereby missing valuable opportunities for teaching more junior doctors. Hospital 3 performed significantly better at the skill SA-Gathering Information compared with the other three hospitals. Observers noted that registrars took on a formal leadership role under the supervision of consultants. There was a noticeable increase in situational awareness behaviours related to organisational issues, such as waiting times, staffing levels and monitoring staff communications (eg, overhearing nurses speaking about a sick patient) as they appeared to take on more responsibility for the coordination of the department. Although not statistically significant, observers noted that registrars in Hospital 2 performed particularly poorly in this skill. There was no identifiable lead doctor, and the nurse in charge took on the main leadership role.
(Enlarge Image)
Figure 3.
Results of the Kruskall–Wallis test. This figure is produced in colour in the online journal. (Please visit the website to view the colour figure).