Commissioned Officers’ Effectiveness Report (COER)
Office of Commissioned Corps Force Management
The COER underwent revision in order to obtain a more objective reflection of performance and a more accurate record of needs for force management. The scale was changed from A through E or F, to 1 through 7, and the new COER contains clearly visible guidelines next to the scale. An additional change increases accountability in the evaluation process by instituting an opportunity for the Reviewing Official (RO) to provide input to the Rater prior to the release of the scores to the officer.
The purpose of the study was to compare ratings assigned using the old scale during the 2009 annual cycle with scores by the same Rater using the new scale based on the same Attachment I used in 2009 to measure performance. The test was to determine whether the scores using the old scale would be similar to the scores obtained using the new scale. The questions with the F option were excluded from the comparison since the new scale does not contain this option.
A randomly selected sample of 121 officers was provided consent forms for participation. Of the 54 officers who provided consent, 50 were rated by their 2009 Raters using the new scale. The data obtained was a non-parametric independent set. Of the 50 officers, 72 percent were in a clinical profession, 62 percent had more than 10 years of service, 56 percent were senior officers and 54 percent were rated by a fellow officer. There were no statistical significant differences among officers in the entire Corps, the sample, or those rated, except that women were more likely (OR 11, CI 2.1-71.1) to be in a clinical professional category and that Junior officers were more likely (OR 4.1, CI 1.1-16.7) to be rated by a Corps officer.
Due to incomplete data for three 2009 COERs, 47 ratings of officers were paired for comparison. The proportion of officers who received E ratings among the 18 questions individually using the old scale ranged from 70 percent to 86 percent, while the proportion of officers who received ‘7’ ratings among the eight elements using the new scale ranged from 26 percent to 36 percent. Using the old COER, 66 percent were rated with all Es for each of the 13 non-F questions. With the new COER, 14 percent were rated with ‘7’ for each of the eight elements. While ratings in the old COER were limited to ‘D’ and ‘E,’ the new ratings ranged from ‘3’ to ‘7.’ For the rating factor “overall performance,” Question 18 on the old COER and Element 8 on the new COER, 41 received a rating of ‘E’ with the old COER and 13 received a rating of ‘7’ using the new COER.
The ratings using the old COER were different from the ratings using the new COER. The factors were paired for comparison and the result was that the ratings were different for each factor and individual. The comparative analysis of the old ratings to the new ratings did not statistically support the test assumption that the ratings would be similar. The distribution of the ratings using the new instrument was more random with higher variability. The limitations of the study include pre-selection, small sample size, non-participation and non-response. This study did not include a cognitive analysis of the instrument.
In summary, based on the analysis, the 2009 ratings were different from the new ratings. Ratings using the new COER are independent of previous ratings, and the new and old ratings should not be compared or used to assess an officer’s growth from 2009 to 2010.
The higher variability of the new COER does appear to lend itself more readily to identifying areas for improvement. Individually and Corps-wide, the new instrument may help to target areas for further development and thereby enhance quality. However, this will depend on the extent to which evaluations maintain professional integrity and accurately reflect the performance of the officer.
More information will be available through the Commissioned Corps Management Information System Web site and additional notices will be released using the listserv as more COER information becomes available. If you have any questions and comments, you can email CAPT Christine J. Benally at Christine.Benally@hhs.gov
 |
 |
 |
HHS, Office of Public Health and Science |