Understanding performance ratings: Dynamic performance, attributions, and rating purpose.

The present two studies integrate and extend the literatures on dynamic performance, performance attributions, and rating purpose, making several important contributions. First, examining attributions of dynamic performance, Study 1 predicted that performance mean and trend would affect judged ratee ability and effort and that performance variation would affect locus of causality; both predictions were supported by the results. Second, investigating the interaction between dynamic performance and rating purpose, Study 2 predicted that performance mean would have a stronger impact on administrative than on developmental ratings, whereas performance trend and variation would have a stronger impact on developmental than on administrative ratings; again, both predictions were borne out by the results. Third, both studies found that performance trend interacted with performance mean and variability to predict overall ratings. Fourth, both studies replicated main effects of dynamic performance characteristics on ratings in a different culture and, in Study 2, a sample of more experienced managers. (PsycINFO Database Record (c) 2010 APA, all rights reserved)     

Distributional ratings of performance: Further examination of a new rating format.

Examined several aspects of construct validity evidence for a distributional format by comparing it with a Behavioral Observation Scale (BOS) rating form and determining whether raters were sensitive to differences in performance variability. Raters were assigned to 1 of 2 instructional procedures, 1 of 2 form orderings (BOS or distributional rating first), and 1 of 5 conditions of performance variability. Ss rated an instructor's performance after viewing 4 videotaped excerpts of his lectures. Mean ratings were lower using the distributional format relative to the BOS format. The distributional ratings indicated that Ss were sensitive to the different variability conditions. The potential of distributional ratings for providing a richer source of performance information than more traditional ratings is discussed. (PsycINFO Database Record (c) 2010 APA, all rights reserved)     

Effect of prior performance information on ratings of present performance: Contrast versus assimilation revisited.

Examined the effects of knowledge of a ratee's prior performance on evaluations of present performance. Subjects received knowledge of either good or poor prior performances and then viewed and rated a videotape depicting average performance. In Study 1, some subjects received knowledge of the ratee's prior performance by directly viewing videotapes of good or poor ratee behavior, whereas others only reviewed written performance ratings completed by those subjects who had actually viewed the ratee. A contrast effect occurred when knowledge of prior performance was obtained by observing ratee behavior, but an assimilation effect occurred when knowledge of prior performance was obtained by reviewing performance ratings. In Study 2, subjects viewed videotapes of good or poor performances prior to viewing an average performance by the same ratee. However, the separate ratee performances were observed over a more realistic time interval than that used in Study 1 (3 weeks vs. 1 h). No significant contrast effects were observed. In Study 3, subjects reviewed written ratings of prior performances before viewing an average videotape. Subjects who reviewed extremely good (or poor) prior performance ratings provided more extreme ratings of the "average" performance than did subjects who reviewed less extreme ratings. (PsycINFO Database Record (c) 2010 APA, all rights reserved)     

Application of the psychophysical law to performance evaluation.

In the area of job performance, it was hypothesized that evaluations, e.g., "poor" to "excellent," were based ont the observation of "performance specimens" where a performance specimen is defined as "an incident of relevant performance which at the time of observation was classed as uncommonly effective or uncommonly ineffective." It was further hypothesized that the psychophysical law (y = kxn) would describe the relationship between the number of specimens observed (x) and resulting evaluations (y) of performance. Finally, it was hypothesized that Steven's criterion for prothetic continua would be satisfied—concave downward curve when ratio estimation scale values for sets of performance specimens are plotted against corresponding category scale values. Using simulated performance ratings as well as actual performance evaluations (performance ratings of apprentices, professors, supervisors, and executives) the above hypotheses appear to have been verified. (PsycINFO Database Record (c) 2010 APA, all rights reserved)     

Self–other agreement in job performance ratings: A meta-analytic test of a process model.

This meta-analysis explores agreement in self- and supervisory ratings of job performance (k = 128 independent samples). It suggests a 3-stage model of the rating process and reviews the empirical evidence for the relevance of each of these 3 stages to an understanding of agreement in ratings. The proposed 3-stage model serves as the guiding rationale for the examination of an extensive set of variables that moderate rater agreement. Results are reported for 2 indicators of rater agreement (correlational and mean-level agreement). Self-supervisor ratings yielded an overall correlation of .22 (ρ = .34; k = 115; n = 37,752). Position characteristics and the use of nonjudgmental performance indicators were the main moderators. Leniency in self-ratings is indicated by higher mean levels of self-ratings compared with supervisory ratings. Within Western samples, performance self-ratings showed leniency (d = 0.32, Δ = .49; k = 89; n = 35,417) dependent on contextual features, scale format, and scale content. (PsycINFO Database Record (c) 2010 APA, all rights reserved)     

The Impact of Job Complexity and Performance Measurement on the Temporal Consistency, Stability, and Test-Retest Reliability of Employee Job Performance Ratings.

Although research has shown that individual job performance changes over time, the extent of such changes is unknown. In this article, the authors define and distinguish between the concepts of temporal consistency, stability, and test-retest reliability when considering individual job performance ratings over time. Furthermore, the authors examine measurement type (i.e., subjective and objective measures) and job complexity in relation to temporal consistency, stability, and test-retest reliability. On the basis of meta-analytic results, the authors found that the test-retest reliability of these ratings ranged from .83 for subjective measures in low-complexity jobs to .50 for objective measures in high-complexity jobs. The stability of these ratings over a 1-year time lag ranged from .85 to .67. The analyses also reveal that correlations between performance measures decreased as the time interval between performance measurements increased, but the estimates approached values greater than zero. (PsycINFO Database Record (c) 2010 APA, all rights reserved)     

The moderating influence of job performance dimensions on convergence of supervisory and peer ratings of job performance: Unconfounding construct-level convergence and rating difficulty.

Meta-analysis was used to cumulate the correlations between supervisor and peer ratings for different dimensions of job performance. The general pattern in the data suggested that raters from the same organizational level disagree as much as raters from different levels. Methods were used to separate the effects of rating difficulty and lack of construct-level convergence on the correlation between supervisor and peer ratings. The authors found complete construct-level convergence for ratings of overall job performance, productivity, effort, job knowledge, quality, and leadership but not for ratings of administrative competence, interpersonal competence, and compliance or acceptance of authority. Higher rating difficulty was more strongly associated with lower mean observed peer-supervisor correlations than were construct-level disagreements between peers and supervisors. Implications for research and practice are discussed. (PsycINFO Database Record (c) 2010 APA, all rights reserved)     

Rating format research revisited: Format effectiveness and acceptability depend on rater characteristics.

Rating format research has largely been ignored since F. J. Landy and J. L. Farr's (see record 1980-08528-001) call for a moratorium over a decade ago. Their conclusion that ratings were not affected by changes in scale format was based on research that treated all raters alike. However, individuals differ in the way in which they perceive and integrate information. This article investigates the proposition that differences in rating accuracy associated with different rating formats are contingent on rater characteristics. The study tested the rating accuracy and affective reactions toward performance appraisal of field-dependent (FD) and field-independent (FIN) raters on 4 different performance measures. As hypothesized, FINs were more accurate raters than FDs only when scale formats were holistic, and only FDs' ratings were significantly affected by the level of structure in the scale format. FIN raters were also more confident in their ratings and less frustrated and confused with the rating task than were FDs. (PsycINFO Database Record (c) 2010 APA, all rights reserved)     

Assessing Measurement Equivalence Across Rating Sources: A Multitrait-Multirater Approach.

Research examining the structure of multisource performance ratings has demonstrated that ratings are a direct function of both who is doing the rating (rating source) as well as what is being rated (performance dimension). A separate line of research has focused on the extent to which performance ratings are equivalent across sources. To date no research has examined the measurement equivalence of multisource ratings within the context of both dimension and rating source direct effects on ratings. We examine the impact of both performance dimension and rating source as well as the degree of measurement equivalence across sources. Results indicate that (a) the impact of the underlying performance dimension is the same across rating sources, (b) the impact of rating source is substantial and only slightly smaller than the impact of the underlying performance dimension, and (c) the impact of rating source differs substantially depending on the source. (PsycINFO Database Record (c) 2010 APA, all rights reserved)     

The roles of rater goals and ratee performance levels in the distortion of performance ratings.

The goal-directed perspective of performance appraisal suggests that raters with different goals will give different ratings. Considering the performance level as an important contextual factor, we conducted 2 studies in a peer rating context and in a nonpeer rating context and found that raters do use different rating tactics to achieve specific goals. Raters inflated their peer ratings under the harmony, fairness, and motivating goal conditions (Study 1, N = 103). More important, raters inflated their ratings more for low performers than for high and medium performers. In a nonpeer rating context, raters deflated ratings for high performers to achieve the fairness goal, and they inflated ratings for low performers to motivate them (Study 2, N = 120). (PsycINFO Database Record (c) 2010 APA, all rights reserved)     

The relative importance of task, citizenship, and counterproductive performance to global ratings of job performance: A policy-capturing approach.

A review of research on job performance suggests 3 broad components: task, citizenship, and counterproductive performance. This study examined the relative importance of each component to ratings of overall performance by using an experimental policy-capturing design. Managers in 5 jobs read hypothetical profiles describing employees' task, citizenship, and counterproductive performance and provided global ratings of performance. Within-subjects regression analyses indicated that the weights given to the 3 performance components varied across raters. Hierarchical cluster analyses indicated that raters' policies could be grouped into 3 homogeneous clusters: (a) task performance weighted highest, (b) counterproductive performance weighted highest, and (c) equal and large weights given to task and counterproductive performance. Hierarchical linear modeling indicated that demographic variables were not related to raters' weights. (PsycINFO Database Record (c) 2010 APA, all rights reserved)     

Employee responses to formal performance appraisal feedback.

Investigated the attitudinal impacts of receiving formal performance appraisal feedback. Based on the suggestion that the feedback that one is "satisfactory" will be disconfirming for many feedback recipients, it was hypothesized that attitudes toward the performance appraisal systems and organizational commitment will decrease and remain lower for those receiving satisfactory ratings, whereas the attitudes of those receiving higher appraisal ratings will remain unchanged. The hypotheses were tested on panels of management and nonmanagement employees (the latter receiving new appraisals 12 mo after their managers) in 2 federal agencies over a 30-mo period using perceived and actual performance ratings. There was a significant and stable drop in the organizational commitment of satisfactory employees after the introduction of formal appraisals, with mixed results for attitudes toward the appraisal system. Findings suggest that potentially negative consequences of implicitly comparative formal performance appraisals can occur for those performing at a satisfactory, but not outstanding, level. (21 ref) (PsycINFO Database Record (c) 2010 APA, all rights reserved)     

Measurement and assessment issues in performance appraisal.

Performance appraisal is a topic that is of both theoretical interest and practical importance. As such, it is one of the most researched topics in industrial and organisational psychology. Several measurement issues are central to performance appraisal including: (a) how performance has been measured, (b) how to improve performance appraisal ratings, (c) what is meant by performance, and (d) how the quality of ratings has been defined. Each of these are discussed along with the shortcomings of the extant literature in helping to come to grips with these important issues. Next, some of the new challenges facing performance appraisal, given its historical focus on single individuals being evaluated, are highlighted. In particular, the appraisal problems inherent in the assessment of team performance and the complexities inherent in multisource feedback systems are covered. We conclude with a short discussion of the litigious issues that can arise as a result of poor performance management practises. (PsycINFO Database Record (c) 2010 APA, all rights reserved)     

The relative importance of task and contextual performance dimensions to supervisor judgments of overall performance.

Although evidence supports the unique contribution of task performance and contextual performance to overall evaluations, little is known about the relative contribution that specific dimensions of contextual performance make to overall performance judgments. This study evaluated the extent to which supervisors consider task and contextual performance by using relative weights (J. W. Johnson, 2000) to statistically describe the relative importance of specific dimensions of each type of performance to overall performance ratings. Within each of 8 job families in a large organization, each of 4 dimensions of contextual performance made not only a unique contribution but a relatively important contribution to the overall evaluation. Evidence also supports the adaptive performance dimension of handling work stress as an aspect of contextual performance and job–task conscientiousness as an aspect of both task and contextual performance. (PsycINFO Database Record (c) 2010 APA, all rights reserved)     

Effectiveness of performance feedback from behaviorally anchored rating scales.

Compared a behaviorally anchored rating scale (BARS) developed according to the procedure of H. Bernardin et al (see record 1976-08614-001) with a summated rating scale. Using both scale formats (designed to evaluate teaching effectiveness), 859 undergraduates rated 32 instructors during spring classes, and 314 undergraduates rated 19 instructors during summer classes. Students rated instructors halfway through the course and at its end. Instructors received feedback from the 1st rating period on either the BARS (including written behavioral observations) or on the summated rating format. Analyses of covariance revealed no format effect on the performance ratings obtained during the 2nd rating period in the spring experiment. BARS feedback, however, produced greater behavioral change among instructors than did feedback from the alternative format used during the summer experiment. (34 ref) (PsycINFO Database Record (c) 2010 APA, all rights reserved)     

Ethnic group differences in measures of job performance: A new meta-analysis.

The authors conducted a new meta-analysis of ethnic group differences in job performance. Given a substantially increased set of data as compared with earlier analyses, the authors were able to conduct analyses of Black-White differences within more homogeneous categories of job performance and to reexamine findings on objective versus subjective measurement. Contrary to one perspective sometimes adopted in the field, objective measures are associated with very similar, if not somewhat larger, standardized ethnic group differences (ds) than subjective measures across a variety of indicators. This trend was consistent across quality, quantity, and absenteeism measures. Further, work samples and job knowledge tests are associated with larger ds than performance ratings or measures of absenteeism. Analysis of Hispanic-White standardized differences shows that they are generally lower than Black- White differences in several categories. (PsycINFO Database Record (c) 2010 APA, all rights reserved)     

Relations among criteria of police performance.

Job performance measures consisting of 35 objective indices and ratings on 8 behaviorally anchored rating scales (BARS) were available for 795 nonminority (mean age, 29.8 yrs) and 147 minority (mean age, 28.2 yrs) police officers. Eight of the 35 objective measures, plus age and job tenure, were used as predictors of the sum of the 8 BARS. Identical predictor sets validly forecast supervisory ratings in both minority and nonminority groups whether or not age and tenure were included. Unit weights were inferior to regression weights in both groups. It is concluded that supervisory ratings are linearly predictable from objective performance indices for both minority and nonminority subordinates, a finding that comports with civil rights legislation and recent US Supreme Court decisions. (19 ref) (PsycINFO Database Record (c) 2010 APA, all rights reserved)     

The five-factor model of personality and managerial performance: Validity gains through the use of 360 degree performance ratings.

This study investigated the usefulness of the five-factor model (FFM) of personality in predicting two aspects of managerial performance (task vs. contextual) assessed by utilizing the 360 degree performance rating system. The authors speculated that one reason for the low validity of the FFM might be the failure of single-source (e.g., supervisor) ratings to comprehensively capture the construct of managerial performance. The operational validity of personality was found to increase substantially (50%–74%) across all of the FFM personality traits when both peer and subordinate ratings were added to supervisor ratings according to the multitrait–multimethod approach. Furthermore, the authors responded to the recent calls to validate tests via a multivariate (e.g., multitrait–multimethod) approach by decomposing overall managerial performance into task and contextual performance criteria and by using multiple rating perspectives (sources). Overall, this study contributes to the evidence that personality may be even more useful in predicting managerial performance if the performance criteria are less deficient. (PsycINFO Database Record (c) 2010 APA, all rights reserved)