Dimension and exercise variance in assessment center scores: A large-scale evaluation of multitrait-multimethod studies.

This study addresses 3 questions regarding assessment center construct validity: (1) Are assessment center ratings best thought of as reflecting dimension constructs (dimension model), exercises (exercise model), or a combination? (2) To what extent do dimensions or exercises account for variance? (3) Which design characteristics increase dimension variance? To this end, a large set of multitrait-multimethod studies ( N=34) were analyzed, showing that assessment center ratings were best represented (i.e., in terms of fit and admissible solutions) by a model with correlated dimensions and exercises specified as correlated uniquenesses. In this model, dimension variance equals exercise variance. Significantly more dimension variance was found when fewer dimensions were used and when assessors were psychologists. Use of behavioral checklists, a lower dimension-exercise ratio, and similar exercises also increased dimension variance. (PsycINFO Database Record (c) 2010 APA, all rights reserved)     

Does the assessment center scoring method affect the cross-situational consistency of ratings?

Examined the effect of 2 different assessment center scoring methods on the cross-situational consistency of assessor ratings. A scoring method that focused on dimensions yielded similar heterotrait-nonmethod and monotrait-heteromethod correlations compared with the more typical within-exercise procedure. As in previous research, dimension ratings correlated higher with different dimensions in the same exercise than with ratings of the same dimensions in other exercises. In light of this as well as other studies, the authors concluded that different changes appear necessary to increase convergent validity. (PsycINFO Database Record (c) 2010 APA, all rights reserved)     

Constructs and assessment center dimensions: Some troubling empirical findings.

Maintains that exercise design alone serves as weak evidence for construct validity due to the complexity of the typical assessment center rating and evaluation process. Evidence that assessment ratings accurately reflect the complex traits they purport to measure has not been provided to date. To explore this issue, the interrelationships among dimensional ratings between and within exercises in 3 assessment centers were examined. 559 candidates for upper-level management from a multinational firm, a civil service commission, and a retailer were assessed on managerial skills (including communication and interpersonal skills). Postexercise ratings of assessment dimensions were correlated and factor analyzed. The resulting factor pattern for all 3 organizations represented exercises rather than dimensions. In 2 of the organizations, the mean correlation among across-exercise ratings of individual dimensions was near zero. These findings suggest that assessment ratings do not measure the intended constructs. (20 ref) (PsycINFO Database Record (c) 2010 APA, all rights reserved)     

Multiple assessment center exercises and job relatedness.

P. R. Sackett and G. F. Dreher (see record 1982-31460-001) examined the internal construct validity for 3 different sets of assessment center data. Based on the failure of the exercise ratings to satisfy construct validity requirements, Sackett and Dreher concluded that assessment centers should not rely on a content validation strategy. The present authors question the finality of Sackett and Dreher's contention. In addition, clarification of the psychometric bases for multiple exercises in assessment centers is presented. The results of several studies that have examined the convergent and discriminant validities of internal assessment center ratings are discussed with respect to the expanded explanation of the purposes of multiple exercises. The causes and the impact of assessee variance across exercises are also considered. (12 ref) (PsycINFO Database Record (c) 2010 APA, all rights reserved)     

Revised Estimates of Dimension and Exercise Variance Components in Assessment Center Postexercise Dimension Ratings.

The authors reanalyzed assessment center (AC) multitrait-multimethod (MTMM) matrices containing correlations among postexercise dimension ratings (PEDRs) reported by F. Lievens and J. M. Conway (2001). Unlike F. Lievens and J. M. Conway, who used a correlated dimension-correlated uniqueness model, we used a different set of confirmatory-factor-analysis-based models (1-dimension-correlated Exercise and 1-dimension-correlated uniqueness models) to estimate dimension and exercise variance components in AC PEDRs. Results of reanalyses suggest that, consistent with previous narrative reviews, exercise variance components dominate over dimension variance components after all. Implications for AC construct validity and possible redirections of research on the validity of ACs are discussed. (PsycINFO Database Record (c) 2010 APA, all rights reserved)     

Large-scale investigation of the role of trait activation theory for understanding assessment center convergent and discriminant validity.

This study used trait activation theory as a theoretical framework to conduct a large-scale test of the interactionist explanation of the convergent and discriminant validity findings obtained in assessment centers. Trait activation theory specifies the conditions in which cross-situationally consistent and inconsistent candidate performances are likely to occur. Results obtained by aggregating correlations across 30 multitrait-multimethod matrices supported the propositions of trait activation theory, shedding a more positive light on the construct validity puzzle in assessment centers. Overall, convergence among assessment center ratings was better between exercises that provided an opportunity to observe behavior related to the same trait, and discrimination among ratings within exercises was generally better for dimensions that were not expressions of the same underlying traits. Implications for assessment center research and practice are discussed. (PsycINFO Database Record (c) 2011 APA, all rights reserved)     

A meta-analytic evaluation of the impact of dimension and exercise factors on assessment center ratings.

Two recent reviews have attempted to summarize findings quantitatively regarding assessment center (AC) construct-related validity (i.e., Lance, Lambert, Gewin, Lievens, & Conway, 2004; Lievens & Conway, 2001). Unlike these previous studies, which reanalyzed individual multitrait-multimethod (MTMM) matrices from previously published research, the authors recoded and combined past matrices into a single MTMM matrix. This matrix, comprised of 6 dimensions each measured by 6 exercises, was then analyzed, providing a more generalizable set of results. Both dimensions and exercises were found to contribute substantially to AC ratings. Specific dimensions (i.e., communication, influencing others, organizing and planning, and problem solving) appeared more construct valid than others (i.e., consideration/awareness of others and drive). Implications for AC design and practice are discussed. (PsycINFO Database Record (c) 2010 APA, all rights reserved)     

Situational specificity in assessment center ratings: A confirmatory factor analysis.

Assessment center ratings of eight abilities from each of five situational exercises were examined for their cross-situational consistency and discriminant validity. A series of confirmatory factor analyses revealed that the ratings were largely (if not totally) situation specific, and that assessors failed to distinguish among the eight target abilities. These results combined with previous research suggest that the assessment center method measures mainly situation-specific performance, not cross-situational managerial abilities. We suggest that the intended constructs might be better measured if more ability-related behaviors were elicited within each exercise and if the cognitive demands placed on assessors were reduced. (PsycINFO Database Record (c) 2010 APA, all rights reserved)     

Situation specificity of behavior and assessment center validation strategies: A rejoinder to Neidig and Neidig.

Responds to R. D. Neidig and P. J. Neidig's (see record 1984-13649-001) criticism of the present authors' (see record 1982-31460-001) conclusion that the failure to find consistency of assessment center dimension ratings across exercises constitutes a threat to the use of content validity to show job relatedness. The present authors contend that situational differences support the argument that the assessor judgment process is too complex to be justified on content-validity grounds alone. A reconceptualization of assessment centers is offered that relies on identifying critical managerial roles, designing exercises to simulate these roles, and evaluating effectiveness in each exercise. The conditions under which content validity is appropriate for showing the job relatedness of this type of center are specified. (11 ref) (PsycINFO Database Record (c) 2010 APA, all rights reserved)     

Are rating dimensions in assessment centers transparent for participants? Consequences for criterion and construct validity.

Investigates whether the extent to which participants recognize rating dimensions in assessment centers has an effect on performance. Results showed that people who more accurately identify dimensions perform better. Convergent validity of dimension ratings is greater when participants accurately perceive that the same dimension is being evaluated in 2 exercises. In the discussion, the author considers how the ability to adequately perceive rating dimensions (i.e., transparency) may influence criterion validity. (PsycINFO Database Record (c) 2010 APA, all rights reserved)     

Validity of assessment center ratings for the prediction of performance ratings and school climate of school administrators.

Describes the validation of an assessment center used to select school administrators on the basis of ratings of 153 school administrators who had participated in the center between 1976 and 1981. Behaviorally anchored rating scales were obtained from supervisors, teachers, and support staff; and measures of 7 school climate dimensions were provided by teachers, students, and support staff. Results show a significant relationship between an overall assessment center placement recommendation and supervisory, teacher, and support staff ratings on most performance dimensions. Climate measures were significantly related to few assessment-center ratings regardless of the source of the climate measure. It is suggested that the assessment center proved valid for a sample from widely distant geographic areas and school districts of differing sizes and levels. (16 ref) (PsycINFO Database Record (c) 2010 APA, all rights reserved)     

Prediction of obligatory exercise by exercise-related imagery.

Obligatory exercise is a compulsive behavior pattern in which exercise dominates daily life at the expense of other activities and lack of exercise produces withdrawal symptoms. This study examined the hypothesis that obligatory exercise is motivated similarly to eating disorders (cf. S. P. Coen & B. M. Ogles, 1993) and would be predicted by appearance-related imagery. Obligatory exercise (J. K. Thompson & L. Pasman, 1991) and exercise imagery (H. A. Hausenblas, C. R. Hall, W. M. Rodgers, & K. J. Munroe, 199) were assessed before and after a 10-week exercise program in 243 adult exercises. Regression analyses revealed that imagery accounted for 20% of the variance in obligatory exercise. Appearance-related imagery did not predict significantly obligatory exercise. Energy-related imagery was the strongest predictor. Obligatory exercise may not be as associated with appearance-related concerns as eating disorders, suggesting different motivational bases for these 2 behavioral patterns. (PsycINFO Database Record (c) 2010 APA, all rights reserved)     

The frame of reference approach as a solution to an assessment center dilemma.

This article describes how the frame of reference (FOR) approach to rater training for performance appraisal purposes (H. J. Bernardin, 1979; H. J. Bernardin & M. R. Buckley, 1981) was applied to traditional assessment center ratings and rater training. The method by which an FOR was established for the assessment center ratings is presented, including (a) definitions of dimensions of performance, (b) definitions of qualitative levels of performance within each dimension, and (c) specific behavioral examples of levels of performance on an item-by-item basis within dimensions. The resulting FOR was used to structure the training and certification of raters with the expectation of minimizing sources of rater unreliability. Implications for assessment center reliability, validity, and employee perceptions are also discussed. (PsycINFO Database Record (c) 2010 APA, all rights reserved)     

Number of assessment center dimensions as a determinant of assessor accuracy.

Undergraduates (N?=?131) were trained as assessors, who evaluated the performance of confederates in an assessment center simulation on 3, 6, or 9 dimensions. Number of dimensions significantly affected some assessment center judgments but not others. Ss who rated a small number of dimensions classified behaviors more accurately and made more accurate ratings than did Ss who rated a large number of dimensions. Number of dimensions did not affect the accuracy of assessors' observations nor the discriminant validity of their dimension ratings. Given these results and the findings of others (e.g., J. R. Hinrichs and S. Haanpera; see record 1978-20114-001), developers of assessment centers should limit the cognitive demands placed on assessors by, for example, minimizing the number of dimensions assessors are required to process. (PsycINFO Database Record (c) 2010 APA, all rights reserved)     

Validity of behavioral assessment for predicting military recruiter performance.

Determined the validity of an assessment center designed to help select US Army recruiters. 16 experienced recruiters assessed 57 soldiers entering the Army's recruiter school, and assessment ratings were correlated with subsequent performance in training. A composite of assessment ratings yielded corrected validities of near .50. In contrast, first-impression evaluations, ratings based on a structured interview, and scores on a test developed to predict success in military recruiting correlated near zero with performance in training. Results confirm that valid assessment does not require behavioral scientists as assessors, and analyses suggest that statistical composites of assessment ratings on individual exercises may be slightly more valid than clinical consensus judgments made after discussion of assessment performance. (24 ref) (PsycINFO Database Record (c) 2010 APA, all rights reserved)     

Effects of high- vs moderate-intensity exercise on natural killer cell activity

The effect of 45 min of high- (80% VO2max) vs moderate- (50% VO2max) intensity treadmill exercise on natural killer cell cytotoxic activity (NKCA) was investigated in 10 well-conditioned (66.0 +/- 1.9 ml.kg-1.min-1), young males (22.1 +/- 1.3 yr). Blood samples were taken before and immediately after exercise, with three more samples taken during 3.5 h of recovery, and analyzed for proportion of NK cells (CD3-CD16+CD56+) and NKCA. Exercise at 80% vs 50% VO2max resulted in a greater immediate postexercise increase in proportion of NK cells, followed by a 1-h and 2-h decrease below preexercise levels for both intensity conditions. NKCA rose significantly above preexercise levels following high- but not moderate-intensity exercise. For both exercise intensity conditions, NKCA tended to drop below preexercise levels by 1 h postexercise, rising back to preexercise levels by 3.5 h postexercise. When NKCA was expressed on a per-NK cell basis, however, no change relative to preexercise levels occurred following moderate-intensity exercise, while a significant increase occurred after 2-h recovery from high-intensity exercise. These data demonstrate that both high- and moderate-intensity exercise are associated with significant shifts in circulating proportions of NK cells which significantly influence interpretation of NKCA data based on assays using separated mononuclear cells.     

Cardiovascular effects of desipramine in children and adults during exercise testing

OBJECTIVE: In light of recent reports of sudden death in children being treated with desipramine (DMI), 3 of which were associated with physical exercise, the authors examined the effects of DMI on exercise in children and adults before and during DMI treatment. METHOD: Before treatment, 22 subjects (9 children, 13 adults) participated in a graded treadmill exercise test. Outcome measures included exercise tolerance, cardiovascular, and electrocardiographic parameters at progressive intensity levels and serum norepinephrine (NE) levels before and after exercise testing. Subjects were then treated with DMI, titrated to an average DMI dosage of 3 mg/kg, and underwent repeated exercise testing. RESULTS: DMI treatment was associated with a significant elevation of circulating NE levels in the pre-exercise assessment. Exercise tolerance was not affected by DMI, and blood pressure and heart rate effects were modest. The cardiovascular impact of DMI treatment was similar in children and adults. One 31-year-old subject exhibited a brief episode of ventricular tachycardia associated with exercise during DMI treatment. CONCLUSIONS: DMI has only minor effects on the cardiovascular response to exercise, and these effects do not appear age-related. However, DMI may increase the risk of exercise-associated arrhythmias in rare individuals.     

Alterations in energy balance with exercise

The doubly labeled water method for measuring average daily metabolic rate (ADMR) in combination with resting metabolic rate (RMR) allows one to assess the relation between exercise and energy balance. Three topics were included in an analysis of available data: 1) the limits of energy turnover in relation to physical performance for the achievement of energy balance, 2) the effect of an exercise intervention on daily energy turnover and its components, and 3) the effect of exercise on body composition. In the general population, physical activity level (PAL: ADMR/RMR) ranges between 1.2 and 2.2-2.5. There is no sex difference in the level of physical activity. Higher PAL values can be maintained by training and supplementation of the diet with energy-dense, carbohydrate-rich formulas. Exercise training does not influence spontaneous activity except in the elderly. In sedentary subjects, exercise training does not influence RMR when body weight is maintained. An exercise-induced increase in ADMR is about twice the training load. Exercise induces an increase in fat-free mass, especially in women, and a decrease in fat mass. Women tend to preserve energy balance and consequently loss of fat mass is significantly less.     

Propranolol and exercise as a screening test for growth hormone deficiency

There is a need for a safe, inexpensive, and reliable screening test for growth hormone (GH) reserve. Exercise has been utilized for this purpose but false-negative responses (inadequate GH release in non-GH-deficient patients) has limited the effectiveness of this stimulus as a screening test. Beta-adrenergic blockade (propranolol) was used to enhance the effect of exercise on GH release. Thirty-two non-GH-deficient children and five GH-deficient children were evaluated. All of the non-GH-deficient children responded to propranolol and exercise with serum GH levels exceeding 7 ng/ml. The peak serum GH levels in the five GH-deficient patients did not exceed 4 ng/ml. Propranolol and exercise appears to be an effective screening test for GH function.