Statistical power of tests of the situational specificity hypothesis in validity generalization studies: A cautionary note.

Evaluated the statistical power of the Callender-Osburn method for testing the situational specificity hypothesis in validity generalization studies. The Schmidt-Hunter 75% rule for testing the situational specificity hypothesis was also studied with regard to its sensitivity for detecting both Type I and Type II errors. Results show that both the Callender-Osburn procedure and Schmidt-Hunter 75% rule lacked sufficient statistical power to detect low-to-moderate true validity variance when sample size was below 100. (13 ref) (PsycINFO Database Record (c) 2010 APA, all rights reserved)     

Sampling variance in the correlation coefficient under indirect range restriction: Implications for validity generalization.

The authors conducted Monte Carlo simulations to investigate whether indirect range restriction (IRR) on 2 variables X and Y increases the sampling error variability in the correlation coefficient between them. The manipulated parameters were (a) IRR on X and Y (i.e., direct restriction on a third variable Z), (b) population correlations ρxy, ρxz, and ρyz and (c) sample size. IRR increased the sampling error variance in rxy to values as high as 8.50% larger than the analytically derived expected values. Thus, in the presence of IRR, validity generalization users need to make theory-based decisions to ascertain whether the effects of IRR are artifactual or caused by situational-specific moderating effects. (PsycINFO Database Record (c) 2010 APA, all rights reserved)     

Child/adolescent behavioral and emotional problems: Implications of cross-informant correlations for situational specificity.

Our purpose in this article was to determine the degree of consistency between different informants' reports of the behavioral/emotional problems of subjects aged from 1? to 19 years. We found 269 samples in 119 studies for meta-analyses of Pearson rs between ratings by parents, teachers, mental health workers, observers, peers, and the subjects themselves. The mean rs between all types of informants were statistically significant. The mean rs were .60 between similar informants (e.g., pairs of parents), .28 between different types of informants (e.g., parent/teacher), and .22 between subjects and other informants. Correlations were significantly higher for 6- to 11-year-olds than for adolescents, and for undercontrolled versus overcontrolled problems, although these differences were not large. The modest correlations between informants indicate that child and adolescent problems are not effectively captured by present-versus-absent judgments of problems. Instead, the variations between reports by different informants argue for assessment in terms of multiple axes designed to reflect the perceived variations in child and adolescent functioning. (PsycINFO Database Record (c) 2010 APA, all rights reserved)     

Validity generalization in the context of situational models.

A primary objective of validity generalization (VG) analysis is to decompose the between-situation variance in validities into (1) variance attributable to between-situation differences in statistical artifacts and (2) variance attributable to between-situation differences in (unidentified) situational moderators. This process is based on the assumption that the effects of statistical artifacts on validities are independent of the effects of situational moderators on validities. The present article seeks to question the independence assumption by theoretically integrating situational variables into the VG estimation process. It is shown that the independence assumption may be untenable because at least 1 artifact (criterion reliability) is a function of, rather than independent of, the situational variables that moderate validities. An alternative approach to VG analysis is recommended. This approach rests heavily on proactive research designs in which potential situational moderators are included in the generalizability analysis. (PsycINFO Database Record (c) 2010 APA, all rights reserved)     

A likelihood-based model for validity generalization.

The pair (ri, ni) is an observed correlation coefficient, ri, with sample size ni. Given s independent pairs, each arising from a setting in which the test and criterion measurement are bivariate normal, a mixture model may be used to make the following inferences about the unknown population parameters: (a) The number t of different population correlation coefficients, ρj, j?=?1,…, t, supported by the data, (b) point values of the ρj, (c) the proportion of ri associated with each ρj, and (d) the variance among the ρj all may be estimated from the s pairs. Maximum likelihood estimation equations are given. Approximate confidence intervals and tests may be constructed. Monte Carlo examples and a real-data example are provided. Procedures for correcting artifacts of range restriction and unreliability are briefly discussed. (PsycINFO Database Record (c) 2010 APA, all rights reserved)     

Validity generalization and situational specificity: A second look at the 75% rule and Fisher's z transformation.

We analyzed the James, Demaree, and Mulaik (1986) critique of validity generalization, contending that the work (a) is not relevant to the real-world use of validity generalization in organizations, (b) has overlooked the bulk of the evidence against the situational specificity hypothesis, and (c) has confused the processes of hypothesis testing and parameter estimation in validity generalization and has made incorrect statements about the assumptions underlying both. In addition, the critique of the 75% rule is a statistical power argument; the procedures for use of confidence intervals are erroneous; there is no double correction of artifacts in validity generalization; the bias in the correlation (r) and the sampling error formula for r is well-known, trivial in magnitude, and has no empirical significance; and the use of the Fisher's z transformation of r in validity generalization studies and other meta-analyses creates an unnecessary inflationary bias in estimates of true validities and provides no benefits. (PsycINFO Database Record (c) 2010 APA, all rights reserved)     

Validity generalization and situational moderation in the life insurance industry.

Investigated the effect of company differences on the validity generalization of the Aptitude Index Battery, a noncognitive, weighted, biographical inventory used in the selection of life insurance agents. Data analyzed for 12,453 agents contracted by 12 companies indicate that although the validity could be generalized across the 12 companies, up to 38% of the variance in coefficients across companies was due to company differences. Further analyses of 2 a-priori company groups supported the hypothesis of situational moderation of validity coefficients. Utility estimates suggested that the net gain in productivity for one company group was about 50% above that expected in the other group. (12 ref) (PsycINFO Database Record (c) 2010 APA, all rights reserved)     

Two new procedures for studying validity generalization.

Several Monte Carlo studies examined the accuracy of 2 new procedures in estimating population true validity mean and variance. Results indicate that 1 of the new procedures provided slightly more accurate estimates than the procedures of F. L. Schmidt and J. E. Hunter (see record 1978-11448-001) and J. C. Callender and H. G. Osburn (see record 1981-00257-001). From a practical point of view, however, the estimates from the various procedures were quite comparable. (15 ref) (PsycINFO Database Record (c) 2010 APA, all rights reserved)     

Incremental validity of situational judgment tests.

Using 3 different samples, the authors assessed the incremental validity of situational judgment inventories (SJIs), relative to job knowledge, cognitive ability, job experience, and conscientiousness, in the prediction of job performance. The SJI was a valid predictor in all 3 samples and incrementally so in 2 samples. Relative to the other predictors, SJI's partial correlation with performance, controlling for the other 4 predictors, was superior in most comparisons. Subgroup differences on the SJI also appear to be less than those for cognitive ability and job knowledge, but greater than differences in conscientiousness. The SJI should prove to be a valuable additional measure in the prediction of job performance, but several additional areas of research are suggested. (PsycINFO Database Record (c) 2010 APA, all rights reserved)     

Expectancy theory and alcohol: The situational insensitivity hypothesis.

Examines the role of alcohol-related expectancies as predictors of adult and adolescent problem drinking and offers a hypothesis as to why expectancy factors lacking in situational specificity predict problem drinking so successfully. It is argued that problem drinkers rely excessively on alcohol to obtain reinforcers, failing to employ different behaviors as a function of different stimulus contexts. Problem drinkers are seen as showing situational insensitivity in this overreliance. Thus, the situation-general nature of the expectancy factors may account for their success in predicting the situation-general behavior of problem drinking. (PsycINFO Database Record (c) 2010 APA, all rights reserved)     

Reliability and validity of the situational interview for a sales position.

The reliability and validity of a situational interview (e.g., Latham, Saari, Pursell, & Campion, 1980) were examined for a sales position. In a pilot study, the interrater reliability of the interview was higher than that typically observed for interviews (r?=?.84). However, the estimate was probably inflated because the reliability was computed on the same data used in the item analysis phase of interview development. In a predictive validation study, the situational interview was shown to be valid (r?=?.45) in the prediction of sales productivity. After correcting for attenuation in the criterion, a validity coefficient of .47 was observed. Future research directions with respect to the situational interview are discussed. (PsycINFO Database Record (c) 2010 APA, all rights reserved)     

Face validity in psychological assessment: Implications for a unified model of validity.

Comments on S. Messick's (see record 1996-10004-001) paper on the validity of psychological assessment. R. F. Bornstein argues that the concept of face validity is missing from Messick's discussion of the validity literature and that no model of unified validity can be complete without it. According to Bornstein, integrating the concept of face validity into Messick's framework would not be difficult as it is directly related to at least 4 of the 6 components of unified validity described by Messick. (PsycINFO Database Record (c) 2010 APA, all rights reserved)     

A note on validity generalization procedures.

Reviews validity generalization (VG) procedures and finds them to be subject to the logical fallacy of "affirming the consequent." It is argued that alternative models may explain variation in validity coefficients as well as the cross-situational consistency model espoused by many users of the VG approach. Moreover, some of the assumptions that form the statistical foundation of VG work are open to question. It is suggested that Fisher z transformations, which remove most of these problems and usually produce more conservative estimates of the degree to which sampling error may account for variability in correlations, be used. It is also recommended that a more stringent criterion than the 75% decision rule for rejecting the situational specificity hypothesis be used. (64 ref) (PsycINFO Database Record (c) 2010 APA, all rights reserved)     

An unbiased correction for sampling error in validity generalization studies.

A correction for variance due to sampling error is part of all the statistical procedures used to study validity generalization. This correction is analogous to estimation of a variance component in a random effects analysis of variance. The correction usually has a large influence on the results of validity generalization analyses. James, Demaree, and Mulaik (1986) recently questioned the correction for sampling error, suggesting that several of the assumptions required for its derivation are questionable. In this article, I provide an alternative correction for sampling error that is shown to be (exactly) unbiased. A comparison of the calculations involved in the computation of the unbiased correction with the calculations involved in the calculation of the usual correction suggests that the 2 corrections will rarely differ substantially. Hence, the usual correction is not seriously biased. (PsycINFO Database Record (c) 2010 APA, all rights reserved)     

A meta-analysis of the situational interview: An enumerative review of reasons for its validity.

The premise underlying the situational interview (SI) is that intentions predict behaviour. The distinguishing characteristics of the SI include the depiction of a dilemma in the interview question, the use of an item scoring guide, and summative scoring of items to yield a total interview score. A meta-analysis of SI studies indicated that the weighted mean observed criterion-related validity of the SI is .35 (n?=?1010), and the mean corrected criterion-related validity is .47. An enumerative review of the literature revealed that the SI is applicable across a diversity of jobs, participants, performance criteria, and countries. The criterion-related validation evidence for the SI is explained in terms of content, construct, and incremental validity, freedom from interviewer biases, and practicality. The article concludes with a comparison of the SI to interviews that measure either past behaviours or future intentions where the interview question does not contain a dilemma or scoring guide, and performance dimensions rather than each individual item are scored. (PsycINFO Database Record (c) 2010 APA, all rights reserved)     

Bootstrap estimates of standard errors in validity generalization.

Bootstrapping is introduced as a method for approximating the standard errors of validity generalization (VG) estimates. A Monte Carlo study was conducted to evaluate the accuracy of bootstrap validity-distribution parameter estimates, bootstrap standard error estimates, and nonparametric bootstrap confidence intervals. In the simulation study the authors manipulated the sample sizes per correlation coefficient, the number of coefficients per VG analysis, and the variance of the distribution of true correlation coefficients. The results indicate that the standard error estimates produced by the bootstrapping procedure were very accurate. It is recommended that the bootstrap standard-error estimates and confidence intervals be used in the interpretation of the results of VG analyses. (PsycINFO Database Record (c) 2010 APA, all rights reserved)     

Development of inhibition during childhood: Evidence for situational specificity and a two-factor model.

The inhibition of 99 children was observed from the start of preschool through Grade 1 in multiple settings: adult strangers, dyadic play with unfamiliar and familiar peers, and regular free play in class. A multisetting–multimethod–multioccasion analysis revealed (a) a high longitudinal stability of inhibition toward strangers and a medium stability of inhibition in class, (b) a decreasing consistency between inhibition in class and inhibition toward strangers, (c) an increasing consistency between inhibition in class and being ignored or rejected by classmates, and (d) no detrimental effect of children's inhibition toward strangers on their dyadic play with familiar peers. These results are discussed in terms of a 2-factor model of inhibition that is linked to J. A. Gray's (1982; see also PA, Vol 75:35718) concept of the behavioral inhibition system. It is assumed that both unfamiliarity and social–evaluative concerns contribute to individual differences in inhibition in childhood. (PsycINFO Database Record (c) 2010 APA, all rights reserved)     

Achievement motives and individual differences in situational specificity of behavior.

Based on the different images projected by research on high- and low-anxious individuals and on the kinship between the anxiety concept and the motive-to-avoid-failure concept, differences between achievement-oriented and failure-threatened individuals in responsiveness to situational variation were predicted. 167 Norwegian 7th graders participated and were administered the Achievement Motives Scale. The hypothesis received support from persistence data from 2 situations: The intraindividual variation in persistence (time) across the situations was greater among achievement-oriented than among failure-threatened Ss. These differences could not be explained by achievement motivation theory. Results also illustrate that there was no inherent contradiction between high intraindividual variability and high correlations between behavior measures from different situations. (39 ref) (PsycINFO Database Record (c) 2010 APA, all rights reserved)     

Bias in validity generalization variance estimates: A reply to Hoben Thomas.

In a recent article H. Thomas (see record 1989-10691-001) derived the expected value of the true validity variance estimate used in validity generalization studies. Based on computations of the expected values for certain scenarios, Thomas made a number of critical assertions regarding the variance estimate. This article shows that Thomas's arguments regarding deficiencies in the variance estimate used in the validity generalization studies are misleading. Contrary to Thomas's extremely negative assessment of the situation, there is no really convincing reason to doubt or abandon the estimates of true validity variance obtained in applied research from the Callender-Osborn and other closely related methods. Rather, there is strong evidence to indicate that populations of true validities with meaningful differences in mean and variance can be reliably distinguished, provided that a sufficient amount of base data are available. (PsycINFO Database Record (c) 2010 APA, all rights reserved)     

Predictive validity of situational constraints in general versus specific performance domains.

The stability, predictive validity, and construct validity of constraints were investigated in 3 studies. College students' situational constraint scores pertaining to a more specific performance domain (specific classes) better predicted a similarly specific criterion measure (class grade) relative to constraint scores pertaining to an aggregate performance domain (all classes) and correlated with a global criterion measure (semester grade point average or GPA). In 2 other studies, it was found that self-reported constraints were independent of attributional style. Finally, data from a 3rd study indicated that goal difficulty mediated the constraint–performance relationship. These data suggest that the criterion-related validity of self-report measures of constraints might be attenuated when accounting for global success criteria relative to more specific component success criteria; constraints may be less prone to cognitive-style correlates than heretofore assumed; and the constraint–performance relationship may be mediated by more proximal motivational constructs such as aspiration level. (PsycINFO Database Record (c) 2010 APA, all rights reserved)