Banding in Personnel

1

Running head: BANDING IN PERSONNEL SELECTION WITHIN A CSEM FRAMEWORK

Banding in Personnel Selection Within a CSEM Framework Daniel C. Y. Kuang Biddle Consulting Group Jim Higgins Biddle Consulting Group

Cite as: Kuang, D. C.Y., & Higgins, J. (2008, April). Banding in personnel selection within a CSEM framework. In Hurtz, G. (Chair), Integrating Conditional Standard Errors of Measurement into Personnel Selection Practices. Symposium conducted at the meeting of the Society for Industrial and Organizational Psychology, San Francisco, CA.

Banding in Personnel

2

Banding in Personnel Selection Within a CSEM Framework

Methods of banding in personnel selection have received considerable attention (see e.g., Campion et al., 2001; Cascio, Goldstein, Outtz, & Zedeck, 1995; Schmidt & Hunter, 1995). More recently, however, advances in statistical methods have highlighted issues related to classical test theory (CTT) approaches to establishing bands that may affect their validity and utility (Bobko & Roth, 2004, 2005; Raju, Price, Oshima, & Nering, 2007). Indeed, the psychometric weaknesses associated with estimating the standard error of measurement (SEM) with CTT methods are widely understood (see e.g., Allen & Yen, 1979; Nunnally & Bernstein, 1994). Alternatives based on conditional standard error of measurement (CSEM) models are more accurate and have been available for over 50 years (Feldt, Steffen, & Gupta, 1985; Lord, 1984). This paper presents a more updated and contemporary approach to establishing score bands using CSEM methods. Classical Banding The history, philosophy, and rationale underlying banding as a method of increasing workforce diversity is well documented in the 1995 special edition of Human Performance (v. 8). Current banding methodologies are based on classical test theory (CTT) models of SEM where the SEM is assumed to be the same across the full range of test scores. The estimated SEM is used to establish the standard error of the difference (SED), which is the standard deviation of the difference in two independent scores. To ensure that scores in adjacent bands differ with a 95% confidence interval, the width of a band is set to 1.96×SED. Therefore, banding methods based on classical test theory are established based on one SEM, which assumes that reliability is the same across the full score range.

Banding in Personnel

3

CSEM-Based Banding More accurate psychometric models recognize that test reliability varies across the score range and conditional SEMs (CSEM) are estimated at each point along the score range (Feldt & Brennan, 1989; Qualls-Payne, 1992). Consequently, methods of banding within the CSEM framework are different than traditional methods. Instead of a single SEM assumed for all scores across a test, the SEM differs at each score across the score range within the CSEM framework. Traditional banding methods, which are based on one SEM, cannot be applied in the CSEM framework. This presentation addresses this challenge by demonstrating a new method of banding which was developed to work within the CSEM framework. CSEM-Based Banding Model The underlying logic of the proposed banding method is not new; it was developed in the standardized educational testing context (e.g. GRE). Much like banding in the CTT framework, the obtained bands in the proposed method must: (1) account for test unreliability and (2) be significantly different than adjacent bands. In operation, the process of establishing bands is driven by a simple rule: the lower-bound of a given band must not significantly overlap with the upper-bound of the band immediately below it. In keeping with traditional banding methodology, a 95% confidence interval is established for each band. Arguably, however, the confidence level can be adjusted to meet situational demands (Campion et al., 2001). Application of CSEM-Based Banding Model In application, the first band is established by the top score observed on a test1. The upper-bound of Band-1 is established by the top score and the lower-bound of Band-1 is

TopScore  1.96  CSEM TopScore . The next band is established by score-i with an upper bound 1

To avoid the banding issues detailed by Schmidt & Hunter (1995), the new banding model is an extension of the top-score-reference model.

Banding in Personnel

4

( i  1.96  CSEM i ) that is significantly less than the lower-bound of Band-1 ( TopScore  1.96  CSEM TopScore ). To identify score-i, an iterative procedure is required, whereby the upper-bound for a given score is computed based off of its observed CSEM and compared to the lower-bound of Band-1. Band-2, has a unique upper-bound ( i  1.96  CSEM i ) and lowerbound ( i  1.96  CSEM i ). The next band is established through an iterative search for score-j with an upper-bound ( j  1.96  CSEM j ) that is less than the lower-bound of Band-2

( i  1.96  CSEM i ). This process can be repeated until all test scores are exhausted. To demonstrate, Table 1 details this process with a hypothetical dataset. Table 1. Example—Establishing bands with CSEM. Test Score

Band

CSEM

95% CI-Lower Bound

95% CI-Upper Bound

20

1

0.20

20

19

1

1.10

20-1.96 X 0.2 = 19.61 19-1.96 X 1.1 = 16.84

18

1

1.20

18+1.96 X 1.2 = 20.35

17

2

1.30

18-1.96 X 1.2 = 15.65 17-1.96 X 1.3 = 14.45

16

1.40

16-1.96 X 1.4 = 13.26

16+1.96 X 1.4 = 18.74

15

2 2

1.50

15-1.96 X 1.5 = 12.06

15+1.96 X 1.5 = 17.94

14

2

1.60

14-1.96 X 1.6 = 10.86

14+1.96 X 1.6 = 17.14

13

2

1.70

13-1.96 X 1.7 = 9.67

13+1.96 X 1.7 = 16.33

12

2

1.80

12-1.96 X 1.8 = 8.47

12+1.96 X 1.8 = 15.53

11

2

2.00

11-1.96 X 2.0 = 7.08

11+1.96 X 2.0 = 14.92

10

3

2.10

10-1.96 X 2.1 = 5.88

10+1.96 X 2.1 = 14.12

9

3

2.10

9-1.96 X 2.1 = 4.88

9+1.96 X 2.1 = 13.12

19+1.96 X 1.1 = 21.16 17+1.96 X 1.3 = 19.55

In Table 1, the lower bound for Band-1, which is established by the top score (20), is 19.61. The next score with an upper bound that is less than 19.61, is 17 (19.55). Given this, Band-1 is comprised of all scores between 18 and 20. Applying this method, Band-2 is obtained; the scores range between 11-17. These obtained bands are graphed in Figure 1.

Banding in Personnel

5

Figure 1. CSEM-Based (95% Confidence Interval) Bands CSEM-Based 95%CI- Bands 0.5

Band-1 Band-2 Band-3

0.4

0.3

0.2

0.1

0 5

10 Test Score

15

20

Unlike SEM-based bands, the characteristic of the CSEM-bands vary. Applying the proposed methods, however, we are able to obtain bands that meet traditional banding requirements: (1) the bands account for test unreliability and (2) the scores within each band are significantly different than those of adjacent bands Conclusion Banding based on classical test theory methods suffer from a lack of precision and other psychometric issues (Bobko & Roth, 2004). The criticisms detailed by Bobko and Roth (2004) highlights fundamental flaws that may be addressed if bands are established within a CSEM framework. The application of CSEM in banding is new and no methods exist to properly establish bands. Applying existing methods within the CSEM framework is inappropriate. This presentation hopes to address this gap by providing a comprehensive method of banding within a CSEM framework.

Banding in Personnel

6

Reference Allen, M. & Yen, W. (1979). Introduction to measurement theory. Monterey, CA: Brooks/Cole. Biddle, D., Kuang, D. C.Y., & Higgins, J. (2007, March). Test use: ranking, banding, cutoffs, and weighting. Paper presented at the Personnel Testing Council of Northern California, Sacramento. Bobko, P. & Roth, P. L. (2004). Personnel Selection with top-score-referenced banding: On the inappropriateness of current procedures. International Journal of Selection and Assessment, 12, 291-298 Bobko, P. & Roth, P. L. (2005). Banding selection scores in human resource management decisions: Current inaccuracies and effect of conditional standard errors. Organizational Research Methods, 8, 259-273. Campion, M., Outtz, J., Zedeck, S., Schmidt, F., Kehoe, J., Murphy, K. and Guion, R. (2001) The controversy over score banding in personnel selection: Answers to 10 key questions. Personnel Psychology (Scientist-Practitioner Forum), 54, 149–185. Cascio, W., Goldstein, I., Outtz, J. and Zedeck, S. (1995) Twenty issues and answers about sliding bands. Human Performance, 8, 227–242. Feldt, L. S., & Brennan, R. L. (1989). Reliability. In R. L. Linn (Ed.), Educational measurement, 3rd edition. (pp. 105-146). Phoenix, AZ: American Council on Education/Macmillan Publishing. Feldt, L. S., Steffen, M., & Gupta, N. C. (1985). A comparison of five methods for estimating the standard error of measurement at specific score levels. Applied Psychological Measurement, 9, 351-361.

Banding in Personnel

7

Lord, F. M. (1984). Standard errors of measurement at different ability levels. Journal of Educational Measurement, 21, 239-243. Nunnally, J. C. & Bernstein, I. H. (1994). Psychometric Theory. New York, NY: McGraw Hill. Qualls-Payne, A. L. (1992), A comparison of score level estimates of the standard error of measurement, Journal of Educational Measurement, 29 (3), 213–225. Raju, N. S., Price, L. R., Oshima, T. C., & Nering, M. L. (2007). Standardized conditional SEM: A case for conditional reliability. Applied Psychological Measurement, 31, 169-180. Schmidt, F. and Hunter, J. (1995) The fatal internal contradiction in banding: Its statistical rationale is logically inconsistent with its operational procedures. Human Performance, 8, 203–214. U.S. Equal Opportunity Employment Commission, U. S. Civil Service Commission, U.S. Department of Labor, U.S. Department of Justice. (1978). Uniform guidelines on employee selection procedures. Federal Register, 43, 38295–38309.