Why Your Selection Assessment's Validity Coefficient Tells You Almost Nothing

What Taylor and Russell Figured Out 85 Years Ago That We're Still Ignoring

Here’s a question that should keep HR professionals and I-O psychologists up at night: Why do we still talk about assessment validity as if it’s the only number that matters?

Open any meta-analysis on personnel selection. Read any vendor’s technical manual. What do you see? Tables of correlation coefficients. “Structured interviews achieve r = .42!” “Biodata predicts with r = .38!” “Our personality assessment has a validity of .19!” As if that single number tells you whether the assessment is actually useful.

Spoiler: it doesn’t.

In 1939, Taylor and Russell published a paper demonstrating something profound: the utility of a selection assessment depends on three factors, not one. Yes, validity matters. But so do your selection ratio (how many applicants per position) and your base rate (how many would succeed without any selection).

The punchline? An assessment with r = .40 might improve your success rate by 5% in one context and 45% in another. Same assessment. Same validity. Completely different utility.

We’ve known this for 85 years.

And yet when was the last time you heard HR professionals and I-O psychologists talk about selection ratios alongside their validity coefficients? When did a practitioner ask “What’s our base rate?” before buying an assessment?

But here’s the uncomfortable truth: without knowing your selection ratio and base rate, that correlation coefficient tells you almost nothing about whether the assessment will actually improve your hiring.

So next time someone shows you a validity coefficient, ask about how the actual utility of this coefficient depends on your selection ratio and base rate.

If they can’t answer, you don’t have enough information to make a decision.

(For more on this applied to personality and intelligence in selection, see Sofia’s dissertation below).

References:

Sjöberg, S. (2014). Utilizing research in the practice of personnel selection: General mental ability, personality, and job performance (Doctoral dissertation, Department of Psychology, Stockholm University). Stockholm University. https://su.diva-portal.org/smash/person.jsf?pid=authority-person%3A90208

Taylor, H. C., & Russell, J. T. (1939). The relationship of validity coefficients to the practical effectiveness of tests in selection: Discussion and tables. Journal of Applied Psychology, 23(5), 565–578.