Determine the required sample size for estimating a single population mean

Joumal of Occupational and Organisational Psychology (1998), 71, 161-164 Printed in Great Britain 1 6 1
© 1998 The British Psychological Society




Sample size in validity studies of personnel
selection

Jesus F. Salgado*
Departamento de Psicologia Socialy Bdsica, Universidad de Santiago de Compostela,
15706 Santiago de Compostela, Spain



This report presents the results of several meta-analyses in which samples in the
articles published in foumal of Applied Psychology, foumal of Occupational and
Organisational Psychologf and Personnel Psychology from 1983 to 1994 were integrated.
The findings show that in the last decade a small increase in sample size has been
observed in relation to sample size of studies published two or three decades ago.
However, the mean sample continues to be of a small size. This last finding
suggests that sampling error continues to be the most important artifactual error
for explaining the variability of vahdity coefficients. Some implications for the
confidence intervals and power analysis are discussed.


Validity studies of personnel selection need to be conducted in samples with a large
number of individuals, and this characteristic has been for decades one of the main
reasons preventing validity studies from being carried out. Frequently, sample size
has been considered as the first problem in conducting validational processes due
to the difficulty in obtaining samples with appropriate numbers. This belief was
reinforced by Ghiselli (1973), who found many samples with fewer than 50
participants. However, the most relevant research on sample size in validity studies
was conducted by Lent, Auerbach & Levin (1971). These researchers, compiling
validity studies published between 1954 and 1969, showed that the median of the
sample sizes was only 68 participants. In a more recent study, Monahan &
Muchinsky (1983), using the articles published in the journal Personnel Psychology
between 1950 and 1979, found that the mean sample size for nine occupational
groups ranged from 58 to 125 participants, and the standard deviation ranged from
31 to 173. The mean sample size for all occupational groups was 88. Furthermore,
in 60 per cent of the studies conducted in the 1960s and 1970s the sample size was
smaller than 100 participants. No further research has been carried out on this topic
since the Monahan & Muchinsky study.
Furthermore, the question of sample size is relevant for a theoretical reason,
concerning the validity generalization hypothesis (VGH; Schmidt & Hunter, 1977).
According to Schmidt & Hunter (1981), sampling error explains an average of 72
per cent of the variability of validity coefficients. However, Schmitt, Gooding, Noe
*Requests for reprints.
162 fesiis F. Salgado
& Kirsh (1984) found that sampling error only explained 8 per cent of observed
variance and the mean sample size for all predictors was 637 participants.
On the other hand, some editorial and political changes have occurred in the last
25 years, which could be relevant to the question of sample size in validity studies.
For example, in the 1960s the section on Validity Exchange in the Personnel
Selection, published in Personnel Psychology, was discontinued. In 1978 the Depart-
ment of Labor in the USA published the Federal Guidelines for Personnel
Selection, and since the 1980s stricter publication standards have been requested
by scientific journals. These reasons along with the appearance of VGH suggest
to some researchers that the sample si2e of validity studies might have been
augmented in the 1980s and 90s (e.g. Schmitt, personal communication, 1995). If
sample size has grown over the years, the relevance of sampling error for explaining
the variability of validity in personnel selection studies becomes less important and
the role of other artifactual errors might have become more relevant (e.g. criterion
reliability, range restriction, construct validity, etc.).
The main objective of this research is to determine if the number of participants
in samples of validity studies is now greater than 15 or 20 years ago. In order to
answer this question, the sample sizes of validity studies published in the three
major journals of industrial and organizational psychology were recorded and
analysed.

Method
Procedure
All criterion-oriented validity studies published between 1983 and 1994 in t\\e Journal of Applied
Psychology QAV), Journal of Occupational and Organisational Psychology (JOOP, formerly Journal of
Occupational Psychology) and Personnel Psychoto^ (PP) were identified and their samples recorded. A
total of 86 studies were obtained: 43 published in JAP, 17 in JOOP and 26 in PP. In total, 147
independent samples were used, 58 from JAP, 48 from JOOP and 41 from PP.


Results
The samples recorded were grouped using sample type (civil versus military) and
journal (JAP, JOOP and PP) as criteria. The samples composed of participants
assessed in public organizations (e.g. US Employment Service) were excluded from
these analyses because of their very low numbers. One hundred and eighteen were
civil samples, 45 from JAP, 44 from JOOP and 29 from PP. The maximum sample
had 1097 participants and the minimum sample had 25. These two samples were
published in JAP. In JOOP, the biggest sample had 399 individuals and the smallest
sample had 29. In the case of PP, the maximum and minimum samples had 629 and
42 participants, respectively. In total, the mean sample size was 153 participants,
being 157 for JAP, 102 for JOOP and 224 for PP. These results appear to show that
samples in PP have a slightly higher number than JAP samples. However, as the
mean is very much influenced by extreme values, and the range is very large, the
median offers a better picture of sample distribution. For PP the median was 196
participants, for JAP 122, for JOOP 70 and for the total 113. With respect to SD,
Sample si^e in personnel selection 163
in all cases it is larger than the median. All these results combined indicate that
sample distributions are not-normal, downwardly biased.
In the case of military studies, 29 samples were recorded in total, 13 from JAP,
4 from JOOP and 12 from PP. The maximum sample had 78 041 participants and
the minimum sample had 69. These two samples were in papers published in PP.
In the case of JAP, the maximum and minimum samples had 37 437 and 125
participants, respectively. In JOOP, the maximum and minimum samples had 1790
and 567 individuals, respectively. In total, the mean sample size was 5975, being
6683 for JAP, 1045 for JOOP and 6852 for PP. These results appear to show that
JAP and PP have samples with a similar mean, which is lower for JOOP. For PP
the median was 239 participants, for JAP 1036, for JOOP 913 and for the total it
was 450. With respect to SD, in all cases it was very much larger than the mean. As
in the civil samples, if in the military samples all results are combined the findings
show that sample distributions are not-normal, downwardly biased.
Comparing the civil and military distributions one can see that military samples
are a lot larger than civil samples, for both the mean and the median and also SD.
The average sample in civil studies (using 118 samples) is 39 times lower than the
average sample used in military studies. With respect to the median, the civil is 5
times lower than the military, and with respect to the SD, the civil is 106 times
lower. Therefore, the civil and military samples have very different distributions and
they are not comparable. As a consequence, it might not be appropriate to group
military and civil samples in the same meta-analytic study.


Discussion
The results of this study indicate that in the last 12 years the mean sample size for
validity studies carried out in civil settings has been around 153 participants. This
finding shows an increase in relation to the mean size found by Lent et al. (1971)
and Monahan & Muchinsky (1983). Similar findings are obtained for the median, a
relevant statistic in this case because it is not influenced by extreme values. Lent
et al. report a median of 68 participants while the present research shows a median
of 113. In other words, the samples currently used in validity studies of personnel
selection are remarkably higher than those used in previous decades. In research
published in the last decade, samples have approximately nearly twice as many
participants as 20 or 30 years ago. This finding suggests a relevant change in
research practice carried out in the field of personnel selection. An interesting
finding was that, in all analyses, the median sample size was lower than the mean.
This result indicates that the majority of studies used a number of participants
smaller than the mean.
The findings of this research also show the civil and military samples are not
comparable. With respect to the mean, the ratio between military and civil samples
is 39:1, and for the median the ratio is 5:1. Therefore, the grouping of military and
civil samples in a single sample may not be appropriate. If we take into account that
the mean sample is the most relevant variable to compute sampling error variance,
a masking effect could come about when civil and military samples are grouped.
164 fesus F. Salgado
The military samples would reduce the explanatory power of sampling error as a
source of variability in civil validity studies.
In connection with the problem of sample size as an artifactual error that can
explain a large percentage of the variance found in validity coefficients, the results
are consistent with the explanatory role attributed to sample size in the validity
generalization hypothesis. Although the sample size has grown in the last decade
compared to previous decades, samples continue to be small.
Some implications of the changes in sample sizes can be seen when thinking in
terms of confidence intervals for validity coefficients or in terms of power analysis.
For example, a selection procedure with an observed validity of about .30 (i.e. a
medium effect size), with N= 68 (Lent et al.'s median), the 95 per cent confidence
interval is approximately 24 correlation points wide (i.e. from .07 to about .55).
With N= S8 (Monahan & Muchinsky's mean), this interval is about 19 points
(from about .10 to .48). With N- 113 (median of civilian studies in this report), the
interval is about 17 points (from about .12 to .46). Therefore, it can be seen that we
are moving towards narrower confidence intervals, but there is still considerable
uncertainty associated with sample results.
Thinking in terms of power analysis, for r = .30 (a medium effect size), and
assuming that there is no range restriction and that the criterion reliability is
perfect, most of the studies in PP have a power of .99 (A^= 196), in JAP .92
(A^= 122) and in JOOP .72 (N= 70). From this point of view, studies published in
JAP and PP are almost certain to detect validity when it exists. However, this is still
not the case for JOOP. Perhaps legal changes in the USA, stricter publication
standards in American journals in connection with sample size, and differences in
company characteristics may be reasons for this difference in power.

Acknowledgements
This research was partially supported by the XUGA Grant No. 21104A95 from the Xunta de Galicia
(Spain). 1 am grateful to three anonymous reviewers for their suggestions.


References
Ghiselli, E. E. (1973). The validity of aptitude tests in personnel selection. Personnet Psychotogy, 26,
461-477.
Lent, R. H., Auerbach, H. A. & Levin, L. S. (1971). Research design and validity assessment. Personnet
Psychologf, 24, 247-274.
Monahan, C. 1. & Muchinsky, P. M. (1983). Three decades of personnel selection research: A
state-of-the-art analysis and evaluation. Journal of Occupational Psychotogy, 56, 215-225.
Schmidt, F. L. & Hunter, J. E. (1977). Development of a general solution to the problem of validity
generalization. Journal of Applied Psychology, 62, 529-540.
Schmidt, F. L. & Hunter, J. E. (1981). Employment testing: old theories and new research findings.
American Psychologist, 36, 1128-1137.
Schmitt, N., Gooding, R. Z., Noe, R. D. & Kirsch, M. (1984). Metaanalyses of validity studies
published between 1964 and 1982 and the investigation of study characteristics. Personnel Psychology,
37, 407-422.

Received 13 May 1996; revised version received 5 June 1997