2/1/05

---

Prefatory Note: The following document clarifies some issues that have arisen around the use of samples to assess both SLOA educational programs and ESOA administrative and service units.  In particular, there was an expectation created by Jim Nichols that any sample would need to include, at a minimum, 30 percent of the population from which it was being taken.  In light of the widely perceived unmanageability of such a requirement, the assessment coordinators, in consultation with Fran Leach, Dean of Instruction for Stanton/Wilmington and Stephanie Smith, Acting Assistant Vice President for Planning and Assessment, conducted extensive research on the matter to determine more reasonable, yet adequately valid and reliable, sampling standards.  What follows is the result of that work.  It is organized into a series of "General Principles" and "Specific Guidelines."

General Principles of Sampling for Assessment Purposes

---

Specific Guidelines for Sampling for Assessment Purposes

1. Remember the general “rule of thumb:” target population less than 30, include all members of the population in the study; population equal to or greater than 30, sampling is acceptable. 
2. A sample of 30 subjects is only a suggested minimum; sample size should be as large as possible, balancing size against time, money, and energy available to conduct the study.  The smaller a sample is, the greater are the chances of selecting subjects who do not, as a group, adequately represent the target population.  This is known as “sampling error.”
3. How a sample is selected is at least as important as its size in determining how representative of the target population it is.  The best way to assure the representativeness of a sample is by selecting it randomly.  In “simple” random selection, all members of the target population have an equal chance of being selected.  This is done by numbering the members of the population and then picking, at random, members until the sample size is reached.  Texts on social and educational research methods typically have tables of random numbers in an appendix for just this purpose.  Also, Microsoft Excel’s “Data Analysis Tool Pak” add-in feature can often perform random selection of data entries.
4. A somewhat different but related technique is to use systematic sampling.  In this approach, sample subjects are selected using some selection rule that is applied in a systematic (i.e., consistent) way.  For example, selecting every n^th person (e.g., every 10^th person, or every 3^rd and 8^th person) out of the population is a form of systematic sampling.  To use this approach, however, it must be assured that the way members of the population are ordered does not involve some sort of ranking that is of consequence to the study (for example, grade point average).
5. The random and systematic selection approaches described above—either one—will work fine as long as there are not any important differences among members of the target population that you want to control for.  For example, if such socio-economic (“S.E.S.”) characteristics as sex, age, income, race or ethnicity matter to how you expect your results to appear, then it is not sufficient to use simple random selection.  You must first stratify, i.e., separate, your population into the subgroups you wish to study separately.  Then, members of the subgroup can be selected randomly, usually up to a proportion of the sample which roughly approximates the subgroup’s proportion of the population.  Other examples of ways to stratify a population might be: day vs. evening students, in-class vs. distance classes, faculty vs. staff, etc.
6. An advantage of systematic selection as applied to stratified populations is that once the population is stratified, the same type of system can be used to select members of the sample, while simultaneously preserving the proportions of the subgroups to the whole population.
7. Sampling for purposes of assessing student learning is different from sampling for purposes of doing a survey.  For most purposes of assessing student learning at the College, the only ways of stratifying students will be day/evening and in-class/distance.   When conducting a survey, especially an attitude or opinion survey, however, the S.E.S. characteristics of the population—and survey sample—may become important.  Furthermore, issues such as the timing of the survey can influence, i.e., bias, the results.  For example, surveying students about their level of satisfaction with the registration process will be affected by when the survey is conducted: attitudes may differ widely during the “open” registration periods as opposed to other times in which registration can occur.  So unless it is the specific purpose of the survey (and the outcome that it is being used to measure) to gauge attitudes at such times, then even the times at which the survey is conducted must also be selected randomly.
8. Sampling for purposes of assessing educational support or administrative services can follow essentially the same guidelines as sampling for the direct assessment of student learning, as covered in items 1-6 above.  In such cases the meaning of “population” may be broader, to include not only students, but also faculty, staff, and even items or events, such as work orders, applications for admissions, etc., and the reasons for stratifying a population may be based on different factors.