Overview

Teaching: 0 min
Exercises: 0 min

Questions

• What is the problem or research question(s) the study will address?

• What are the sources of variability?

• What factors should be considered when determining sampling strategies?

• How will the data be analyzed?

Objectives

• Develop a good experimental question and formulate a testable hypothesis it.

• Name the known or possible sources of variation for a specific study and use design structures for controlling “nuisance variables” such as blocking techniques.

• Describe the population under study and select sampling techniques to maximize the accuracy of your intended inferences.

• Draw your experimental design factors and identify the statistical tests and analytic approaches that will be used.

The statistical test and the analysis must be determined in the planning stages of an experiment.

State the research problem or question

Clearly identify the research problem or question that the experiment will address. What are some sources of variability in the experiment?

• An initial task in planning an experiment is to clearly identify the research problem or question(s) that the experiment will address. This is not a simple matter, but a developed skill that comes with practice.
  • Significant, worthwhile problem or research objective
  • Testable questions or hypotheses (If __, then __)
• What are some main sources of variability in the experiment?
  • Will want to find ways in your design to reduce variability

Describe the population

Define the population to be sampled for the experiment.

Describe how you will sample the population

Samples should be representative of the larger population so that one can draw inferences How widely will your results apply? (what is the scope of inference?)

How will you sample?

• Random sampling of units from the population to control for unknown factors of variance. A random sample means that chance determines the units within the sample.
• Stratified sampling could be a means to more efficiently ensure desired balance of subgroups or strata.
• Cluster sampling: sample intact groups from the population.

Will you randomize? And how will you randomize?

• Randomization of the assigning of units to treatment is a fundamental means for helping to control against selection bias (Fischer, 1935). Here, we are not talking about a random sample, but rather the random assignment of treatment. You want chance to determine the allocation of treatment to yield effect estimates that are unbiased and reliable. An observational study will need to involve greater efforts to control for unknown factors, which will complicate the analysis and subsequently require a very large sample. But with randomization, you are helping to ensure that the variables that can contaminate your results are at least equally balanced between your experimental groups.

• Seek ways to ensure proper randomization. For example, use a computer program or random number tables. Your experimental protocol should document how it will randomize units.

• Determine the type of randomization you will use, such as simple, blocked randomization, stratified randomization, or cluster randomization.

Will you use blocking or stratification?

• Methods of blocking or stratification can increase the precision of your experiment by helping to account for known variation among groups of units.

• Blocking can be used to ensure the number of units assigned to each condition is essentially balanced. It can increase precision if there is more variable between the subgroups/blocks than within.

• Stratification can be used to ensure the balance between conditions for a particular factor (e.g., sex, age).

How large will your samples be?

• Determination of the sample size that is needed is important to good experimental design because it involves financial costs and ethical issues.

• Underpowered experiments raise the risk of making a Type II error, which involves failing to reject the null hypothesis when in fact it is not true, or in other words, failing to find a difference when there actually is a difference. Confidence in the findings is diminished when the experiment is underpowered, unnecessarily increasing the risk that the study will have to be repeated, which comes at a large financial cost.

• Overestimation of the number of needed units also can increase costs through the extra money spent on unnecessary units and the researcher time involved.

• Ethics: strive to reduce the number of animals used and refine the severity

• Note that sample size should mostly be determined by the number of randomized (experimental) units.

• Sample size is affected by the effect size and the variability. [demonstration with R in following lesson?]

Perform power analysis to determine sample size.

• An a-priori power analysis allows you to decide the necessary sample size to detect a specified intervention effect.

• Post-hoc power calculations falsely assume that the sample ES represents the population ES.

• The size of the effect is the key thing you are after with your experiment, not a p-value. What is the minimum ES that would be considered important by the field?

• Your analytic approach will influence your power calculations.

• Consider the implications of your design type to the power analysis.

Specify the experimental design

1. What are the experimental units?
2. Describe the variables: background, constant, primary, uncontrollable
3. Treatment structure: Drawing a design template is a good way to view the structure of the design factors. Understanding the layout of the design through the visual representation of its primary factors will greatly help you later to construct an appropriate statistical model.
4. Design structure (i.e. blocks, completely randomized design, randomized complete block design, etc.)

Decide what statistical tests will be conducted and how the data will be analyzed The statistical test and the analysis must be determined during the planning stages of an experiment.

• Blocking and stratification have to be taken into account in your data analysis.

See this checklist for preparing a proposal and conducting a study.

Challenge

Sketch out a timeline for your experiment. When do you plan to start your experiment? When is your grant due? When will you start a pilot or exploratory study? When will you consult a statistician?

Key Points

• A good experimental question is one that is worthwhile answering and that raises one or more testable hypotheses given constraints such as time, resources, etc. (Fry, p460)

• Identify and categorize the experimental units and the other variables in the study (e.g., background, constant, primary, uncontrollable) [Should there be discussion of measurement?]

• Good experimental design is strategic in its sampling methods by considering randomization, blocking or stratification methods, and sample size

• Good design planning also identifies the statistical tests and analysis while considering other study aspects (e.g., hypotheses, variables, design structures, and samples).