This article is another in a series that discusses research methods frequently used in prehospital and disaster research. A common type of submission to Prehospital and Disaster Medicine is research based on a pre-test and post-test evaluation of an education curriculum, triage scheme, or simulation training method. This is particularly true of studies comparing or proposing validation of mass-casualty triage algorithms.
Pre-test and post-test research is one of many forms of quasi-experimental design. The term “quasi” means resembling experimental research, but does not imply that the quasi-experimental method is true experimental research. An example of quasi-experimental design is the testing of a new mass-casualty triage system by selecting a group of Emergency Medical Services (EMS) personnel and first having the group participate in a pre-test session based on triage scenarios, participate in a training for a new triage method, and then compare post-test results with pre-test scores. If post-test scores are above the pre-test scores, one assumes the triage training was successful.
Pre-test and post-test design is also used in evaluations of participants attitudes or perceptions relative to an event or to assess comfort in applying the information presented in a training session or with introduction of new concept (acceptance and efficacy study). One would assume that an increase in knowledge or positive attitude that is evident in better scoring on a post-test compared to a pre-test implies better knowledge or perception relative to an intervention applied after the pre-test.
An advantage of a pre-test and post-test study design is that there is a directionality of the research, meaning there is testing of a dependent variable (knowledge or attitude) before and after intervention with an independent variable (training or an information presentation session). This appears to be similar to classic experimental design, yet because participants in the study are most often not randomly assigned, quasi-experimental design is also a correlation (non-experimental) design. Because quasi-experimental research is not truly experimental in design, outcome causality cannot be determined, rather associations between interventions and outcomes are made.
As far back as the 18th Century, pre-test and post-test research methods have been used in many fields, including medicine-nursing, health, mental health, and education. The method has remained in common use because it is a rapid, convenient method to assess a target group to which an intervention has been applied. The literature base is rich with pre-test/post-test studies, which allows for comparison of these studies and meta-analysis of previously published work of this form. Pre-test and post-test evaluation also allows for immediate assessment of an intervention (such as a simulation session) and provides a means for rapid refinement of instructor teaching or simulation technique. In addition to being a convenient research method, pre-test and post-test design allows for statistical analysis of data using established statistical methods.
Pre-test and post-test design based on purposeful sampling allows for assessment of specific representatives of a population of interest, but not of the population as a whole. For example, if one wishes to evaluate the effect of a simulation session on the knowledge of a disaster Emergency Medical Team, that team can be included as the participants in a simulation exercise in which a pre-test and a post-test is used to evaluate results. But, the results from such an evaluation are only valid for the Team tested and not other Emergency Medical Teams.
In the 1960s, the validity of quasi-experimental design came into question with a number of papers published that evaluated the various forms of this type of research. Since that time, limitations of pre-post-test study design have been identified. As noted above, the participants in these types of studies are rarely selected by random sampling and represent a convenience or purposeful sample. The lack of a randomized recruitment of participants represents non-probalistic sampling, and therefore, results of such a study can only be applied to the participants and not a general target population. The use of testing, in itself, may add bias to a study. A pre-test will likely sensitize those taking it to the test itself and alert participants to the limited material required to score better on a post-test rather than acquiring adequate general knowledge for the subject of interest. This is a particular problem when the pre-test and post-test are the same or similar. Pre-testing also allows for participants to become more familiar with terminology and allows for ease in taking and scoring higher on a post-test. Another limitation of pre-test and post-test design is the phenomenon of statistical regression or the tendency of a group to move to a common mean as an artifact of repeated testing. In other words, those that scored poorly on pre-testing have nowhere to go but up in score and those that scored high in pre-testing have nowhere to go but down in score of the post-test. Other limitations of pre-test and post-test design include knowledge or attitude “decay,” or changes in retaining information or skills that occur with time. For example, a knowledge-based pre-test and post-test study may show good initial results, but without application of the knowledge gained, concepts will be lost (forgotten) with time unless applied on a frequent basis. This is an even greater problem for attitude assessment pre-test and post-test studies in which attitudes can change rapidly based on personal experience and external stimulants (media, social interactions) with loss of positive results of an intervention over time.
There are a number of methods that can be used to improve validity of pre-test and post-test study designs. One obvious strategy is to select a target group (for example paramedics in a system) and randomly select a group of study participants and randomly select a group of controls. Both the study group and control group would then take the pre-test and post-test at the same time interval, with only the study group receiving the intervention (example, a simulation session). Comparing testing scores for the study and control group addresses some limitations inherent in testing validity. Another method to improve validity is to design a study with a pre-test, immediate post-test, and later post-test (usually six months following the intervention) to allow for consideration of learning or attitude decay and on-going external stimulation. Using different questions relative to general knowledge acquisition or attitude on a pre-test and post-test will also improve validity. Important is that both the pre-test and post-test are validated for showing accuracy in measuring the outcomes of interest prior to being used in the study. Tests should be scored consistently, preferably by a non-biased scorer (grader) who ideally is blinded to the participants for whom the tests apply and is not one who designed or organized the intervention session.
Finally, application of statistical test for evaluation of pre-test and post-test results should be appropriate. Essential is the use of 25% and 75% quartiles for ordinal data medians (such as Likert Scale data) and 95% Confidence Intervals for means and proportions. While probability statistics such as t-Tests and Chi-square analysis may show statistical significance, overlapping of the ranges in measures of central tendency (confidence intervals or quartiles) of the mean or median show a lack of clinical significance and poor practical application for research results.
In summary, quasi-experimental design has been a common research method used for centuries. Pre-test and post-test design is a form of quasi-experimental research that allows for uncomplicated assessment of an intervention applied to a group of study participants. Validity of pre-test and post-test studies is difficult to achieve as the research design has inherent flaws, but strategies such as use of randomization, limiting internal and external bias, and appropriate application of basic statistics allow a researcher to make associations in outcome measures with this popular study design.
