Introduction
Many clinical trials and epidemiologic studies of neurodegenerative disease use repeated cognitive assessments for case ascertainment and quantifying treatment effects. However, recognition of material or familiarity with testing procedures may result in participants maintaining scores above recommended cut-points for impairment, thus escaping detection. Further complicating this picture, practice effects could differ by measure (Hickman et al., Reference Hickman, Howieson, Dame, Sexton and Kaye2000; Caban-Holt et al., Reference Caban-Holt, Bottiggi and Schmitt2005), time between administrations (Cooper et al., Reference Cooper2001), baseline cognitive status (Cooper et al., Reference Cooper, Lacritz, Weiner, Rosenberg and Cullum2004), and presence of neuropathology in the absence of dementia (Galvin et al., Reference Galvin2005). A recent meta-analysis of approximately 1,600 different effect sizes (Calamia et al., Reference Calamia, Markon and Tranel2012) found multiple variables accounting for the degree of change in cognitive test scores over time. More specifically, they found that age is negatively correlated with practice gain over time such that older adults benefit less from repeated administrations. Length of the test–retest interval and its relationship to practice effects also varied by measure, with some practice effects being eliminated after two to three years and others taking as long as seven years to extinguish. Effects of alternate forms were found to be inconsistent. For example, alternate forms were effective in reducing practice effects on verbal list learning measures. Other tests, such as verbal fluency, continued to show practice effects despite alternate form usage.
The Consortium to Establish a Registry for Alzheimer's Disease (CERAD) neuropsychological battery (Morris et al., Reference Morris1989) has been used in epidemiological studies and clinical trials internationally (Fillenbaum et al., Reference Fillenbaum2008). As a result, practice effects in this battery may have tremendous significance. However, only a few studies have examined practice effects on the CERAD battery (Zehnder et al., Reference Zehnder, Blasi, Berres, Spiegel and Monsch2007; Rosetti et al., Reference Rosetti, Cullum, Hynan and Lacritz2010; Burkhart et al., Reference Burkhart2011; Stein et al., Reference Stein2012). Although Stein et al. (Reference Stein2012) examined practice effects in cognitively normal adults, only the selected subtests of Verbal Fluency, Word List Recall, Word List Memory, and Word List Recognition were studied. They found no practice effects for Verbal Fluency and “small but significant gains” on Word List Recall, Memory, and Recognition. Zehnder et al. (Reference Zehnder, Blasi, Berres, Spiegel and Monsch2007) also examined practice effects after one year on the individual components of the German version of the CERAD battery in 374 normal controls and 95 patients with mild Alzheimer's disease (AD). Similar to Stein's (Reference Stein2012) results, they found small increases over time on Word List learning (0.41 ± 0.99), Word List Memory/Delayed Recall (0.33 ± 0.94), and Word List Recognition (0.33 ± 1.14). Figure copy also showed small but significant decreases over one year (−0.18 ± 1.24). No improvement was evident in patients with mild AD.
Of the previously listed studies examining practice effects on the CERAD, only two investigated the total score (Rosetti et al., Reference Rosetti, Cullum, Hynan and Lacritz2010; Burkhart et al., Reference Burkhart2011) developed by Chandler et al. (Reference Chandler2005) as an indicator of global cognitive status. Rosetti et al. (Reference Rosetti, Cullum, Hynan and Lacritz2010) examined 383 normal controls and 655 participants with AD to learn more about progression in AD over time. Participants contributed a baseline assessment and up to four annual follow-up visits. They found that, over a period of four years post-baseline, normal control participants gained an average of 2.8 points on the total score, whereas participants with AD showed a score decline of 22.2 points on average. Although the effect of attrition on degree of change over time was evaluated, this study only investigated attrition in the dementia group and did not assess dropout in normal controls. In addition, Burkhart et al. (Reference Burkhart2011) examined 57 healthy older adults who were at least 65 years of age using the German version of CERAD on study entry and then again at seven- and 90-day follow-up. Short intervals were used to approximate the typical time intervals utilized in postoperative cognitive dysfunction (POCD) research. On re-assessment at day 7, significant practice effects on Word List Learning (4 points) and Word List Recall (1 point) were observed. Mean scores for each Word List subtest (i.e. recall, memory, recognition) were stable at the 90-day follow-up and did not change from the scores obtained on the seven-day follow-up. Other CERAD subtests did not show a practice effect likely due to ceiling effects, as large proportions of their sample performed within one point of the maximum score on the Boston Naming Test (97%) and Constructional Praxis (89%). CERAD total scores increased significantly and were five points higher than baseline on seven-day follow-up and seven points higher than baseline on 90-day follow up. Although Rosetti et al. (Reference Rosetti, Cullum, Hynan and Lacritz2010) and Burkhart et al. (Reference Burkhart2011) examined practice effects on the CERAD total score, those studies did not follow participants beyond three months (Burkhart et al., Reference Burkhart2011), nor did they examine the relative contribution of the individual subtest components (Rosetti et al., Reference Rosetti, Cullum, Hynan and Lacritz2010), or the effect of participant dropout in normally aging individuals. People experiencing cognitive difficulties may withdraw from study participation, which may also serve to skew group performance.
In an effort to replicate and synthesize previous research findings on the CERAD test battery (Morris et al., Reference Morris1989), this study investigates the effect of practice on both the CERAD total score and the individual CERAD subtests administered yearly to a group of cognitively normal older men enrolled in a clinical primary AD prevention trial (Kryscio et al., Reference Kryscio, Mendiondo, Schmitt and Markesbery2004). Similar to previous studies, we expected to find practice effects on the total score and on the Word List subtests. To address the potential effect of attrition on increasing scores in longitudinally collected data, we examined the performance trajectory for those individuals who participated in all five assessment points (n = 182) and for those persons who completed at least two assessments (n = 308).
Methods
PREADViSE participants
The Prevention of Alzheimer's Disease by Vitamin E and Selenium (PREADViSE) trial is a companion study to the Selenium and Vitamin E Cancer Prevention Trial (SELECT) (Klein et al., Reference Klein2003) funded by the National Cancer Institute examining the prevention of prostate cancer in men over the age of 50 years. The methods used in the PREADViSE trial have been published previously (Kryscio et al., Reference Kryscio, Mendiondo, Schmitt and Markesbery2004; Caban-Holt et al., Reference Caban-Holt, Vellas, Fitten, Winblad, Feldman, Grundman, Giacobini and Kurz2006) and are described briefly here.
Recruitment for the PREADViSE trial took place from 2002–2008 at 130 clinical research sites also participating in the SELECT trial in the United States, Canada, and Puerto Rico. Men over the age of 60 years were invited to enroll in PREADViSE. Baseline exclusion criteria included neurological, psychological, and medical conditions or medications affecting cognition or physical ability to participate in evaluation. A subset of cognitively normal participants from eight SELECT and PREADViSE sites was recruited to a validation sub-study to assess the usefulness of a brief memory screening tool (Memory Impairment Screen) that was used in the overall study. Participants in the validation sub-study are collectively referred to as the Normal Aging Group (n = 563), and they comprise the sample from which the current analyses were drawn. PREADViSE participants were continuously enrolled in the Normal Aging Group throughout this period. Examinations were administered annually for a total of five years. Alternating versions of the word list (Rosen et al., Reference Rosen, Mohs and Davis1984; Ferris et al., Reference Ferris1997) were used each year in an attempt to mitigate practice effects.
Participants in the current analysis
Of the 563 participants in the Normal Aging Group, those who completed only one assessment (n = 249) were excluded from the current study as practice effects could not be determined. Participants with more than one assessment who were diagnosed with Mild Cognitive Impairment (MCI) or dementia as a result of their baseline Normal Aging Group evaluation (n = 6) were also excluded. Reasons participants contributed a single data point include a planned cross-sectional validation (n = 172). Specifically, some sites chose to participate only cross-sectionally and thus contributed only one assessment. Other reasons a single data point may have been contributed include withdrawal from the study after first assessment (n = 21) and clinical site closure (n = 56). Information was not collected regarding reasons for study withdrawal among those 21 participants who withdrew following their first assessment. Information regarding reason for withdrawal was also not collected throughout the study. Thus, the current sample consists of 308 participants who met criteria for inclusion in the Normal Aging Group sample, had at least two assessments, and were not clinically impaired at baseline. In order to investigate potential changes in the magnitude of practice effects related to study completion, a sensitivity analysis was conducted with the participants from this sample who provided data at baseline and all four follow-up visits (n = 182; “completers”) as well as those participants who did not complete all assessments (n = 126; “non-completers”). Among those classified as non-completers, participants contributed two (n = 27), three (n = 39), or four (n = 60) assessments.
Most non-completers provided annual consecutive assessments until withdrawing from the study; however, ten individuals labeled non-completers did not have consecutive visits. Two individuals missed the first follow-up but reappeared at the second follow-up, three participants missed the second follow-up but completed the third follow-up, and five individuals missed the third follow-up but participated in the fourth follow-up. Visits were numbered consecutively to indicate number of visits completed (and thus exposures to the tests), not as they occurred. All of the missed consecutive follow-up visits were due to clinical site scheduling conflicts.
Standard protocol approvals, registrations, and patient consents
Written informed consent was obtained from all study participants. All procedures were conducted with the approval of the University of Kentucky Institutional Review Board (IRB) as well as the IRB at each study site.
Analyses
A total score summing the raw subtest scores with the exception of the Mini-Mental State Examination (MMSE) was developed by Chandler et al. (Reference Chandler2005) and is used in the current analysis. The maximum possible total score is 100 and corrections for age, education, and gender are made to the total raw score. The raw corrected score is transformed to a T-score (i.e. a standardized score with a mean of 50 and a standard deviation (SD) of 10) according to the protocol provided in Chandler et al. (Reference Chandler2005)
To examine change over time, mean CERAD T-scores were analyzed using linear mixed models with visit as the only covariate; the first-order autoregressive covariance structure and a random effect for subjects nested within the study site account for the repeated measures. Differences in baseline characteristics between the completers and non-completers were examined with chi-square tests and two sample t-tests. All analyses were conducted using PC-SAS 9.3®.
Results
Participants
Participant characteristics are summarized by group in Table 1. There were no significant differences at baseline in age, education, CERAD T-score, or estimated full-scale IQ between the completers and non-completers. The 21 men who withdrew following their first assessment differed significantly from men who completed two or more assessments in terms of education (p = 0.0426) and estimated full-scale IQ (p = 0.009). Those who withdrew from the study after the first assessment had approximately one year less education and their estimated full-scale IQ was approximately 6.5 points lower than those who did not withdraw after the first assessment. It is unclear why education level and estimated full-scale IQ would be different between those who withdrew and those who did not withdraw. These differences are not explained by age or baseline global cognitive functioning given that the CERAD T-score did not differ significantly between the two groups. Men who were not included due to participation in the cross-sectional study or site closure were not included in this analysis because they did not actually withdraw from the study.
Participant characteristics
Note: aFull-scale IQ estimated with the American Version of the Nelson Adult Reading Test (AmNART; Boekamp et al., Reference Boekamp, Strauss and Adams1995).
Change in practice effects related to study completion
Baseline T-scores for all included participants (n = 308) were approximately normally distributed, with 3.9% of the participants scoring ≥ 65 and 7.5% scoring ≤ 35. At the fifth assessment (n = 182), 11.5% of the participants scored ≥ 65 and 4.4% scored ≤ 35, representing a clear shift in the distribution with a medium effect size. Mean T-scores increase significantly between visits 1 and 4 (Figure 1) with visit 4 mean scores 0.53 SD (4.8 points, Cohen's d = 0.44) above the baseline mean.
Mean CERAD T-scores by visit. Note: All pairwise differences are significant (α = 0.05) except visit 3 vs. visit 5 and visit 4 vs. visit 5.
In order to examine whether the trajectory of performance differed for those who completed the study, the analysis was repeated on completers. The results obtained were quite similar to those of the full sample. Their baseline T-scores followed a similar distribution (n = 182) with 2.2% of the participants scoring ≥ 65 and 7.7% of the participants scoring ≤ 35 (Table 1). Mean T-scores increased significantly between visits 1 and 5 (Figure 1). Visit 4 showed the greatest difference from baseline with mean scores 0.56 SD (4.9 points) above the baseline mean. Mean scores at visits 1 through 4 were examined between completers and non-completers to determine whether group performance differed by visit. No significant differences were observed (see Table S1 published as supplementary material online attached to the electronic version of this paper at http://www.journals.cambridge.org/ipg).
In both completers and non-completers, the performance trajectory changed slightly in our sample between the fourth and fifth assessments, with mean CERAD T-scores falling approximately one point (p = 0.056). This change was not statistically or clinically significant.
Contribution of individual subtest components to overall practice effects
Since study completion did not significantly impact performance trajectory, the sample, including both completers and non-completers, was used to examine the individual components of the CERAD T-score (Figure 2). Although statistically significant practice effects were observed on each subtest, increase was limited in terms of clinical significance and generally showed small effect sizes (i.e. Cohen's d ≤ 0.2; Cohen, Reference Cohen1992). Performance on Animal Naming increased significantly, although not monotonically, between visits 1 and 4 (p = 0.0298); however, the observed difference was less than one point (0.66 ± 0.30 points, Cohen's d = 0.042). Performance on the 15-item Boston Naming Test also increased significantly (p = 0.0195) between visits 1 and 5. Again, however, the observed difference was a fraction of a point (0.15 ± 0.06 points, Cohen's d = 0.15) as the vast majority of participants performed at or near the ceiling at all visits. Scores also increased significantly on Praxis Drawing between visits 1 and 2 (p = 0.0025), again by less than one point (0.21 ± 0.07 points, Cohen's d = 0.19). Praxis Drawing scores did not increase significantly at any other time point with all participants performing at or near the ceiling on this subtest as well.
Mean raw scores from individual CERAD components (at least two visits, n = 308 at visits 1 and 2).
Performance on Word List Learning was associated with a significant increase in scores (p < 0.0001) between baseline and the fourth assessment (1.15 ± 0.28 points, Cohen's d = 0.26). Delayed Recall scores changed by less than one point across all five assessments; however, a statistically significant increase was observed from visit 1 to 4 (p < 0.0001, 0.66 ± 0.16, Cohen's d = 0.26). Finally, the Word List Recognition discrimination score also changed less than one point from baseline across the subsequent four assessments, yet significant differences were also observed from baseline to visit 4 (p = 0.0117, 0.27 ± 0.11 points, Cohen's d = 0.16).
Between visits 4 and 5, a slight decrease was observed on some subtests. Notably, three of the six subtests (i.e. Boston Naming Test, Praxis Drawing, and Word List Recognition) are subject to ceiling effects, and performances on these tests were close to their respective ceilings throughout the entire assessment period. Of the remaining subtests, a slight and nonsignificant decrease was observed in Animal Naming (−0.40 ± 0.26, p = 0.120) and Word List Learning (−0.40 ± 0.24, p = 0.0993). The only subtest showing a significant decline between visits 4 and 5 was Word List Delayed Recall (−0.52 ± 0.13, p < 0.0001, Cohen's d = 0.25).
To further assess the influence of Word List Delayed Recall on the overall CERAD T-score, we examined differences in performance based on how participants’ baseline scores compared to previously reported normative data (Welsh et al., Reference Welsh1994). Impairment was defined as 1.5 SD below the mean. The analyses show that those persons who were impaired at baseline (n = 33) demonstrated a practice effect on the CERAD T-score that dropped significantly between visits 4 (51.01 ± 1.69) and 5 (43.38 ± 2.11, p < 0.0001, Cohen's d = 0.78). In contrast, participants who had normal scores at baseline (n = 275) showed a plateau at visit 4 with no significant change between visits 4 (56.08 ± 0.61) and 5 (55.47 ± 0.68, p = 0.30) (see Figure 3).
Mean CERAD T-scores by CERAD word list delay performance.
Discussion
Multiple factors must be accounted for when interpreting the findings obtained from longitudinal research studies. For example, repeatedly administered cognitive assessments may be subject to practice effects and multi-year longitudinal studies naturally suffer from participant attrition. Practice effects may obscure subtle but important cognitive decline and negatively effect case ascertainment. Furthermore, attrition is often not random and may be related to a variety of participant-specific factors that also have some effect on cognitive performance such as health status or education (Chang et al., Reference Chang, Yang, Tang and Ganguli2009).
Given the prevalent use of the CERAD battery in longitudinal research, this study examined practice effects on the individual subtests and the T-score derived from the CERAD tests (Morris et al., Reference Morris1989). In addition, analyses were repeated between completers and non-completers to determine whether attrition bias may erroneously contribute to longitudinally increasing group mean scores. We observed significant increase in the CERAD T-score performance across five yearly testing sessions with scores increasing by half of an SD by the fourth assessment. It does not appear to be the case that attrition bias resulted in rising scores over time due to drop-out of poor performers. Age, estimated full-scale IQ, and baseline CERAD T-score were not significantly different between completers and non-completers.
Our findings based upon the sample, including both completers and non-completers, were remarkably similar to that found in the Rosetti et al. (Reference Rosetti, Cullum, Hynan and Lacritz2010) study. In their sample of 383 normal control participants, they observed a gain of 2.8 points in the CERAD raw (unadjusted) total score over a period that also included baseline assessment through four years of follow-up. This is reflected in the current study, which saw a raw score increase of slightly less than three points (see Figure 4) over three years in this sample. Similar to Burkhart et al. (Reference Burkhart2011), we also found evidence for ceiling effects on many of the subtests such as Praxis Drawing and Boston Naming. Given that the follow-up period in their study did not extend beyond three months and that they used a German algorithm for providing age, gender, and education adjustments, further comparisons beyond raw subtest scores are not possible.
Mean CERAD raw scores for completers (n = 182).
A nonsignificant decline in CERAD T-scores was observed between visits 4 and 5. However, upon closer inspection, Word List Delayed Recall was the only subtest that significantly influenced the change in trajectory. When the trajectory change was examined in terms of whether participants were considered impaired or unimpaired on the Word List Delayed Recall subtest at baseline, a much more striking contrast is evident. Participants who were not impaired showed a practice effect that plateaued at visit 4, whereas participants who were impaired improved dramatically by visit 4 only to have a large drop in their scores at visit 5. The question of why those who performed poorly on the Word List Delayed Recall subtest at baseline showed such a different pattern of performance on the CERAD T-score cannot be answered with the current data. One may speculate that a cognitive profile, including memory impairments in the context of otherwise normal cognitive functioning, may be indicative of the early stages of a neurodegenerative disease process. Should that be the case, it is possible that the large drop in CERAD T-scores is due to disease pathology reaching a critical threshold that overwhelms the practice effect. Further research with clinically defined or autopsy-confirmed cases of neurodegenerative disease will be needed to shed more light on this possibility.
The current findings have several implications for clinical prevention trials. Longitudinal studies using the CERAD battery should be aware that moderate practice effects were observed in cognitively normal individuals who were screened into the PREADViSE dementia prevention trial. Thus, all studies using the CERAD battery longitudinally may wish to consider whether practice effects were influential in their own findings. Given that cut-scores are frequently the standard for determining whether an individual meets criteria as cognitively impaired, it is important to consider the effect of even small increases in performance over time. Although not evident in our cognitively normal sample, it is easy to envision a scenario in which practice-driven increase leads to maintenance of scores just above conventional cut points even though the participant may be experiencing early-stage cognitive decline that is below the level of clinical detection. In clinical prevention trials that depend heavily upon these screening measures for case ascertainment, this could have deleterious consequences in terms of obtaining a suitable number of participants who meet criteria for impairment.
A separate but related issue concerns the consequences of relying on cut scores derived from a summary score of performance in multiple cognitive domains. When dividing our sample into two groups on the basis of whether they were impaired or unimpaired on the Word List Delayed Recall subtest relative to a normative sample, some differences in performance trajectory were observed. Both groups continued to show practice effects; however, the impaired group showed a steep drop in the CERAD T-score and loss of the practice effect after five years. Given the small number of participants in the impaired group, it is difficult to say whether this finding is truly related to differences in memory performance or whether it is driven primarily by the variability of a small sample. It may be that poor performance on a measure of delayed recall relative to one's normative group is indicative of future global cognitive decline and, potentially, transition to a diagnosis of neurodegenerative disease. Previous work has shown that verbal delayed recall performance is a significant predictor of transition to dementia over five and ten years of follow-up in a non-demented sample (Tierney et al., Reference Tierney, Yao, Kiss and McDowell2005) and in a large sample (n = 3,055) of non-demented primary care patients over 75 years of age followed over three assessment periods occurring at 18-month intervals (Jessen et al., Reference Jessen2011). Thus, it is certainly possible that the memory-impaired group in our sample is actually manifesting early symptoms of AD.
The magnitude of the practice effect was most apparent in the CERAD demographically adjusted T-score, with lesser increases observed in the subtest raw scores and the unadjusted raw total score. CERAD T-scores increased approximately one-half of an SD across four years of assessment whereas CERAD raw total scores increased 0.40 SD over a four-year period. Mean CERAD total scores, both raw and adjusted, increased significantly between visits 1 and 5, which suggested that participants benefitted from repeated exposure to the testing material.
Strengths of this study include a large, geographically diverse sample, including a number of individuals (n = 182) who participated in the study for five consecutive years. This allowed for a careful analysis of whether attrition bias played a role in the findings of a practice effect in our study. Attrition bias did not account for these findings, since there were no significant differences between completers and non-completers. In addition, following individuals for five years provided adequate data to fully discern a moderate practice effect on the CERAD T-score. Even small practice effects may have large consequences for case ascertainment in longitudinal clinical trials. The finding of a moderate practice effect on CERAD T-scores with increase of, on average, five points (i.e. half of an SD) over a follow-up period of four years suggests that corrections may be needed each year, especially when relying on cut-scores to screen for impairment. With improvements of such magnitude, participants may have to experience substantial decline before it becomes evident on longitudinal screening, and this could have serious consequences for intervention studies targeting mild cognitive impairment as an entity for intervention.
This study also has some weaknesses to be addressed in future research. First, our findings are limited to men, given that the study sample is drawn from participants in a prostate cancer prevention trial. In addition, the screening process is by necessity rather minimal in clinical trials; thus, it cannot be concluded with certainty that some participants recruited as normal were actually experiencing mild impairments and were perhaps in the preclinical stage of dementia.
In summary, the current study demonstrated moderate practice effects on the CERAD T-score in a cohort of older men followed longitudinally for four years. In addition, the practice effect was not due to attrition resulting from drop out of poor performers given that no differences were observed in those variables typically affecting performance over time (i.e. age, estimated full-scale IQ, and the baseline CERAD T-score performance). One significant result that should be expanded upon in future research is the finding that participants who are unimpaired on a global measure of cognition may actually be experiencing clinically significant decrements in specific domains of cognitive functioning. For example, a subset of our cognitively “normal” participants performed in the impaired range on the CERAD Word List Delay subtest at baseline. Those participants also demonstrated a different performance trajectory than those who were not impaired at baseline on the Word List Delay subtest. Specifically, impaired participants showed significant practice effects on the CERAD T-score across three follow-up visits followed by a steep decline between visits 4 and 5. Although we are unable as yet to explore this finding in terms of whether these participants eventually transitioned to a diagnosis of dementia, it is possible that the CERAD Word List Delay subtest is a stronger marker of future cognitive decline than the global CERAD T-score. Future longitudinal research following participants until the point of clinical diagnosis may be useful in determining the prospective utility of the individual CERAD subtests.
Conflict of interest
Dr. Melissa Mathews: NIA Grant (Post-doctoral researcher).
Erin Abner: Alltech (SELPLEX clinical trial) Statistician; NIA–ADC Grant (Data Manager, Bio-Data Management Core); NIA–PREADViSE Grant (Data Manager, Statistician); NIA–R01 Grant (Data Manager).
Dr. Allison Caban-Holt: NIA–PREADiVSE Grant (Co-Investigator); NIA-ADCS Home-Based Assessment (Site Investigator); NICHD–R01 Frontal Aging and Down Syndrome (Co-Investigator).
Dr. Richard Kryscio: Alltech Data Safety Monitoring Board (SELPLEX clinical trial); NIA (2 R01 Grants: co‐PI); NIA‐ADC Grant (Leader, Bio‐Data Management Core); NCRR‐CTSA (Leader, BERD Key Function); NCRR–COBRE (Leader, Biostat/Bioinformatics Core); NICHD, NIA (R21, R03, PPG), NIDA, NINDS: Co‐Investigator.
Dr. Frederick Schmitt: NIH: R01 AG038651; R01HD064993; P30 AG028383; R01AG019241.
Description of authors’ roles
This study is not industry sponsored. NIA Grant Support: R01AG019241 and R01AG038651.
Dr. Melissa Mathews developed research question, did data interpretation, and manuscript preparation. Erin Abner developed research question, did data analysis and interpretation and manuscript revisions. Dr. Allison Caban-Holt did data acquisition, data interpretation, and manuscript revisions. Dr. Richard Kryscio developed research question, did data analysis and interpretation, manuscript revisions, and study supervision. Dr. Frederick Schmitt developed research question, and did data interpretation, manuscript revisions, and study supervision.
