Systematic Bias in Traumatic Brain
Injury Outcome Studies Due to Loss to Follow-up
John D. Corrigan, Ph.D., 1 Cynthia Harrison-Felix, M.S., 2
Jennifer Bogner, Ph.D., 1 Marcel Dijkers, Ph.D., 3
Melissa Sendroy Terrill, B.A., 2 and Gale Whiteneck, Ph.D., 2
This project was a joint effort of the Department of Physical
Medicine and Rehabilitation at Ohio State University, the Research Department
at Craig Hospital in Englewood Colorado, and the Department of Rehabilitation
Medicine, Mount Sinai School of Medicine in New City. The project was
supported in part by: 1) the U.S. Department of Education, Office of Special
Education and Rehabilitative Services, National Institute on Disability
and Rehabilitation Research, Ohio State University Grant #H133A70032,
Rehabilitation Institute of Michigan, Detroit, MI Grant #H133A970021,
and Craig Hospital, Englewood, CO Grant # H133A980020; and 2) the Centers
for Disease Control and Prevention, National Center for Injury Prevention
and Control, Division of Acute Care, Rehabilitation Research and Disability
Prevention Cooperative Agreement U17/CCU812447-06 with the Colorado Department
of Public Health and Environment.
One-third to one-half of subjects in studies of long-term
outcomes of traumatic brain injury (TBI) are not included in outcome samples
because of the inability to track them after they leave inpatient treatment.[1]
The National Traumatic Coma Data Bank reported 45% of 300 survivors were
lost at one-year follow-up.[2] In a cross-sectional study conducted as
part of the Rehabilitation Research and Training Center on Functional
Assessment and Evaluation of Research Outcomes, 46% of subjects discharged
from an inpatient TBI rehabilitation unit could not be reached by phone
or mail six months to five years later.[3] Ponsford and colleagues reported
51% of consecutive admissions to the brain injury program at Bethesda
Hospital in Melbourne, Australia were lost to follow-up at two years.[4]
In a longitudinal study at the University of Alabama at Birmingham's Injury
Control Research Center, 37% of the TBI cohort did not participate in
24-month follow-up interviews.[5]
The loss of subjects to follow-up can be a significant threat
to the internal and external validity of outcome studies, when those lost
differ from those found, in the outcomes themselves or in the factors
affecting outcomes. If a subset of the total population to be studied
is systematically removed from a sample, a biased view of outcome may
result. Greenland observed that in cohort studies when loss from the sample
is random and not associated with the relationships under study, bias
is limited by either large sample sizes or overall small magnitude of
sample loss.[6] However, if a variable predisposing to attrition is related
to both study participation and the outcome of interest (e.g., severity
of injury), selection bias will occur.
The means of addressing the potential for selection bias
have fallen into two general categories--methods for preventing loss and
statistical methods for post hoc compensation. Preventing the loss in
the first place is the most desirable approach, and is the only way to
assure that selection bias will not occur. Generally, recommendations
for preventing loss have included taking proactive steps to collect relevant
contact information upon enrollment or during interim follow-ups (e.g.
addresses and phone numbers of subjects and people who are likely to know
their whereabouts), maintaining periodic contact, establishing rapport
between the research staff and subjects, educating subjects on the importance
of the study, and providing financial or other incentives for participation.[7,
8] However, even a relatively high retention rate does not preclude the
possibility of selection bias. Greenland concluded that selection bias
is a possibility whenever a factor influencing study participation also
influences the outcome of interest.[6] The only way to address the actual
likelihood of a significant bias effect is to examine the parameters of
the specific study. Or as Greenland advised "...answering this question
involves using subject matter knowledge [italics added] to derive an epidemiological
judgment of the ‘reasonableness’ of the existence of such a
bias."[page 187][6]
Unfortunately, there is little "subject matter knowledge"
to inform longitudinal studies of TBI. Corrigan and colleagues sought
to identify potential threats to generalization of outcome data by examining
systematic biases created by subject loss at one year follow-up in the
Ohio State University (OSU) Suboptimal Outcomes dataset. [1] This cohort
consists of 400 consecutive admissions to an inpatient acute rehabilitation
unit. From the first 100 patients with traumatic or anoxic brain injury,
88 subjects with a primary diagnosis of TBI were studied one year following
discharge. Subjects were considered lost to follow-up when phone calls,
mail, clinic visits and assistance from family failed to establish contact.
Thirty-four (38.6%) subjects were lost to follow-up, with those intoxicated
at time of injury and those with a prior history of substance abuse more
likely to be lost. Potential effects of the bias in the follow-up sample
were examined for seven sub-optimal outcomes among subjects followed.
The likelihood of working or being in school one year following discharge
was significantly less for those intoxicated at time of injury and those
with a prior history of substance abuse. The results of this study suggested
that systematic bias in longitudinal studies may result from subjects
with substance use problems being lost to follow-up at one year.
The current study was designed to expand on these earlier
examinations of potential biases by using a single, systematic approach
to data analysis for both one- and two-year follow-up in the three databases.
A consistent method across the three datasets was used to characterize
subjects as lost and found. For each database, demographic, injury-related,
and initial treatment data were compared between these two groups, for
each follow-up year. The goal of this study was to identify characteristics
that differ significantly between individuals who are lost and those who
are found. This "subject matter knowledge" of TBI longitudinal
studies will alert researchers conducting TBI research to the potential
for selection bias among factors that mediate outcomes of interest.
Objective: Identify potential
sources of selection bias created by subjects lost to follow-up in studies
of traumatic brain injury (TBI).
Design: Demographic, premorbid,
injury-related, and hospital course characteristics were compared between
subjects lost and found for one- and two-year post-injury follow-ups,
using bivariate tests and logistic regression analysis. Three prospective,
longitudinal datasets were studied -- one single center, one multi-center,
and the third using a statewide incidence surveillance system and follow-up
registry. Subjects were adolescents and adults hospitalized with a diagnosis
of TBI. Subjects were considered lost when no information was collected
from the person with TBI, or only limited information could be obtained
from a proxy, for any reason, including death, refusal, unable to be located,
and unable to be interviewed.
Results: At year 1 follow-up
58.0% to 58.6% of subjects were found; 39.7% to 42.0% of subjects were
found for both years 1 and 2. Regression variables most frequently associated
with loss to follow-up were cause of injury, blood alcohol level, motor
function, hospital payer source, and race/ethnicity. These data strongly
suggest that in follow-up studies of patients with TBIs severe enough
to warrant hospital admission, three categories of research subjects tend
to be missing from samples available for evaluation one to two years post-injury:
(1) persons from socio-economically disadvantaged groups (racial and ethnic
minorities, those with limited education, those unemployed, those dependent
on public funding for payment of hospital bills), (2) persons who have
a pre-injury history of alcohol or other drug abuse, and (3) persons injured
due to self- or other-directed violence. An exception to these tendencies
for poorer follow-up rates occurs for those subjects who are injured severely
enough that even after inpatient acute rehabilitation they have more severe
motor deficits as measured by FIM Motor scores. These individuals are
more available for follow-up, quite likely because they are institutionalized
after rehabilitation, or depend on parents, other family, or professionals
for supervision and support, factors that increase the ability to locate
them.
Conclusions: TBI follow-up
studies may experience selective attrition of subjects who (1) are socio-economically
disadvantaged, (2) have a history of substance abuse, and (3) have violent
injury etiologies. These phenomena are mitigated for those with more severe
motor deficits. Loss to follow-up may be inherent to this population;
however, the high rate and its selective nature are problematic for outcome
studies. These analyses show minimal differences between the three databases
in loss to follow-up. Differences from one database to another might be
expected given that selection criteria, consent procedures, data collection
methods, and resources available for following or re-finding subjects
differed. Thus, poor follow-up rates may be an inherent characteristic
of studies of persons with TBI. For some investigators in this field who
have been frustrated by low response rates in follow-up studies, there
may be consolation in this conclusion. However, the fact that so many
cases are lost, and that the loss is selective, affects seriously the
conclusions that can be drawn from this research. The level of attrition
should be of concern to all researchers with a stake in long-term outcome
research, and an incentive to seek out methods to improve our ability
to draw valid conclusions and generalize from our samples to the population
of persons with TBI.
References:
1. Corrigan JD, Bogner JA, Mysiw WJ, Clinchot D, Fugate
L. Systematic bias in outcome studies of persons with traumatic brain
injury. Arch Phys Med Rehabil 1997;78(2):132-7.
2. Levin HS, Gary HE, Eisenberg HM, Ruff RM, Barth JT, Kreutzer J, et
al. Neurobehavioral outcome 1 year after severe head injury. J Neurosurg
1990;73:699-709.
3. Corrigan JD, Smith-Knapp K, Granger CV. Validity of the functional
independence measure for persons with traumatic brain injury. Arch Phys
Med Rehabil 1997;78(8):828-34.
4. Ponsford JL, Olver JH, Curran C, Ng K. Prediction of employment status
2 years after traumatic brain injury. Brain Inj 1995;9(1):11-20.
5. Webb CR, Wrigley M, Yoels W, Fine PR. Explaining quality of life for
persons with traumatic brain injuries 2 years after injury. Arch Phys
Med Rehabil 1995;76(12):1113-9.
6. Greenland S. Response and follow-up bias in cohort studies. Am J Epidemiol
1977;106(3):184-7.
7. Coen AS, Patrick DC. Minimizing attrition in longitudinal studies of
special populations: an integrated management approach. Eval Program Plann
1996;19(4):309-19.
8. Carroll K. Enhancing retention in clinical trials of psychological
treatments: Practical strategies. In: Onken LS, Blaine JD, Boren JJ, editors.
Beyond the therapeutic alliance: keeping the drug-dependent individual
in treatment. Washington DC: NIDA Research Monograph; 1997. p. 4-24.
