Improving screening accuracy for dementia in a community sample by augmenting cognitive testing with informant report

https://doi.org/10.1016/S0895-4356(03)00009-XGet rights and content

Abstract

This study sought to determine whether the augmentation of cognitive testing with an informant report questionnaire could improve accuracy in screening for dementia in a community setting. The sample consisted of 646 subjects aged 70–93 years. Cognitive state was assessed using the Mini-Mental State Examination (MMSE). Informants completed the 16-item short form of the Informant Questionnaire for Cognitive Decline in the Elderly (IQCODE). Dementia was diagnosed according to DSM-III-R criteria. Combination of the IQCODE with the MMSE resulted in more accurate prediction of caseness than either test alone. Receiver Operating Characteristic analysis demonstrated the superior screening performance of a logical “or” rule and a weighted sum of scores on the two tests over other methods of combination, replicating previous clinically based research. The findings also suggest that the appropriate combination of existing tests may be a fruitful method of improving screening accuracy in a variety of situations.

Introduction

The diagnosis of dementia requires expert clinical judgment to establish the extent and degree of the cognitive impairment that is central to the disorder as well its progression and the absence of alternative diagnostic causes. This is time consuming and expensive.

The use of relatively brief psychometric tests of cognitive performance that require only minimal training for administration is the most common method of primary assessment or screening for dementia in both clinical and epidemiologic case-finding applications. The use of formal methods of informant report is an alternative method of case detection that has received attention in clinical and epidemiologic studies [1], [2]. This involves the administration of a questionnaire to a person who knows the patient or subject well. The questionnaire inquires about aspects of memory and intellectual function in everyday situations compared to earlier in life. In a meta-analysis, Jorm [3] demonstrated no significant difference between the performance of cognitive tests and informant report in screening for dementia.

Both these methods can achieve sensitivities in excess of 0.90 by setting liberal cut points or thresholds for classification of caseness. However, under these circumstances specificity is at best modest, with values below 0.60 frequently being obtained. Even in populations with a high prevalence of dementia, these performance characteristics will leave a nontrivial proportion of true cases of dementia undetected and result in the predictive value of a positive test being so low that the majority of screen-positive individuals are, in fact, not demented [4]. In clinical work this has significant social and medical implications arising from having to deal with large numbers of false positives and false negatives. In epidemiologic settings, depending on the role of screening in the research being undertaken, implications may include imprecise or biased prevalence estimates, or selected samples not having desired characteristics (i.e., presence or absence of the disorder). Straightforward attempts to improve specificity by increasing the stringency of cutoffs are invariably accompanied by a comparable and unacceptable reduction in sensitivity. Not surprisingly, considerable effort has been directed at the development of better methods for the primary assessment or screening for dementia.

Research to improve the precision with which dementia is diagnosed or screened includes the search for biologic markers [5] or physiologic signs [6], the development of new cognitive tests [7] or the construction of refined test batteries [8], and the adjustment of scores from existing tests for confounding factors such as education or sensory disability [9]. Although some of these approaches are promising, their performance has been variable, with the result that the need for better methods of detecting dementia in clinical and epidemiologic applications remains.

Recently, Mackinnon and Mulligan [10] took a different approach to this problem. They sought to improve screening performance by combining the most widely used cognitive screening test—the Mini Mental State Examination (MMSE)—with an informant questionnaire (the Informant Questionnaire for Cognitive Decline in the Elderly, IQCODE). Their approach was specifically aimed at clinical applications: the MMSE was investigated because of its ubiquitous application in clinical settings; the IQCODE is brief, and has been demonstrated to be easy to use and acceptable in these settings [1]. The methods of combining the test results, which included a graphical template, were easy to implement. The basis for the combined classification was transparent to the user. Classifying patients with MMSE scores lower than 24 or IQCODE scores greater 4.0 as cases produced sensitivity of 0.93 and specificity of 0.81. This compared to sensitivity of 0.76 and specificity of 0.90 for the MMSE alone at the same cutoff. It was possible to combine the two tests by weighted sum so that both sensitivity and specificity exceeded 0.90.

Subsequently, Khacaturian and colleagues [11] demonstrated improved screening accuracy for dementia over the use of a single test in a community sample using a two-stage procedure. In the first stage, subjects were provisionally classified according to their scores on a modified version of the MMSE [12]. Subjects for whom the results of this test were not available had scores imputed using the IQCODE. A liberal cutoff for caseness was adopted, resulting in high sensitivity but low specificity. Subjects positive on this test were reviewed using the Dementia Questionnaire (DQ) [13]. Responses to the DQ were reviewed by clinicians and used to derive a second stage screening classification. This procedure yielded a sensitivity of 0.91 and specificity of 0.92.

Although they have different mechanisms and fields of application, the work of both Mackinnon and Mulligan [10] and Khachaturian and colleagues [11] suggests that there is considerable utility in taking existing tests and combining their results to improve performance. Their approaches exploit well-validated, established screening tests, and pose few barriers to immediate adoption. To date, however, research on combining tests has complementary limitations. Khachaturian and colleagues used a consensus rating of DQ responses by at least to clinicians to classify subjects after the initial screening. The DQ itself is a relatively lengthy instrument. This assessment procedure clearly exceeds the resources available in many screening applications, and would not be useful in situations in which a screening decision is required immediately after test administration. In contrast, Mackinnon and Mulligan's [10] approach does not require clinical judgment, is simple to implement, and can yield a screening classification immediately after test scores have been calculated. However, their method was developed and tested in a clinical sample in which the prevalence of dementia was high (55%). Although a consecutive series of patients was recruited, differences in the severity of cognitive impairment and distribution of comorbid conditions make generalizing their findings to other settings hazardous. This applies both to conclusions regarding the rules by which the tests are combined and to the cut points used in applying them. Thus, there is a clear need to replicate their results in a nonclinical sample.

The aim of this article was to determine whether the improvement in dementia screening performance gained using the methods developed by Mackinnon and Mulligan [10] in a clinical sample could be obtained in a community sample in which the prevalence of dementia was within the bounds of that found in unselected populations. Data from a large community-based study of elderly persons (The Canberra Longitudinal Study) were available to address this issue.

Section snippets

Rules for combining test outcomes

The diagnostic performance of psychometric scales and other tests that yield a numerical score is often investigated by means of Receiver Operating Characteristic (ROC) analysis [14]. This approach is effectively restricted to individual tests. The development of optimal combinations of tests has received little attention, and has focused on issues of sequencing and cost [14], [15]. Three methods by which two test results might be combined were discussed by Mackinnon and Mulligan [10]. They

Subjects

The sample was a probability sample of persons aged 70 years and older drawn from the electoral roll for Canberra and the neighboring town of Queanbeyan, and stratified to recruit equal numbers of males and females. The sample and study design has been described previously [18], [19], [20]. Of the 945 participants, interviews including the IQCODE could be obtained from 694 informants. The MMSE scores were not available for 48 participants. Diagnoses of dementia could not be made for 19 persons.

Results

Thirty-six participants met DSM-III-R criteria for dementia. MMSE scores for all participants ranged from 7 to 30 (mean 27.4, SD 2.7) while IQCODE ratings ranged from 3.00 to 5.00 (mean 3.23, SD 0.32). As would be expected, means of both scales differed significantly between demented and nondemented participants (MMSE 22.3 vs. 27.7, IQCOSDE 3.87 vs. 3.19, P<.0001). A scatterplot of MMSE and IQCODE scores is shown in Figure 2. The correlation between the tests was −0.42.

Discussion

These findings replicate in an unselected community setting earlier clinically based results in demonstrating that it is possible to achieve a significant improvement in the performance of a test of cognitive status (the MMSE) by augmenting it with a formal method of informant report (the IQCODE). Just as in the earlier study, comparable levels of sensitivity and specificity were obtained for two methods of combination—a weighted sum of test scores to which a suitable cut point is applied or,

Acknowledgements

The Canberra Longitudinal Study was supported by a Unit grant from the National Health and Medial Research Council and by the Australian Rotary Health Research Fund. Thanks are due to Colleen Doyle, Scott Henderson, Trish Jacomb, Susan Lindsay, and Ruth Scott for their contribution to the design of the survey.

References (31)

  • J.C. Anthony et al.

    Limits of the “Mini-Mental State” as a screening test for dementia and delirium among hospital patients

    Psychol Med

    (1982)
  • A. Mackinnon et al.

    Combining cognitive testing and informant report to increase accuracy in the assessment of dementia

    Am J Psychiatry

    (1998)
  • E.L. Teng et al.

    The Modified Mini-Mental State (3MS) examination

    J Clin Psychiatry

    (1987)
  • J.M. Silverman et al.

    Reliability of the family history method in genetic studies of Alzheimer's disease and related dementias

    Am J Psychiatry

    (1986)
  • H.C. Kraemer

    Evaluating medical tests

    (1982)
  • Cited by (68)

    • Spine Fragility Fracture Prediction Using TBS and BMD in Postmenopausal Women: A Bayesian Approach

      2022, International Journal of Environmental Research and Public Health
    View all citing articles on Scopus
    View full text