Comparison of Classifiers Applied to Confocal Scanning Laser Ophthalmoscopy Data

 

 Buy Article Permissions and Reprints

Summary

Objectives: Comparison of classification methods using data of one clinical study. The tuning of hyperparameters is assessed as part of the methods by nested-loop cross-validation.

Methods: We assess the ability of 18 statistical and machine learning classifiers to detect glaucoma. The training data set is one case-control study consisting of confocal scanning laser ophthalmoscopy measurement values from 98 glaucoma patients and 98 healthy controls. We compare bootstrap estimates of the classification error by the Wilcoxon signed rank test and box-plots of a bootstrap distribution of the estimate.

Results: The comparison of out-of-bag bootstrap estimators of classification errors is assessed by Spearman’s rank correlation, Wilcoxon signed rank tests and box-plots of a bootstrap distribution of the estimate. The classification methods random forests 15.4%, support vector machines 15.9%, bundling 16.3% to 17.8%, and penalized discriminant analysis 16.8% show the best results.

Conclusions: Using nested-loop cross-validation we account for the tuning of hyperparameters and demonstrate the assessment of different classifiers. We recommend a block design of the bootstrap simulation to allow a statistical assessment of the bootstrap estimates of the misclassification error. The results depend on the data of the clinical study and the given size of the bootstrap sample.

• References

• 1 Kononenko I. Machine learning for medical diagnosis: history, state of the art and perspective. Artificial Intelligence in Medicine 2001; 23: 89-109.
  CrossrefPubMedGoogle Scholar
• 2 Mardin CY, Peters A, Horn FK, Jünemann AG, Lausen B. Improving glaucoma diagnosis by the combination of perimetry and HRT measurements. Journal of Glaucoma 2006; 15 (04) 299-305.
  CrossrefPubMedGoogle Scholar
• 3 Quigley HA. Number of people with glaucoma worldwide. British Journal of Ophthalmology 1996; 80: 389-393.
  CrossrefPubMedGoogle Scholar
• 4 Heidelberg-Engineering. Heidelberg Retina Tomograph: Bedienungsanleitung Software Version 2.01. Heidelberg Engineering GmbH; Heidelberg: 1997
  Google Scholar
• 5 Mardin CY, Hothorn T, Peters A, Jünemann AG, Nguyen NX, Lausen B. New glaucoma classification method based on standard HRT parameters by bagging classification trees. Journal of Glaucoma 2003; 12: 340-346.
  CrossrefPubMedGoogle Scholar
• 6 Chan K, Lee T, Sample PA, Goldbaum MH, Weinreb RN. Comparison of machine learning and traditional classifiers in glaucoma diagnosis. IEEE Transactions on Biomedical Engineering 2002; 49 (09) 963-974.
  CrossrefPubMedGoogle Scholar
• 7 Adler W, Hothorn T, Lausen B. Simulation based analysis of classification of medical images. Methods Inf Med 2004; 43 (02) 150-155.
  Article in Thieme ConnectPubMedGoogle Scholar
• 8 Bowd C, Chan K, Zangwill LM, Goldbaum MH, Lee T, Sejnowski TJ. et al. Comparing neural networks and linear discriminant functions for glaucoma detection using confocal scanning laser ophthalmoscopy of the optic disc. Investigative Ophthalmology and Visual Science 2002; 43 (11) 3444-3454.
  PubMedGoogle Scholar
• 9 Peters A, Lausen B, Michelson G, Gefeller O. Diagnosis of glaucoma by indirect classifiers. Methods Inf Med 2003; 42 (01) 99-103.
  Article in Thieme ConnectPubMedGoogle Scholar
• 10 Schwarzer G, Nagata T, D.M Schmelzeisen D, Schumacher M. Comparison of fuzzy inference, logistic regression, and classification trees (CART). Methods Inf Med 2003; 42 (05) 572-527.
  Article in Thieme ConnectPubMedGoogle Scholar
• 11 Duin RPW. A note on comparing classifiers. Pattern Recognition Letters 1996; 17: 529-536.
  CrossrefPubMedGoogle Scholar
• 12 Van-Gestel T, Suykens JA, Baesens B, Viaene S, Vanthienen J, Dedene G. et al. Benchmarking least squares support vector machine classifiers. Machine Learning 2004; 54 (01) 5-32.
  CrossrefPubMedGoogle Scholar
• 13 Markowetz F, Spang R. Molecular diagnosis – classification, model selection and performance evaluation. Methods Inf Med 2005; 44: 438-443.
  Article in Thieme ConnectPubMedGoogle Scholar
• 14 R Development Core Team. R: A Language and Environment for Statistical Computing: R Foundation for Statistical Computing; 2005
  PubMed
• 15 Goldbaum MH, Sample PA, Chan K, Williams J, Lee T-W, Blumenthal E. et al. Comparing machine learning classifiers for diagnosing glaucoma from standard automated perimetry. Investigative Ophthalmology and Visual Science 2002; 43 (01) 162-169.
  PubMedGoogle Scholar
• 16 Sample PA, Goldbaum MH, Chan K, Boden C, Lee T, Vasile C. et al. Using machine learning classifiers to identify glaucomatous change earlier in standard visual fields. Investigative Ophthalmology and Visual Science 2002; 43 (08) 2660-2665.
  PubMedGoogle Scholar
• 17 Jonas J. Biomorphometrie des Nervus Opticus. Stuttgart: Enke; 1989
  Google Scholar
• 18 Ripley BD. Pattern Recognition and Neural Networks. Cambridge, UK: Cambridge University Press; 1996
  Google Scholar
• 19 Venables WN, Ripley BD. Modern Applied Statistics with S. 4th ed. New York: Springer; 2002
  Google Scholar
• 20 Läuter J. Stabile multivariate Verfahren: Diskriminanzanalyse – Regressionsanalyse – Faktoranalyse. Berlin: Akademie Verlag; 1992
  Google Scholar
• 21 Läuter J, Glimm E, Kropf S. Multivariate tests based on left-spherically distributed linear scores. TheAnnals of Statistics 1998; 26 (05) 1972-1988.
  PubMedGoogle Scholar
• 22 Peters A, Hothorn T, Lausen B. ipred: Improved predictors. R News 2002; 2 (02) 33-36.
  PubMedGoogle Scholar
• 23 Hastie T, Tibshirani R, Buja A. Flexible discriminant analysis by optimal scoring. Journal of the American Statistical Association 1994; 1: 1255-1270.
  PubMedGoogle Scholar
• 24 Friedman JH. Multivariate adaptive regression splines (with discussion). The Annals of Statistics 1991; 19: 1-141.
  CrossrefPubMedGoogle Scholar
• 25 Hastie T, Buja A, Tibshirani R. Penalized discriminant analysis. The Annals of Statistics 1995; 23: 73-102.
  CrossrefPubMedGoogle Scholar
• 26 Hastie T, Tibshirani R. Discriminant analysis by gaussian mixtures. Journal of the American Statistical Association 1996; 1: 1255-1270.
  PubMedGoogle Scholar
• 27 Webb A. Statistical Pattern Recognition. 2nd ed. Chichester, England: Wiley; 2002
  Google Scholar
• 28 Heckerling PS, Gerber BS, Tape TG, Wigton RS. Entering the black box of neural networks. Methods Inf Med 2003; 42 (03) 287-296.
  Article in Thieme ConnectPubMedGoogle Scholar
• 29 Seuchter SA, Eisenacher M, Riesbeck M, Gaebel W, Köpcke W. other members of the A.N.I. Study Group Methods for predictor analysis of repeated measurements: application to psychiatric data. Methods Inf Med 2004; 43 (02) 184-191.
  Article in Thieme ConnectPubMedGoogle Scholar
• 30 Rumelhart DE, Hinton GE, Williams RJ. Learning internal representations by error propagation. In: Rumelhart DE, McClelland JL. editors. Parallel distributed processing (Vol 1). Cambridge, MA: MIT Press; 1986. pp 318-362.
  Google Scholar
• 31 Vapnik V. Statistical Learning Theory. New York: Wiley; 1998
  Google Scholar
• 32 Vapnik V. The Nature of Statistical Learning Theory. 2nd ed. New York: Springer; 2000
  Google Scholar
• 33 Meyer D. Support vector machines. R News 2001; 1 (03) 23-26.
  PubMedGoogle Scholar
• 34 Kohonen T, Hynninen J, Kangas J, Laaksonen J, Torkkola K. LVQ_PAK: The learning vector quantization program package: Laboratory of Computer and Information Science. Helsinki University of Technology; 1996
  Google Scholar
• 35 Breiman L. Bagging predictors. Machine Learning 1996; 24 (02) 123-140.
  PubMedGoogle Scholar
• 36 Breiman L. Random forests. Machine Learning 2001; 45 (01) 5-32.
  CrossrefPubMedGoogle Scholar
• 37 Liaw A, Wiener M. Classification and regression by randomForest. R News 2002; 2 (03) 18-22.
  PubMedGoogle Scholar
• 38 Hothorn T, Lausen B. Bundling classifiers by bagging trees. Computational Statistics & Data Analysis 2005; 49: 1068-1078.
  CrossrefPubMedGoogle Scholar
• 39 Dimitriadou E, Hornik K, Leisch F, Meyer D, Weingessel A. e1071: Misc functions of the department of statistics (e1071). TU Wien. 2004
  PubMedGoogle Scholar
• 40 Efron B, Tibshirani R. Improvements on crossvalidation: the .632+ bootstrap method. Journal of the American Statistical Association 1997; 92: 548-560.
  PubMedGoogle Scholar
• 41 Breiman L. Out-of-bag estimation. Technical Report. Berkeley (CA 94708): Statistics Department, University of California; 1996
  Google Scholar
• 42 Envgeniou T, Pontil M, Eliseeff A. Leave one out error, stability, and generalization of voting combinations of classifiers. Machine Learning 2004; 55 (01) 71-97.
  CrossrefPubMedGoogle Scholar
• 43 Mikelberg F, Parfitt C, Swindale N, Graham S, Drance S, Gosine R. Ability of the Heidelberg Retina Tomograph to detect early glaucomatous visual field loss. Journal of Glaucoma 1995; 4: 242-247.
  PubMedGoogle Scholar
• 44 Mardin CY, Horn FK, Jonas J, Budde WM. Preperimetric glaucoma diagnosis by confocal scanning laser tomography of the optic disc. British Journal of Ophthalmology 1999; 83 (03) 299-304.
  CrossrefPubMedGoogle Scholar
• 45 Bathija R, Zangwill LM, Berry CC, Sample PA, Weinreb RN. Detection of early glaucomatous structural damage with confocal scanning laser tomography. Journal of Glaucoma 1998; 127: 7 (2): 121.
  PubMedGoogle Scholar
• 46 Uchida H, Brigatti L, Capriolo J. Detection of structural damage from glaucoma with confocal laser image analysis. Investigative Ophthalmology and Visual Science 1996; 37 (12) 2393-2401.
  PubMedGoogle Scholar