Reprint of Criteria for the clinical evaluation of dental restorative materials

Abstract

Rating scales were developed for several factors that were considered relevant to the problem of clinically evaluating dental restorative materials. Examiners were trained to use the rating scales, and their performance was evaluated in field trials. Data analysis of examiner performance was used to revise the written criteria, and to train the examiners in making consistent judgments of dental restorations. Criteria were adopted when field testing indicated that examiners were able to duplicate their own judgments and judgments of other examiners at a predetermined level of acceptability. Further experience with the rating scales in actual clinical studies led to the consolidation of anterior and posterior criteria, which had been developed separately, and to the deletion of certain rating scales which failed to yield useful information. The rating scales which were finally adopted are for color match, cavo-survace marginal discoloration, anatomic form, marginal adaptation, and caries.

Appendices

Appendices

Appendix I describes Criteria Development and provides data from a study to develop rating scales for marginal adaptation. Appendix II provides data from studies to develop rating scales for evaluating anterior restorations¹ and posterior restorations². Two characteristics developed during these studies have subsequently been eliminated. These are dark deep discoloration (anterior) and surface texture (posterior), both of which proved to be highly susceptible to examiner drift. Contour (anterior) and anatomic form (posterior) were nearly identical in wording, as were marginal integrity (anterior) and marginal adaptation (posterior), and were readily consolidated into a single system for examining both anterior and posterior restorations. The “Oskar” category for marginal adaptation was eliminated. Data on dark deep discoloration and surface texture are presented for the sake of historical accuracy, and to illustrate the point that reaching acceptable levels of performance in training sessions does not guarantee acceptable performance later on. There are no statistical data concerning examiner performance in judging caries at the margins of restorations because it is very difficult to locate a study population having a sufficient proportion of caries at the margin to warrant a special study. Examiner agreement on caries at margins has proved to be higher than for other characteristics judged during the course of clinical studies. However, this could be explained by assuming that examines can usually agree that caries is not present, particularly in studies conducted two to three years following placement.

Appendix III presents those statistical methods that are appropriate for data derived from rating scales used in clinical trials. Appendix IV contains references.

Appendix I

Criteria Development

The historical development of the rating scale for one characteristic (marginal adaptation) will be traced in this section to illustrate the methodology employed in developing rating scales for all the characteristics which comprise the criteria for evaluating dental restorative materials. Separate criteria were originally developed for evaluating anterior and posterior restorations, utilizing separate field trials. Data pertaining to these trials are provided in this Appendix.

The following outline summarizes the steps which were taken to select characteristics and develop associated rating scales:

• 1. Literature review and discussion to select relevant characteristics for clinical evaluation.

• 2. Development of written criteria to describe each characteristic selected for evaluation.

• 3. Clinical trial of the criteria using a small number of patients, followed by discussion among the examiners, and modification of the written criteria to remove ambiguities and to more closely specify the operational definitions of judgmental categories.

• 4. Consultation with a statistician to remove logical inconsistencies in the written criteria and to arrange judgmental categories so none were superfluous and none captured an overwhelming majority of responses.

• 5. Training of examiners in using revised criteria.

• 6. Testing of examiners using criteria in a survey.

• 7. Criteria revision, examiner re-training, and further testing as needed.

The closeness of adaptation between restorative material and tooth structure is a characteristic that most investigators agree must be assessed in evaluating restorations.^{6,7,8,9,10,11} Closeness of adaptation has been investigated by clinical assessment and by laboratory methods involving the measurement of dye penetration and the tracing of radioactive isotopes along the interface¹². Although written criteria had previously been developed by the Materials and Technology Branch to assess the marginal integrity of anterior restorations,^1,13 the development of the first written criteria for marginal adaptation was carried out as if no previous model existed.

Phonetic code words were used to reduce misunderstandings when ratings were given orally by examiners. Alphabetic ratings also emphasize that the rating scales are considered to be ordinal, not interval. The first criteria for marginal adaptation were written as follows.

Alfa:

The explorer does not “catch” when drawn across the restoration-tooth margin either from tooth to restoration or from restoration to tooth. If a “catch” exists, there is no visible crevice along the periphery of the restoration. The edge of the restoration appears to adapt closely to the tooth structure along the entire periphery of the restoration.

Bravo:

The explorer does “catch” and there is visible evidence of a crevice into which the explorer will penetrate, indicating that the edge of the restoration does not closely adapt to the tooth structure. The dentin or base is not exposed, and the restoration is not mobile, fractured, or missing in part or in toto.

Charlie:

The explorer penetrates into crevice indicating that a space exists between the restoration and the tooth structure. The dentin or the base is exposed at the periphery, but the restoration is not mobile, fractured, or missing in part or in toto.

Delta:

The restoration is mobile, fractured, or missing in part or in toto.

Oscar:

Marginal adaptation cannot be assessed due to an excess of restorative material at the margin.

Five dentists were trained in the use of the criteria at the Dental Health Center. During the first session the rationale for the criteria, the rating system, the coding system, and the record forms were explained and discussed. Ten restorations were then rated by the instructor to clinically illustrate the rating system. After the instructor explained the reason for assigning each rating the trainees examined the same restorations. Each trainee was encouraged to explain his interpretation of each characteristic and thus his reasons for agreement or disagreement with ratings assigned by the instructor. Where disagreements occurred, the categories were again explained so that all examiners would invoke the same concepts when using the rating scales.

A statistical consultant prepared notes on the training session, which were distributed to each examiner prior to the second session. The notes contained a resume of the discussion during the first training session. The following comments were noted:

• 1. All visible margins are to be examined.

• 2. Code Delta is used when the restoration is grossly fractured, that is, a fracture at the isthmus or when there is a fracture more than 1/2 mm from the margin or where the restoration is missing more than 1/2 mm from the margin. Fractures or loss of material less than 1/2 mm from the margin should be classified as Bravo or Charlie. Overt secondary caries is also included in this category.

• 3. Code Bravo should be used only when there is a visible crevice where the explorer catches.

• 4. Code Charlie should be used when there is evidence of secondary caries at the margin.

The examination procedures employed during the second training session simulated those to be used in the field surveys. The session was conducted in two phases. On the first day the instructor and each of the trainees independently examined 29 posterior restorations. The ratings were recorded and discussion was not allowed. Two days later the examiners rated the same 29 restorations for a second time. Two additional patients were examined to keep the examiners occupied, and to reduce discussion about the criteria between examinations. When the second examination was completed for all examiners, a discussion period was held with the patients present. Any restoration which presented a rating problem for any examiner was reviewed and discussed. Notes were prepared from this session also, and distributed to the examiners at a staff meeting prior to the first field survey. The comments from this session were as follows:

• 1. The objective is to assess the adaptation of the restorative material to the tooth structure at the margin.

• 2. Code Charlie should be used when there is evidence of secondary caries in dentin at the margin, or any exposure of the dentin or base at the margin.

The ratings assigned by the examiners during the training session were tallied on a sequential analysis graph to determine if an acceptable performance level had been attained. Performance was considered acceptable during this phase of training when examiners agreed with their own judgments and with consensus judgments more than 75 percent of the time. (Consensus was defined as a majority opinion.) Acceptable performance levels were attained, so the first field survey was scheduled at a nearby Coast Guard Station. The examiners were recruits, mainly eighteen to twenty years of age. Each restoration was examined twice by the five examiners, the first and the second examination being separated by two days. For both examinations combined, there were a total of 1,280 judgments of marginal adaptation and 128 restorations. This distribution of ratings is given below in Table 1:

Table 1 All ratings: marginal adaptation (survey number one)

Table 2 provides the consensus ratings for the 128 restorations that were examined in Survey Number One, and Table 3 provides the ratings that were given by each examiner. Examiners A, B, C had previously been calibrated in using rating scales developed for evaluating anterior restorations, but examiners A and C had considerably more experience in rating restorations placed for comparative studies of dental materials. Examiners Y and Z were attempting to use rating scales for the first time. The relatively small number of “Charlie” and “Delta” ratings seemed reasonable in view of the age of recruits, and the probability that their restorations were not very old.

Table 2 Consensus ratings: marginal adaptation (number and percent; survey number one)

Table 3 Marginal adaptation: ratings by examiner (survey number one)

Table 4 provides the duplicate ratings given by each examiner from one examination to the next. The most experienced examiners were best able to duplicate their judgments, while the least experienced examiners did not perform as well. Only one examiner achieved the goal of eighty-five percent self-agreement, so further training was considered necessary.

Table 4 Duplicate ratings by examiner, marginal adaptation (survey number one)

Tabulations providing the nature if disagreements (for examiners compared with consensus and examiners compared with self) were obtained for Survey Number One, but owing to the small number of “Charlie” and “Delta” ratings, they were not particularly useful in deciding how to modify the written criteria for marginal adaptation. Generally, most disagreements were between “Alfa” and “Bravo” ratings.

Survey Number Two

Survey Number Two was based on the course of action a practicing dentist would be likely to follow in a clinical situation; that is, to re-examine a restoration in six months, to replace it for preventive reasons, or to replace it immediately because of damage to the tooth structure. The examiners were asked to separately list any aesthetic comments they might have concerning a restoration. The instructions provided to the examiners are duplicated below:

Instructions: Assign one of the following ratings to each restoration, and give your reason for the rating. More than one reason may be given. Do not base the rating on aesthetic qualities. If you have a comment on the aesthetic qualities of the restoration, enter it under “b” in the comment box.

Alfa	Bravo	Charlie	Delta
Replacement unnecessary	Replacement Questionable	Replace for preventive reasons	Replace immediately

Survey Number Two utilized 22 patients from a Sixth United States Army unit stationed at Fort Baker near San Francisco. One hundred twenty-two restorations were examined by the same examiners that had participated in the first survey.

Analysis of results consisted of arranging the reasons given for ratings in a matrix which revealed how often factors were named as the sole reason for ratings and how often they were named in conjunction with other factors. For 582 non-Alfa ratings, factors were mentioned 205 times without other factors being named. Of these 205, “Margin” or “Open Margin” accounted for nearly seventy percent of the total, “Caries” for about ten percent, “Inadequate Extension” for about eight percent and “Fractured Margin” for about five percent. The ratings associated with these factors are presented in Tables 5, 5a, 5b.

Table 5 Factors associated with non-Alfa ratings (survey number two)

Table 5a Factor matrix: posterior criteria study (survey number two)

Table 5b Number of times factor mentioned, posterior criteria study (survey number two)

It can be seen that the most prevalent reason given for non-Alfa ratings was poor margins, and that most of these were assigned a “Bravo” rating, meaning that the examiner would like to see the restoration again in six months. Half of the ratings associated with inadequate extension were “Charlie,” meaning that the restoration should be replaced for preventive reasons. When caries was mentioned as the sole reason for giving a rating, the rating was always “Delta.”

Since most of the participants in Survey Number Two had some experience in using rating scales, survey results were checked by having several dentists not employed at the Dental Health Center and not familiar with the written criteria perform the same task. The results indicated that both trained and untrained examiners give the same reasons for assigning ratings to restorations, in the same order of importance.

The major conclusions regarding the criteria as used in the first survey, based on the results of surveys one and two, were that the written criteria described factors that are relevant clinical indicators of the status of restorations, but that some changes in oral instructions for using the criteria needed to be made in order to have the criteria reflect single rather than composite factors, and to restrict the examiner’s judgment to what could be seen rather than what might be inferred. A new recording form was designed for Survey Number Three, with boxes for examiners to indicate the presence of caries and discoloration. Oral instructions were modified to eliminate the presence of caries or discoloration as a reason for assigning a rating for marginal adaptation. In other words, judgments were confined to the operational definitions presented in the written criteria, which were unchanged from those used in the first survey. The notes which had been prepared after the first and second training sessions were rescinded; they had served the purpose of resolving controversy among the examiners during training, but logical analysis revealed that they contributed little to specific definition of the factors to be rated. Additional experience made it easier for the examiners to accept the criteria as written, without serious disagreements over minor diagnostic points.

It was felt that the opportunity to note caries and discoloration would help the examiners to let their judgments be guided by the written criteria.

Results of survey number three

Prior to conducting the final field survey, a demonstration was conducted by the instructor, using extracted teeth with amalgam restorations, and silver plated models of posterior teeth containing restorations. A full range of conditions were represented for each characteristic. Following the demonstration with the extracted teeth and models, one patient with approximately ten posterior restorations was examined by the instructor and each examiner. The ratings were not recorded but the reasons for assigning ratings were reviewed by the examiners.

Survey Number Three was conducted at the Recruit Training Center, United States Coast Guard Base, Alameda, California. The examinees were recruits who had recently reported for duty. The five dentists who had participated in the previous field tests served as examiners, and recorders were obtained from the survey group. Randomly selected posterior quadrants were examined for each patient, yielding a total of 185 restorations which were rated twice by each examiner during trials held two days apart. Over 97 percent of the restorations were dental amalgams.

The distribution of 1850 ratings for marginal adaptation, obtained in the two examination sessions, is shown in Table 6. Consensus ratings are given in Table 7.

Table 6 All ratings, marginal adaptation (number and percent, survey number three)

Table 7 Consensus ratings, marginal adaptation (number and percent; survey number three)

Marginal Adaptation ratings were distributed in every rating category, but the distribution, shown in Table 8, indicated that not all examiners were rating restorations identically. Examiner Z (one of the least experienced) appears to have been the most critical in assessing margins, especially in the “Alfa-Bravo” zone.

Table 8 Marginal adaptation ratings by examiner (survey number three)

Although the examiners differed among themselves in rating marginal adaptation, they were able to duplicate their own ratings from the first to the second examination fairly well, as shown in Table 9. Because of inter-examiner disagreements, however, further training in using the rating scales for this characteristic was planned.

Table 9 Duplicate ratings by examiner, marginal adaptation (survey number three)

Table 10 indicates that most of the disagreements with consensus were of an “Alfa-Bravo” nature; this was not surprising, since most of the margins in this patient group had been rated as “Alfa” or “Bravo.” As shown in Table 11, intra-examiner disagreements followed the same pattern as the disagreements with consensus.

Table 10 Agreement and disagreement with consensus ratings by examiner, marginal adaptation (survey number three)

Table 11 Inter-examiner agreement and disagreement by examiner, marginal adaptation (survey number three)

The percent inter-examiner agreement and intra-examiner agreement in judging marginal adaptation on Survey Number Three is given in Table 12 Inter-examiner agreement ranged from 57.2 to 85.4 percent, and intra-examiner agreement ranged from 79.4 to 88.6 percent.

Table 12 Percent inter- and intra-examiner agreement, marginal adaptation (survey number three)

Graph 1 summarizes self-agreement for all examiners in the third survey. For example, examiner A agreed with his own judgments 82.7 percent of the time. Since this percentage can be expected to vary randomly from trial to trial, it is useful to determine what the range of variation is likely to be. The graph indicates that at the 95 percent level of confidence self-agreement under Marginal Adaptation for examiner A was between 77.5 percent and 87.5 percent. It was encouraging to note that in no case did the lower limit for any examiner fall below 74 percent.

Graph 1

95 percent confidence intervals for self-agreement by examiner (survey number three)

Appendix II

Survey Results

Table 1 All ratings (survey number three), anterior criteria study, all characteristics

Table 2 Consensus ratings (survey number three), anterior criteria study, all characteristics

Table 3a Color match ratings by examiner, anterior criteria study (survey number three)

Table 3b Cavosurface marginal discoloration ratings by examiner, anterior criteria study (survey number three)

Table 3c Dark deep discoloration ratings by examiner, anterior criteria study (survey number three)

Table 3d Contour ratings by examiner, anterior criteria study (survey number three)

Table 3e Marginal integrity ratings by examiner, anterior criteria study (survey number three)

Table 4a Percent inter- and intra-examiner agreement, anterior criteria study, color match (survey number three)

Table 4b Percent inter- and intra-examiner agreement, anterior criteria study, cavo-surface marginal discoloration (survey number three)

Table 4c Percent inter- and intra-examiner agreement, anterior criteria study, dark deep discoloration (survey number three)

Table 4d Percent inter- and intra-examiner agreement, anterior criteria study, contour (survey number three)

Table 4e Percent inter- and intra-examiner agreement, anterior criteria study, marginal integrity (survey number three)

Graph 1

95 percent confidence intervals for self-agreement by examiner, anterior criteria study (survey number three)

Table 6 All ratings (survey number three), posterior criteria study, all characteristics

Table 7 Consensus ratings (survey number three), posterior criteria study, all characteristics

Table 8b Surface texture ratings by examiner, posterior criteria study (survey number three)

Table 8b Anatomic form ratings by examiner, posterior criteria study (survey number three)

Table 8c Marginal adaptation ratings by examiner, posterior criteria study (survey number three)

Table 9a Percent inter- and intra-examiner agreement, posterior criteria study, surface texture (survey number three)

Table 9b Percent inter- and intra-examiner agreement, posterior criteria study, anatomic form (survey number three)

Table 9c Percent inter- and intrae-xaminer agreement, posterior criteria study, marginal adaptation (survey number three)

Graph 2

95 percent confidence intervals for self-agreement by examiner (survey number three)

Appendix III

Statistical Methods

1. Sequential Analysis

Sequential analysis is useful for testing examiner performance, since each comparison can be dichotomized as either “agree” or “disagree.” The advantage of sequential trials is that there is no fixed number of cases – the experiment can be ended when a decision is reached, thus eliminating unnecessary work. In addition, Type I error, α, and Type II error, β, are both specified in advance, in contrast to the usual arrangement where Type I error is fixed, and Type II error must be calculated. An excellent discussion of sequential analysis can be found in Chilton¹⁴. For the studies reported in this volume, α and β were both set at. 0.10, and the region of no decision was between 0.75 and 0.85. The sequential worksheet used in these studies appears at the end of the appendix.

2. Data Analysis for Clinical Studies

In a completely randomized experiment where test and control groups of teeth constitute independent samples, and the outcome consists of graded results, a modified Wilcoxon 2-sample Test¹⁵ is an excellent way of testing for statistical differences. The advantage of this test is that it makes use of data having an ordinal arrangement, while chi-square does not. However, when test and control teeth appear in the same mouth, with one randomly designated as test while the other is control, a test of significance for related samples must be used. The sign test is appropriate, providing that there are not too many tied pairs. An alternative is to assign numeric values to the letter ratings, and to apply a test to the distribution of differences between pairs. The choice of assigned values makes little difference in the outcome of this statistic.

An experiment using matched pairs has the advantage of controlling for environmental and biological factors, while independent samples depend upon randomization for control. Kish¹⁶ presents a good discussion of the relation of statistical tests to variables that are controlled, uncontrolled, or randomized.

3. Methodological Note

Since the teeth of any given patient are not independent of each other, the patient is the unit of analysis. This presents no problem when each patient in a study has either one test or one control restoration (the case of independent samples), or when each patient has one pair of test-control restorations (the case of related samples). However, when these numbers are exceeded, some method must be devised to represent each patient by a single score before applying a test of significance.

Graph 3

Sequential analysis worksheet

Appendix IV

References

1. MORGAN, S. C.: Criteria for the Clinical Evaluation of Dental Anterior Restorative Materials. Residency Report, Dental Health Center, San Francisco, California, 1966.

2. PARKER, W. A.: A Study of Criteria for Clinical Evaluation of Posterior Restorative Materials. Residency Report, Dental Health Center, San Francisco, California, 1967.

3. HOROWITZ, H. S. and PETERSON, J. K.: Evaluation of Examiner Variability and the Use of Radiographs in Determining the Efficacy of Community Fluoridation, Archs Oral Biol., II: 867–875,1966.

4. MARKEN, K.-E.: The Training of Observers, pp., 81–87. In Advances in Fluorine Research and Dental Caries Prevention, Vol. 4, Proceeding of the 12th Congress of the European Organization for Research on Fluorine and Dental Caries Prevention, Utrecht, The Netherlands, 8–11 June, 1965, Oxford, Pergamon Press, 1966.

5. GUILFORD, J. P.: Psychometric Methods, New York, McGraw-Hill Book Company, Inc., p. 292, 1954.

6.PHILLIPS, R. W., BOYD, D. A., HEALEY, N. J., and CRAWFORD, W. H.: Clinical Observations on Amalgams with Known Physical Properties, J. Dent. Res., 22: 167–172, 1943.

7. WILSON, C. J. and RYGE, G.: Clinical Study of Dental Amalgam, J. Amer. Dent. Ass., 66: 673–771, 1963.

8. PAFFENBARGER, G. C., SCHOONOVER, I. C., and SOUDER, W.: Dental Silicate Cements: Physical and Chemical Properties and a Specification, J. Amer. Dent. Ass., Vol. 25, January 1938.

9. CIVJAN, S. and BRAUER, G. M.: Physical Properties of Cements Based on Zinc Oxide, Hydrogenated Rosin, o-Ethoxybenzoic Acid, and Eugenol, J. Dent. Res., 43: 281–299, 1964.

10. CIVJAN, S. and BRAUER, G. M.: Clinical Behavior of o-Ethoxybenzoid Acid - Eugenol - Zinc Oxide Cements, J. Dent. Res. 44: 80–83, 1965.

11. MACRAE, P. D., ZACHERL, W., and CASTALDI, C. R.: A Study of Defects in Class II Dental Amalgam Restorations in Deciduous Molars, J. Canad. Dent. Ass., 28: 491-502, 1962.

12. GOING, R. E., MASSLER, J., and DUTE, H. L.: Marginal Penetration of Dental Restorations by Different Radioactive Isotopes, J. Dent. Res., 39: 273–284, 1960.

13. RYGE, G.: Dental Materials, pp., 253-267. In Goldman, H. M., Forrest, S. P., Byrd, D. L., and McDonald, R. E. (eds.): Current Therapy in Dentistry, Vol. 3. Saint Louis, The C. V. Mosby Company, 1968.

14. CHILTON, N. W.: Design and Analysis in Dental and Oral Research. J. B. Lippincott, 1967.

15. ARMITAGE, P.: Tests for Linear Trends in Proportions and Frequencies, Biometrics, 11: 375–386, Sept. 1955.

16. KISH, L.: Some Statistical Problems in Research Design, Amer. Soc. Review, 24: 328–388, June, 1959.