Construct validity of modified time-interval analysis in measuring stuttering and trained speaking patterns

https://doi.org/10.1016/j.jfludis.2011.11.006Get rights and content

Abstract

Purpose

The aim of the present study was to examine the construct validity of modified time-interval analysis. This measure allows judgments on stuttered and fluent speech as well as trained speaking patterns and might be valuable for outcome studies.

Method

Construct validity was investigated in an intervention study with two treatment groups (24 clients received stuttering modification treatment, 30 clients received fluency modification treatment) and a control group (38 stuttering adults). All participants were interviewed during surprise phone calls before and after treatment; the speech samples were analyzed by means of modified time-interval analysis and stuttering frequency counts.

Results

The outcomes confirmed prior hypotheses for the most part. First, the amount of trained speaking patterns after therapy was significantly higher in both treatment groups than in the control group. Secondly, longitudinal changes in the treatment groups met prior expectations based on differing treatment goals and exceeded the changes in the control group. Modified time-interval analysis was sufficiently sensitive to detect changes of speech fluency, but underestimated spontaneous fluent speech when trained speaking patterns were applied.

Conclusion

The present study supports construct validity of modified time-interval analysis in measuring stuttering and trained speaking patterns, but also reveals a lack of accuracy.

Educational objectives: At the end of this activity the reader will be able to (a) explain different forms of validity in relation to the use of modified time-interval analysis, (b) evaluate whether construct validity of modified time-interval analysis has been supported by the outcomes of an intervention study and (c) describe the usefulness and limitations of modified time-interval analysis for future research.

Highlights

► Modified time-interval analysis measures speech fluency after stuttering treatment. ► We investigated construct validity of this measure in an intervention study. ► Changes in speech fluency were captured by modified time-interval analysis. ► Modified time-interval analysis is less accurate than traditional frequency counts.

Introduction

Stuttering is a disorder of speech fluency that affects approximately 1% of the world's population. Throughout its course, stuttering usually develops in complexity and severity and can become a socially disabling condition for adults (Craig, 2000). Stuttering in adulthood can lead to social phobia, elevated levels of distress and negative mood states and has a significant negative impact on quality of life (Craig et al., 2009, Cummins, 2010, Iverach et al., 2009). Available treatment interventions for adolescents and adults are numerous and include among others behavioral intervention as well as the use of assistive devices (e.g., DAF), and pharmaceutical agents (Bloodstein & Bernstein Ratner, 2008). Among the behavioral approaches fluency modification and stuttering modification are the most common (Conture & Curlee, 2007). Although both approaches address stuttering, they differ in their treatment goals. Fluency modification aims for controlled fluency by use of a novel speaking pattern and for spontaneous fluency (Guitar & McCauley, 2010). However, therapists prefer the use of controlled fluency as this has the benefit of facilitating transfer and maintenance. Stuttering modification also seeks to achieve controlled fluency (e.g., use of preparatory sets), but includes acceptable stuttering (e.g., stuttering without secondary behaviors, use of pull-outs or cancellations) and spontaneous fluency in their treatment goals (Guitar, 2006). According to Guitar (2006), the results of both approaches are quite similar in that they both involve a modified style of speaking (or controlled fluency).

A substantial body of literature has demonstrated that behavioral treatment of stuttering adults can be effective, resulting in a significant decrease in stuttering frequency (Andrews et al., 1980, Bothe et al., 2006, Herder et al., 2006). In almost all of these studies stuttering frequency counts, expressed by the percentage of stuttered syllables or words, have been used to demonstrate changes in fluency after treatment. Despite its popularity, the reliability of stuttering frequency counts have often been questioned because of reports of low interjudge agreement (Curlee, 1981, Ingham and Cordes, 1992, Kully and Boberg, 1988). As an alternative, Cordes, Ingham, Frank, and Ingham (1992) introduced the time-interval analysis. This analysis does not focus on individual stuttering events but on the occurrence of a stuttering event within a defined time interval. Meanwhile, satisfactory interjudge and intrajudge agreement could be demonstrated for both time-interval analysis (Cordes and Ingham, 1994a, Cordes and Ingham, 1994b, Cordes and Ingham, 1995, Cordes and Ingham, 1996, Cordes et al., 1992, Einarsdóttir and Ingham, 2008, Ingham et al., 1993a, Ingham et al., 1993b) and syllable-based stuttering frequency counts (Cordes and Ingham, 1994a, Cordes and Ingham, 1994b, Yaruss, 1998).

Regardless of the basic unit over which stuttering frequency is averaged (syllables, words or time intervals), it is interesting to note that the use of trained speaking patterns (which results in controlled fluency) is not considered in stutter count measures. As argued earlier, judges might be inclined to count most fluency modification techniques as fluent syllables, whereas stuttering modification techniques, such as a pull-out, might rather be scored as stuttered syllables (Alpermann, Huber, Natke, & Willmes, 2010). For this reason, Natke (2005a) suggested a modification of time-interval analysis, called modified time-interval analysis, by adding the category “trained speaking patterns” to the existing categories “fluent” and “stuttered”. Such, spontaneously fluent speech could be measured under the category “fluent” while the use of controlled fluency could be measured by the category “trained speaking pattern”. A modified version of time-interval analysis might be valuable as an additional measure in outcome studies and it might be useful for exploring the relationship between discontinuation of trained speaking patterns and (long-term) post-treatment relapse.

Modified time-interval analysis relies on the assumption that trained speaking patterns can be identified and distinguished from spontaneous fluency in a reliable way. Indeed, several authors showed that post-treatment speech deviates from spontaneous fluency in its naturalness and that this difference in naturalness can be measured reliably (Franken et al., 1995, Martin et al., 1984, Onslow et al., 1992a, Runyan et al., 1990, Teshima et al., 2010). These results were corroborated by findings that the speech of stuttering adults changes after treatment in terms of acoustic parameters such as reduced variability of vowel duration (Onslow et al., 1992b, Packman et al., 1994).

More specifically, Onslow and O’Brian (1998) showed that experienced clinicians can judge the presence of prolonged speech (gentle onsets, soft contacts, and continuous vocalization) in different adults who stutter with high intrajudge (94.7%) and interjudge agreement (98.2%). For a stuttering modification approach, Eichstädt, Watt, and Gierson (1998) demonstrated substantial intrajudge (κ = 0.74) and interjudge (κ = 0.69) agreement in the measurement of stuttering modification techniques (prolongation, pull-out, cancellation).

Besides, Alpermann et al. (2010) investigated interjudge and intrajudge agreement of modified time-interval analysis for German stuttering specialists. It was found that, overall, interjudge and intrajudge agreement met the typical requirement of 80% and was comparable to the level of agreement among American fluency specialists only on stuttered and fluent speech (Cordes & Ingham, 1995). In a follow-up study the results could be extended by demonstrating that inexperienced clinicians can also make reliable and accurate judgments with modified time-interval analysis after having received training.

As described above, modified time-interval analysis is a promising tool for measuring stuttered and fluent speech as well as the use of trained speaking patterns (controlled fluency) reliably. As the three categories (fluent, stuttered, trained speaking patterns) are evaluated at the same time, this measure has the advantage of being easy and quick in execution. However, the co-occurrence of the three categories within one time interval is problematic. Even for a person who stutters frequently, there will always be some spontaneously fluent syllables or words within a time interval. Similarly, stuttering could co-occur with one or more trained speaking patterns within one time interval. Furthermore, the three categories might not be distinguishable at all, e.g., a pull-out might be considered as stuttering or a very natural and unobtrusive use of prolonged speech might sound like spontaneous fluency. Acknowledging these threats to the validity of modified time-interval analysis, we designed this study to investigate the validity of this measure.

In general, three basic types of validity can be distinguished (Schiavetti & Metz, 2006). Content validity reflects how well a measure samples the intended behavior or characteristic to be measured. Criterion validity refers to the degree to which a measure correlates with a known indicator of the behavior or characteristic it is supposed to measure. Finally, construct validity reflects the ability of an instrument to measure an abstract concept, or construct (Portney & Watkins, 2009), in this case the ability of modified time-interval analysis to measure trained speaking patterns. According to Messick (1989), construct validity also subsumes other types of validity, such as criterion-related validity and content validity (p. 17), and must be pursued to justify the use of a test or measure. While, historically, primary emphasis in construct validation has been placed on patterns of relationships among item scores or between test scores and other measures, Messick (1989) stated, “[] probably even more illuminating of core meaning […] are studies of performance differences over time, across groups and settings, and in response to experimental treatments and manipulations” (p. 17). Following Messicks considerations, the purpose of this study was to investigate the construct validity of modified time-interval analysis by means of an intervention study with two treatment groups and one control group. Whereas the first treatment group received treatment according to the fluency modification approach, the other group was treated according to the stuttering modification approach.

Several commonly employed analysis procedures were used to assess construct validity of the feasibility of modified time-interval analysis. The known groups method (Portney & Watkins, 2009) implied for this study that the two treatment groups could be distinguished from the control group by the use of trained speaking patterns. Longitudinal construct validity, also referred to as sensitivity to change (Liang, 2000), included the capacity of modified time-interval analysis to measure statistically significant change (due to the interventions). In this context, effect sizes were calculated as they are currently the most accepted and widely applied indices of sensitivity to change (Igl, 2007, Streiner and Norman, 2008). Finally, syllable-based stuttering frequency counts enabled investigation of convergent and discriminant validity as dimensions of construct validity (Cordes & Ingham, 1994a). Specifically, we investigated the following hypotheses:

“Known groups validity”: only after treatment, both treatment groups will use significantly more instances of trained speaking patterns than the control group.

Sensitivity to change: (a) the percentage of stuttered time intervals will decrease from pre- to post-assessment significantly more in both treatment groups than in the control group. (b) The amount of trained speaking patterns will increase significantly more in the treatment groups than in the control group. (c) The percentage of spontaneously fluent time intervals will decrease significantly more in the fluency modification group than in the control group. In the stuttering modification group this percentage will increase significantly more than in the control group.

Convergent and discriminant validity: (a) the percentages of fluent and stuttered time intervals of modified time-interval analysis will correlate significantly with the percentage of stuttered syllables. (b) The percentage of time intervals with trained speaking patterns will not correlate significantly with the percentage of stuttered syllables.

Section snippets

Participants

A total of 92 stuttering adults provided the data for investigating construct validity of modified time-interval analysis. These adults belonged to three different groups: (a) twenty-four participants (20 men and 4 women; mean age 31 years, range 16–62 years) who received a 1-year long group-therapy following a stuttering modification-approach, (b) 30 clients (22 men and 8 women; mean age 25 years, range 14–52 years) who followed a two-week group-treatment based on the fluency

Known groups validity

“Known groups analysis” for the percentage of trained speaking patterns showed only minor group differences before treatment (see Table 1). This was supported by the one-way ANOVA, which failed to reveal a significant difference between the three groups (F(2,89) = 2.09, p = .130). At post-assessment, the number of trained speaking patterns increased in both treatment groups, whereas the percentage of trained speaking patterns in the control group remained about the same (see Table 1). Because of

Interpretation of main findings based on hypotheses

The purpose of this study was to investigate the construct validity of modified time-interval analysis. Results confirmed the validity of modified time-interval analysis as these scores were most clearly consistent with our hypotheses.

Conclusions

In summary, evidence from this study supports the construct validity of modified time-interval analysis. It seems that this measure allows more detailed insight into changes of fluency after different treatment approaches than counts of stuttering frequency only. However, modified time-interval analysis tends to underestimate spontaneous fluent speech and overestimates reductions in stuttering after therapy. Consequently, it would be premature to abandon frequency counts in favor of modified

Acknowledgments

This work was supported by a research grant to the first author from the German association of speech-language therapy (DBL e.V.), and a dissertation grant from Zuyd University Heerlen (NL). We thank students Stephanie Fischer and Katharina Schwambach for their analyses in this study and express our gratitude to the clients who participated in this study.

Anke Alpermann was born in Mannheim, Germany. She studied speech-language pathology at the HAN University of Applied Sciences and the RWTH Aachen University. Recently she finished her doctoral studies at the Department of Neurology, Section Neurolinguistics, RWTH Aachen University and works now in a private practice specialized in stuttering treatment.

References (47)

  • O. Bloodstein et al.

    A handbook on stuttering

    (2008)
  • E. Boberg et al.

    Long-term results of an intensive treatment program for adults and adolescents who stutter

    Journal of Speech and Hearing Research

    (1994)
  • A.K. Bothe et al.

    Stuttering treatment research 1970–2005: I. Systematic review incorporating trial quality assessment of behavioral, cognitive, and related approaches

    American Journal of Speech Language Pathology

    (2006)
  • J. Cohen

    Statistical power analysis for behavioral sciences

    (1977)
  • E.G. Conture et al.

    Stuttering and related disorders of fluency

    (2007)
  • A.K. Cordes et al.

    The reliability of observational data: II. Issues in the identification and measurement of stuttering events

    Journal of Speech and Hearing Research

    (1994)
  • A.K. Cordes et al.

    Time-interval measurement of stuttering: Effects of training with highly agreed or poorly agreed exemplars

    Journal of Speech and Hearing Research

    (1994)
  • A.K. Cordes et al.

    Judgments of stuttered and nonstuttered intervals by recognized authorities in stuttering research

    Journal of Speech and Hearing Research

    (1995)
  • A.K. Cordes et al.

    Time-interval measurement of stuttering: Establishing and modifying judgment accuracy

    Journal of Speech and Hearing Research

    (1996)
  • A.K. Cordes et al.

    Time-interval analysis of interjudge and intrajudge agreement for stuttering event judgments

    Journal of Speech and Hearing Research

    (1992)
  • A.R. Craig

    The developmental nature and effective treatment of stuttering in children and adolescents

    Journal of Developmental and Physical Disabilities

    (2000)
  • R.F. Curlee

    Observer agreement on disfluency and stuttering

    Journal of Speech and Hearing Research

    (1981)
  • P. Finn et al.

    Stutterers’ self-ratings of how natural speech sounds and feels

    Journal of Speech and Hearing Research

    (1994)
  • Anke Alpermann was born in Mannheim, Germany. She studied speech-language pathology at the HAN University of Applied Sciences and the RWTH Aachen University. Recently she finished her doctoral studies at the Department of Neurology, Section Neurolinguistics, RWTH Aachen University and works now in a private practice specialized in stuttering treatment.

    Walter Huber was born in 1945, is a professor emeritus of neurolinguistics at the Neurology Department of the RWTH Aachen University in Germany, where he has been directing the speech/language clinic and the aphasia ward. Dr. Huber has been trained in general linguistics at the Free University in Berlin and at Harvard/MIT in Cambridge, Massachusetts. He obtained his doctoral degree (Dr. Phil.) in 1980 with a dissertation on generative syntax of German, and his medical habilitation in 1985 for his research on aphasia. Since 1991, Dr. Huber has been coordinating the newly developed study program for teaching and research logopedics. His ongoing research activities are still in the field of aphasia with recent emphasis on the study of functional reorganization by means of brain imaging. Other research interests are disorders of language development, fluency disorders, dyslexia and sign language. Dr. Huber has published three books, two standardized test batteries and more than 140 research articles. In 2000, he and his team have received the Helmut-Bauer Award for Rehabilitation from the German Neurological Society.

    Ulrich Natke was born in Bremen, Germany. After studying mathematics he worked as a lecturer and researcher with special interest in fluency disorders at the Institute of Experimental Psychology of the Heinrich-Heine-University Düsseldorf. In 1999 he finished his doctoral dissertation about sensorimotor control of fluent and stuttered speech. Ulrich Natke is the author of a German monograph on stuttering. As a person who stutters he is also engaged in the fields of self-help, stuttering treatment, and evaluation of treatment outcomes.

    Klaus Willmes was born in Arnsberg, Germany. He holds a M.Sc. degree both in mathematics and psychology from the RWTH Aachen University. In 1987 he finished his dissertation in psychology about multivariate permutation tests at the University of Trier, Germany. His habilitation was at the University of Bielefeld, Germany, in 1994 on psychometrics in neuropsychology. Since 1997 he is full professor of Neuropsychology at the Medical Faculty of the RWTH Aachen University, closely collaborating with the Section Neurolinguistics.

    View full text