Construct validity of modified time-interval analysis in measuring stuttering and trained speaking patterns
Highlights
► Modified time-interval analysis measures speech fluency after stuttering treatment. ► We investigated construct validity of this measure in an intervention study. ► Changes in speech fluency were captured by modified time-interval analysis. ► Modified time-interval analysis is less accurate than traditional frequency counts.
Introduction
Stuttering is a disorder of speech fluency that affects approximately 1% of the world's population. Throughout its course, stuttering usually develops in complexity and severity and can become a socially disabling condition for adults (Craig, 2000). Stuttering in adulthood can lead to social phobia, elevated levels of distress and negative mood states and has a significant negative impact on quality of life (Craig et al., 2009, Cummins, 2010, Iverach et al., 2009). Available treatment interventions for adolescents and adults are numerous and include among others behavioral intervention as well as the use of assistive devices (e.g., DAF), and pharmaceutical agents (Bloodstein & Bernstein Ratner, 2008). Among the behavioral approaches fluency modification and stuttering modification are the most common (Conture & Curlee, 2007). Although both approaches address stuttering, they differ in their treatment goals. Fluency modification aims for controlled fluency by use of a novel speaking pattern and for spontaneous fluency (Guitar & McCauley, 2010). However, therapists prefer the use of controlled fluency as this has the benefit of facilitating transfer and maintenance. Stuttering modification also seeks to achieve controlled fluency (e.g., use of preparatory sets), but includes acceptable stuttering (e.g., stuttering without secondary behaviors, use of pull-outs or cancellations) and spontaneous fluency in their treatment goals (Guitar, 2006). According to Guitar (2006), the results of both approaches are quite similar in that they both involve a modified style of speaking (or controlled fluency).
A substantial body of literature has demonstrated that behavioral treatment of stuttering adults can be effective, resulting in a significant decrease in stuttering frequency (Andrews et al., 1980, Bothe et al., 2006, Herder et al., 2006). In almost all of these studies stuttering frequency counts, expressed by the percentage of stuttered syllables or words, have been used to demonstrate changes in fluency after treatment. Despite its popularity, the reliability of stuttering frequency counts have often been questioned because of reports of low interjudge agreement (Curlee, 1981, Ingham and Cordes, 1992, Kully and Boberg, 1988). As an alternative, Cordes, Ingham, Frank, and Ingham (1992) introduced the time-interval analysis. This analysis does not focus on individual stuttering events but on the occurrence of a stuttering event within a defined time interval. Meanwhile, satisfactory interjudge and intrajudge agreement could be demonstrated for both time-interval analysis (Cordes and Ingham, 1994a, Cordes and Ingham, 1994b, Cordes and Ingham, 1995, Cordes and Ingham, 1996, Cordes et al., 1992, Einarsdóttir and Ingham, 2008, Ingham et al., 1993a, Ingham et al., 1993b) and syllable-based stuttering frequency counts (Cordes and Ingham, 1994a, Cordes and Ingham, 1994b, Yaruss, 1998).
Regardless of the basic unit over which stuttering frequency is averaged (syllables, words or time intervals), it is interesting to note that the use of trained speaking patterns (which results in controlled fluency) is not considered in stutter count measures. As argued earlier, judges might be inclined to count most fluency modification techniques as fluent syllables, whereas stuttering modification techniques, such as a pull-out, might rather be scored as stuttered syllables (Alpermann, Huber, Natke, & Willmes, 2010). For this reason, Natke (2005a) suggested a modification of time-interval analysis, called modified time-interval analysis, by adding the category “trained speaking patterns” to the existing categories “fluent” and “stuttered”. Such, spontaneously fluent speech could be measured under the category “fluent” while the use of controlled fluency could be measured by the category “trained speaking pattern”. A modified version of time-interval analysis might be valuable as an additional measure in outcome studies and it might be useful for exploring the relationship between discontinuation of trained speaking patterns and (long-term) post-treatment relapse.
Modified time-interval analysis relies on the assumption that trained speaking patterns can be identified and distinguished from spontaneous fluency in a reliable way. Indeed, several authors showed that post-treatment speech deviates from spontaneous fluency in its naturalness and that this difference in naturalness can be measured reliably (Franken et al., 1995, Martin et al., 1984, Onslow et al., 1992a, Runyan et al., 1990, Teshima et al., 2010). These results were corroborated by findings that the speech of stuttering adults changes after treatment in terms of acoustic parameters such as reduced variability of vowel duration (Onslow et al., 1992b, Packman et al., 1994).
More specifically, Onslow and O’Brian (1998) showed that experienced clinicians can judge the presence of prolonged speech (gentle onsets, soft contacts, and continuous vocalization) in different adults who stutter with high intrajudge (94.7%) and interjudge agreement (98.2%). For a stuttering modification approach, Eichstädt, Watt, and Gierson (1998) demonstrated substantial intrajudge (κ = 0.74) and interjudge (κ = 0.69) agreement in the measurement of stuttering modification techniques (prolongation, pull-out, cancellation).
Besides, Alpermann et al. (2010) investigated interjudge and intrajudge agreement of modified time-interval analysis for German stuttering specialists. It was found that, overall, interjudge and intrajudge agreement met the typical requirement of 80% and was comparable to the level of agreement among American fluency specialists only on stuttered and fluent speech (Cordes & Ingham, 1995). In a follow-up study the results could be extended by demonstrating that inexperienced clinicians can also make reliable and accurate judgments with modified time-interval analysis after having received training.
As described above, modified time-interval analysis is a promising tool for measuring stuttered and fluent speech as well as the use of trained speaking patterns (controlled fluency) reliably. As the three categories (fluent, stuttered, trained speaking patterns) are evaluated at the same time, this measure has the advantage of being easy and quick in execution. However, the co-occurrence of the three categories within one time interval is problematic. Even for a person who stutters frequently, there will always be some spontaneously fluent syllables or words within a time interval. Similarly, stuttering could co-occur with one or more trained speaking patterns within one time interval. Furthermore, the three categories might not be distinguishable at all, e.g., a pull-out might be considered as stuttering or a very natural and unobtrusive use of prolonged speech might sound like spontaneous fluency. Acknowledging these threats to the validity of modified time-interval analysis, we designed this study to investigate the validity of this measure.
In general, three basic types of validity can be distinguished (Schiavetti & Metz, 2006). Content validity reflects how well a measure samples the intended behavior or characteristic to be measured. Criterion validity refers to the degree to which a measure correlates with a known indicator of the behavior or characteristic it is supposed to measure. Finally, construct validity reflects the ability of an instrument to measure an abstract concept, or construct (Portney & Watkins, 2009), in this case the ability of modified time-interval analysis to measure trained speaking patterns. According to Messick (1989), construct validity also subsumes other types of validity, such as criterion-related validity and content validity (p. 17), and must be pursued to justify the use of a test or measure. While, historically, primary emphasis in construct validation has been placed on patterns of relationships among item scores or between test scores and other measures, Messick (1989) stated, “[] probably even more illuminating of core meaning […] are studies of performance differences over time, across groups and settings, and in response to experimental treatments and manipulations” (p. 17). Following Messicks considerations, the purpose of this study was to investigate the construct validity of modified time-interval analysis by means of an intervention study with two treatment groups and one control group. Whereas the first treatment group received treatment according to the fluency modification approach, the other group was treated according to the stuttering modification approach.
Several commonly employed analysis procedures were used to assess construct validity of the feasibility of modified time-interval analysis. The known groups method (Portney & Watkins, 2009) implied for this study that the two treatment groups could be distinguished from the control group by the use of trained speaking patterns. Longitudinal construct validity, also referred to as sensitivity to change (Liang, 2000), included the capacity of modified time-interval analysis to measure statistically significant change (due to the interventions). In this context, effect sizes were calculated as they are currently the most accepted and widely applied indices of sensitivity to change (Igl, 2007, Streiner and Norman, 2008). Finally, syllable-based stuttering frequency counts enabled investigation of convergent and discriminant validity as dimensions of construct validity (Cordes & Ingham, 1994a). Specifically, we investigated the following hypotheses:
“Known groups validity”: only after treatment, both treatment groups will use significantly more instances of trained speaking patterns than the control group.
Sensitivity to change: (a) the percentage of stuttered time intervals will decrease from pre- to post-assessment significantly more in both treatment groups than in the control group. (b) The amount of trained speaking patterns will increase significantly more in the treatment groups than in the control group. (c) The percentage of spontaneously fluent time intervals will decrease significantly more in the fluency modification group than in the control group. In the stuttering modification group this percentage will increase significantly more than in the control group.
Convergent and discriminant validity: (a) the percentages of fluent and stuttered time intervals of modified time-interval analysis will correlate significantly with the percentage of stuttered syllables. (b) The percentage of time intervals with trained speaking patterns will not correlate significantly with the percentage of stuttered syllables.
Section snippets
Participants
A total of 92 stuttering adults provided the data for investigating construct validity of modified time-interval analysis. These adults belonged to three different groups: (a) twenty-four participants (20 men and 4 women; mean age 31 years, range 16–62 years) who received a 1-year long group-therapy following a stuttering modification-approach, (b) 30 clients (22 men and 8 women; mean age 25 years, range 14–52 years) who followed a two-week group-treatment based on the fluency
Known groups validity
“Known groups analysis” for the percentage of trained speaking patterns showed only minor group differences before treatment (see Table 1). This was supported by the one-way ANOVA, which failed to reveal a significant difference between the three groups (F(2,89) = 2.09, p = .130). At post-assessment, the number of trained speaking patterns increased in both treatment groups, whereas the percentage of trained speaking patterns in the control group remained about the same (see Table 1). Because of
Interpretation of main findings based on hypotheses
The purpose of this study was to investigate the construct validity of modified time-interval analysis. Results confirmed the validity of modified time-interval analysis as these scores were most clearly consistent with our hypotheses.
Conclusions
In summary, evidence from this study supports the construct validity of modified time-interval analysis. It seems that this measure allows more detailed insight into changes of fluency after different treatment approaches than counts of stuttering frequency only. However, modified time-interval analysis tends to underestimate spontaneous fluent speech and overestimates reductions in stuttering after therapy. Consequently, it would be premature to abandon frequency counts in favor of modified
Acknowledgments
This work was supported by a research grant to the first author from the German association of speech-language therapy (DBL e.V.), and a dissertation grant from Zuyd University Heerlen (NL). We thank students Stephanie Fischer and Katharina Schwambach for their analyses in this study and express our gratitude to the clients who participated in this study.
Anke Alpermann was born in Mannheim, Germany. She studied speech-language pathology at the HAN University of Applied Sciences and the RWTH Aachen University. Recently she finished her doctoral studies at the Department of Neurology, Section Neurolinguistics, RWTH Aachen University and works now in a private practice specialized in stuttering treatment.
References (47)
- et al.
Measurement of trained speech patterns in stuttering: Interjudge and intrajudge agreement of experts by means of modified time-interval analysis
Journal of Fluency Disorders
(2010) - et al.
The impact of stuttering on the quality of life in adult people who stutter
Journal of Fluency Disorders
(2009) Fluency disorders and life quality: Subjective wellbeing vs health related quality of life
Journal of Fluency Disorders
(2010)- et al.
Evaluation of the efficacy of a stutter modification program with particular reference to two new measures of secondary behaviors and control of stuttering
Journal of Fluency Disorders
(1998) - et al.
The effect of stuttering measurement training on judging stuttering occurrence in preschool children who stutter
Journal of Fluency Disorders
(2008) - et al.
Interclinic differences in stuttering-event counts
Journal of Fluency Disorders
(1992) - et al.
Screening for personality disorders among adults seeking speech treatment for stuttering
Journal of Fluency Disorders
(2009) - et al.
An investigation of interclinic agreement in the identification of fluent and stuttered syllables
Journal of Fluency Disorders
(1988) - et al.
Post-treatment speech naturalness of comprehensive stuttering program clients and differences in ratings among listener groups
Journal of Fluency Disorders
(2010) - et al.
Meta-analysis of the effect of stuttering treatment
Journal of Speech and Hearing Research
(1980)
A handbook on stuttering
Long-term results of an intensive treatment program for adults and adolescents who stutter
Journal of Speech and Hearing Research
Stuttering treatment research 1970–2005: I. Systematic review incorporating trial quality assessment of behavioral, cognitive, and related approaches
American Journal of Speech Language Pathology
Statistical power analysis for behavioral sciences
Stuttering and related disorders of fluency
The reliability of observational data: II. Issues in the identification and measurement of stuttering events
Journal of Speech and Hearing Research
Time-interval measurement of stuttering: Effects of training with highly agreed or poorly agreed exemplars
Journal of Speech and Hearing Research
Judgments of stuttered and nonstuttered intervals by recognized authorities in stuttering research
Journal of Speech and Hearing Research
Time-interval measurement of stuttering: Establishing and modifying judgment accuracy
Journal of Speech and Hearing Research
Time-interval analysis of interjudge and intrajudge agreement for stuttering event judgments
Journal of Speech and Hearing Research
The developmental nature and effective treatment of stuttering in children and adolescents
Journal of Developmental and Physical Disabilities
Observer agreement on disfluency and stuttering
Journal of Speech and Hearing Research
Stutterers’ self-ratings of how natural speech sounds and feels
Journal of Speech and Hearing Research
Cited by (4)
Effectiveness of Stuttering Modification Treatment in School-Age Children Who Stutter: A Randomized Clinical Trial
2023, Journal of Speech, Language, and Hearing ResearchComparison of percentage of syllables stuttered with parent-reported severity ratings as a primary outcome measure in clinical trials of early stuttering treatment
2018, Journal of Speech, Language, and Hearing ResearchEvent- and interval-based measurement of stuttering: A review
2015, International Journal of Language and Communication Disorders
Anke Alpermann was born in Mannheim, Germany. She studied speech-language pathology at the HAN University of Applied Sciences and the RWTH Aachen University. Recently she finished her doctoral studies at the Department of Neurology, Section Neurolinguistics, RWTH Aachen University and works now in a private practice specialized in stuttering treatment.
Walter Huber was born in 1945, is a professor emeritus of neurolinguistics at the Neurology Department of the RWTH Aachen University in Germany, where he has been directing the speech/language clinic and the aphasia ward. Dr. Huber has been trained in general linguistics at the Free University in Berlin and at Harvard/MIT in Cambridge, Massachusetts. He obtained his doctoral degree (Dr. Phil.) in 1980 with a dissertation on generative syntax of German, and his medical habilitation in 1985 for his research on aphasia. Since 1991, Dr. Huber has been coordinating the newly developed study program for teaching and research logopedics. His ongoing research activities are still in the field of aphasia with recent emphasis on the study of functional reorganization by means of brain imaging. Other research interests are disorders of language development, fluency disorders, dyslexia and sign language. Dr. Huber has published three books, two standardized test batteries and more than 140 research articles. In 2000, he and his team have received the Helmut-Bauer Award for Rehabilitation from the German Neurological Society.
Ulrich Natke was born in Bremen, Germany. After studying mathematics he worked as a lecturer and researcher with special interest in fluency disorders at the Institute of Experimental Psychology of the Heinrich-Heine-University Düsseldorf. In 1999 he finished his doctoral dissertation about sensorimotor control of fluent and stuttered speech. Ulrich Natke is the author of a German monograph on stuttering. As a person who stutters he is also engaged in the fields of self-help, stuttering treatment, and evaluation of treatment outcomes.
Klaus Willmes was born in Arnsberg, Germany. He holds a M.Sc. degree both in mathematics and psychology from the RWTH Aachen University. In 1987 he finished his dissertation in psychology about multivariate permutation tests at the University of Trier, Germany. His habilitation was at the University of Bielefeld, Germany, in 1994 on psychometrics in neuropsychology. Since 1997 he is full professor of Neuropsychology at the Medical Faculty of the RWTH Aachen University, closely collaborating with the Section Neurolinguistics.