Kyungho Lee
ABSTRACT
A conductor provides a single unified vision of how to interpret and perform music. However, perceiving a conductor's musical intention and expression is quite challenging as they convey information to performers with subtle, nuanced, and highly individualized gestures. This artwork visualizes the conductor's gestures in order to give the audience a better understanding of its expressivity. To represent the expressivity of the gestures, we created motion profiles over eight frames, at 30 frames per second, and compared them to previously modeled gestures using three motion factors, called Weight, Space and Time from related concepts in Laban Movement Analysis (LMA). Based on this, we have created a real-time, interactive visualization that is driven by the motion factor parameters. The visualization receives the input video stream, and it is transformed into a representation of the three motion factors extracted from the real-time conducting gestures.
- Ashley, R. The pragmatics of conducting: Analyzing and interpreting conductors' expressive gestures. In Sixth International Conference on Music Perception and Cognition, ICMPC'00 (2000).Google Scholar
- Buck, B., MacRitchie, J., and Bailey, N. J. The interpretive shaping of embodied musical structure in piano performance. Empirical Musicology Review 8, 2 (2013), 92--119.Google ScholarCross Ref
- Camurri, A., Volpe, G., Poli, G. D., and Leman, M. Communicating expressiveness and affect in multimodal interactive systems. IEEE Multimedia 12, 1 (2005), 43--53. Google ScholarDigital Library
- Cox, D. J. Astral Projection: Theories of Metaphor, Philosophies of Science, and the Art of Scientific Visualization. PhD thesis, University of Plymouth, 2008.Google Scholar
- Devendorf, L., and Ryokai, K. AnyType: provoking reflection and exploration with aesthetic interaction. In Proceedings of the SIGCHI Conference on Human Factors in Computing Systems, ACM (2013), 1041--1050. Google ScholarDigital Library
- Forbes, A. G., Höllerer, T., and Legrady, G. Generative fluid profiles for interactive media arts projects. In Proceedings of the Symposium on Computational Aesthetics, CA'13, ACM (2013), 37--43. Google ScholarDigital Library
- Gambetta, C. L. Conducting outside the box: Creating a fresh approach to conducting gesture through the principles of Laban Movement Analysis. D.m.a., University of North Carolina at Greensboro, 2005.Google Scholar
- James, J., Ingalls, T., Qian, G., Olsen, L., Whiteley, D., Wong, S., and Rikakis, T. Movement-based interactive dance performance. In Proceedings of the 14th annual ACM international conference on Multimedia, MM'06 (2006), 470--480. Google ScholarDigital Library
- Jensenius, A. R., Wanderley, M. M., Gody, R. I., and Leman, M. Musical gestures: Concepts and methods in research. Routledge, 2010.Google Scholar
- Kanungo, T., Mount, D. M., Netanyahu, N. S., Piatko, C. D., Silverman, R., and Wu, A. Y. An efficient k-means clustering algorithm: Analysis and implementation. Pattern Analysis and Machine Intelligence, IEEE Transactions on 24, 7 (2002), 881--892. Google ScholarDigital Library
- Kendon, A. Do gestures communicate? a review. Research on Language & Social Interaction 27, 3 (1994), 175--200.Google ScholarCross Ref
- Laban, R. v. The Mastery of Movement on the Stage. Macdonald and Evans, 1950.Google Scholar
- Latulipe, C., Carroll, E. A., and Lottridge, D. Evaluating longitudinal projects combining technology with temporal arts. In Proceedings of the SIGCHI Conference on Human Factors in Computing Systems, CHI '11, ACM (2011), 1835-1844. Google ScholarDigital Library
- Lau, A., and Moere, A. V. Towards a model of information aesthetics in information visualization. In Information Visualization, 2007. IV'07. 11th International Conference of, IV'07 (2007), 87--92. Google ScholarDigital Library
- Lee, E., Kiel, H., Dedenbach, S., Grüll, I., Karrer, T., Wolf, M., and Borchers, J. iSymphony: An adaptive interactive orchestral conducting system for digital audio and video streams. In CHI '06 Extended Abstracts on Human Factors in Computing Systems, CHI EA '06, ACM (2006), 259--262. Google ScholarDigital Library
- Lee, K. Towards an Improved Baton Technique: The Application and Modification of Conducting Gestures Drawn from the Methods of Rudolf, Green and Saito for Enhanced Performance of Orchestral Interpretations. D.m.a dissertation, The University of Arizona, 2008.Google Scholar
- Lee, K. Visualizing expressiveness in conducting gestures. Proceedings of the IEEE VIS 2014 Arts Program (2014).Google Scholar
- Mathews, M. V. The radio baton and conductor program, or: Pitch, the most important and least expressive part of music. Computer Music Journal (1991), 37--46.Google ScholarCross Ref
- McNeill, D. Gesture and thought. University of Chicago Press, 2008.Google Scholar
- Nakra, T. A. M. Toward an understanding of musical gesture: Mapping expressive intention with the digital baton. Master's thesis, Massachusetts Institute of Technology, Program in Media Arts & Sciences, 1996.Google Scholar
- O'Brien, H. L., and Toms, E. G. What is user engagement? a conceptual framework for defining user engagement with technology. Journal of the American Society for Information Science & Technology 59, 6 (2008), 938--955. Google ScholarDigital Library
- Petersen, M.G., I. O. K. P., and Ludvigsen, M. Aesthetic interaction: a pragmatist's aesthetics of interactive systems. In Proceedings of the 5th conference on Designing interactive systems: processes, practices, methods, and techniques, DIS'04 (2004). Google ScholarDigital Library
- Sand, P., and Teller, S. Video matching. In ACM SIGGRAPH 2004 Papers, SIGGRAPH '04, ACM (2004), 592--599. Google ScholarDigital Library
- Shiffman, D. Reactive. In ACM SIGGRAPH 2004 Emerging Technologies, SIGGRAPH '04, 22--. Google ScholarDigital Library
- Subyen, P., Maranan, D., Schiphorst, T., Pasquier, P., and Bartram, L. EMVIZ: the poetics of movement quality visualization. In Proceedings of the International Symposium on Computational Aesthetics in Graphics, Visualization, and Imaging (2011), 121--128. Google ScholarDigital Library
- Veit, M., and Herrmann, S. Model-view-controller and object teams: A perfect match of paradigms. In Proceedings of the 2nd international conference on Aspect-oriented software development, ACM (2003), 140--149. Google ScholarDigital Library
- Wakin, D. J. Breaking conductors' down by gesture and body part, 2012.Google Scholar
- Winkler, T. Making motion musical: Gesture mapping strategies for interactive computer music. In Proceedings of the International Computer Music Conference, ICMC'95 (1995), 261--264.Google Scholar
- Wright, M. Open sound control: an enabling technology for musical networking. Organised Sound 10, 3 (2005), 193--200. Google ScholarDigital Library
Index Terms
- Express it!: An Interactive System for Visualizing Expressiveness of Conductor's Gestures
Recommendations
Isochords: visualizing structure in music
GI '07: Proceedings of Graphics Interface 2007Isochords is a visualization of music that aids in the classification of musical structure. The Isochords visualization highlights the consonant intervals between notes and common chords in music. It conveys information about interval quality, chord ...
Personal orchestra: conducting audio/video music recordings
WEDELMUSIC'02: Proceedings of the Second international conference on Web delivering of musicPersonal Orchestra lets anybody conduct an electronic orchestra at a new level of realism: users interact not with a synthetic, but an original audio and video recording of a real orchestra--the Vienna Philharmonic. Nevertheless, they can interactively ...
Towards soundpainting gesture recognition
AM '14: Proceedings of the 9th Audio Mostly: A Conference on Interaction With SoundIn this article, we describe our recent research activities on gesture recognition for soundpainting applications. Soundpainting is a multidisciplinary live composing sign language for musicians, actors, dancers, and visual artists. These gestures are ...
Comments