Neva Cherniavsky
Richard E. Ladner
ABSTRACT
The MobileASL project aims to increase accessibility by enabling Deaf people to communicate over video cell phones in their native language, American Sign Language (ASL). Real-time video over cell phones can be a computationally intensive task that quickly drains the battery, rendering the cell phone useless. Properties of conversational sign language allow us to save power and bits: namely, lower frame rates are possible when one person is not signing due to turn-taking, and signing can potentially employ a lower frame rate than fingerspelling. We conduct a user study with native signers to examine the intelligibility of varying the frame rate based on activity in the video. We then describe several methods for automatically determining the activity of signing or not signing from the video stream in real-time. Our results show that varying the frame rate during turn-taking is a good way to save power without sacrificing intelligibility, and that automatic activity analysis is feasible.
- 3GToday. http://www.3gtoday.com/, 2006.Google Scholar
- L. Aimar, L. Merritt, E. Petit, M. Chen, J. Clay, M. Rullgrd, C. Heine, and A. Izvorski. x264 - a free h264/AVC encoder. http://www.videolan.org/x264.html, 2005.Google Scholar
- A. Cavender, R. E. Ladner, and E. A. Riskin. MobileASL: Intelligibility of sign language video as constrained by mobile phone technology. In Assets '06: Proceedings of the 8th international ACM SIGACCESS conference on Computers and accessibility, pages 71--78, New York, NY, USA, 2006. ACM Press. Google ScholarDigital Library
- C.-C. Chang and C.-J. Lin. LIBSVM: A Library for Support Vector Machines, 2001. Software available at http://www.csie.ntu.edu.tw/~cjlin/libsvm. Google ScholarDigital Library
- J.-C. Chen, K. M. Sivalingam, P. Agrawal, and S. Kishore. A comparison of MAC protocols for wireless local networks based on battery power consumption. In INFOCOM '98, pages 150--157, 1998.Google Scholar
- F. Ciaramello and S. Hemami. "Can you see me now?" an objective metric for predicting intelligibility of compressed american sign language video. In Human Vision and Electronic Imaging 2007, January 2007.Google ScholarCross Ref
- J. Coates and R. Sutton-Spence. Turn-taking patterns in deaf conversation. In Journal of Sociolinguistics, pages 507--529, 2001.Google ScholarCross Ref
- S. Collins and K. Petronio. What happens in tactile ASL? In C. Lucas, editor, Pinky Extension and Eye Gaze: Language Use in Deaf Communities, pages 18--37. Galludet University Press, 1998.Google Scholar
- C. Cortes and V. Vapnik. Support-vector networks. Machine Learning, 20(3):273--297, 1995. Google ScholarDigital Library
- L. Feeney and M. Nilsson. Investigating the energy consumption of a wireless network interface in an ad hoc networking environment. In INFOCOM '01, pages 1548--1557, 2001.Google ScholarCross Ref
- R. A. Foulds. Biomechanical and perceptual constraints on the bandwidth requirements of sign language. In IEEE Trans. On Neural Systems and Rehabilitation Engineering, volume 12, pages 65--72, March 2004.Google ScholarCross Ref
- GSMA. General packet radio service. http://www.gsmworld.com/technology/gprs/class.shtml, 2006.Google Scholar
- B. F. Johnson and J. K. Caird. The effect of frame rate and video information redundancy on the perceptual learning of American Sign Language gestures. In CHI '96: Conference companion on Human factors in computing systems, pages 121--122, New York, NY, USA, 1996. ACM Press. Google ScholarDigital Library
- B. L. Loeding, S. Sarkar, A. Parashar, and A. I. Karshmer. Progress in automated computer recognition of sign language. In Computers Helping People with Special Needs, 9th International Conference, ICCHP 2004, Paris, France, July 7-9, 2004, Proceedings, volume 3118 of Lecture Notes in Computer Science, pages 1079--1087. Springer, 2004.Google Scholar
- R. Mitchell. How many deaf people are there in the United States? http://gri.gallaudet.edu/Demographics/deaf-US.php, 2007.Google Scholar
- S. C. W. Ong and S. Ranganath. Automatic sign language analysis: A survey and the future beyond lexical meaning. IEEE Trans. Pattern Anal. Mach. Intell, 27(6):873--891, 2005. Google ScholarDigital Library
- R. J. Radke, S. Andra, O. Al-Kofahi, and B. Roysam. Image change detection algorithms: A systematic survey. IEEE Transactions on Image Processing, 14(3):294--307, 2005. Google ScholarDigital Library
- I. Richardson. vocdex : H.264 tutorial white papers. http://www.vcodex.com/h264.html, 2004.Google Scholar
- G. Sperling, M. Landy, Y. Cohen, and M. Pavel. Intelligible encoding of ASL image sequences at extremely low information rates. In Papers from the second workshop Vol. 13 on Human and Machine Vision II, pages 256--312, San Diego, CA, USA, 1986. Academic Press Professional, Inc. Google ScholarDigital Library
- M. A. Viredaz and D. A. Wallach. Power evaluation of a handheld computer. IEEE Micro, 23(1):66--74, 2003. Google ScholarDigital Library
- L. Wang, W. Hu, and T. Tan. Recent developments in human motion analysis. Pattern Recognition, 36(3):585--601, 2003.Google ScholarCross Ref
Index Terms
- Variable frame rate for low power mobile sign language communication
Recommendations
MobileASL:: intelligibility of sign language video as constrained by mobile phone technology
Assets '06: Proceedings of the 8th international ACM SIGACCESS conference on Computers and accessibilityFor Deaf people, access to the mobile telephone network in the United States is currently limited to text messaging, forcing communication in English as opposed to American Sign Language (ASL), the preferred language. Because ASL is a visual language, ...
Sign Language Recognition, Generation, and Modelling: A Research Effort with Applications in Deaf Communication
UAHCI '09: Proceedings of the 5th International Conference on Universal Access in Human-Computer Interaction. Addressing Diversity. Part I: Held as Part of HCI International 2009Sign language and Web 2.0 applications are currently incompatible, because of the lack of anonymisation and easy editing of online sign language contributions. This paper describes Dicta-Sign, a project aimed at developing the technologies required for ...
Evaluating Intelligibility and Battery Drain of Mobile Sign Language Video Transmitted at Low Frame Rates and Bit Rates
Special Issue (Part 2) of Papers from ASSETS 2013Mobile sign language video conversations can become unintelligible if high video transmission rates cause network congestion and delayed video. In an effort to understand the perceived lower limits of intelligible sign language video intended for mobile ...
Comments