Sundar Murugappan
ABSTRACT
Multitouch surfaces are becoming prevalent, but most existing technologies are only capable of detecting the user's actual points of contact on the surface and not the identity, posture, and handedness of the user. In this paper, we define the concept of extended multitouch interaction as a richer input modality that includes all of this information. We further present a practical solution to achieve this on tabletop displays based on mounting a single commodity depth camera above a horizontal surface. This will enable us to not only detect when the surface is being touched, but also recover the user's exact finger and hand posture, as well as distinguish between different users and their handedness. We validate our approach using two user studies, and deploy the technique in a scratchpad tool and in a pen + touch sketch tool.
Supplemental Material
- H. Benko and A. Wilson. DepthTouch: Using depth- sensing camera to enable freehand interactions on and above the interactive surface. In Proceedings of the IEEE Workshop on Tabletops and Interactive Surfaces, volume 8, 2009.Google Scholar
- P. Brandl, C. Forlines, D. Wigdor, M. Haller, and C. Shen. Combining and measuring the benefits of bi-manual pen and direct-touch interaction on horizontal interfaces. In Proceedings of the ACM Conference on Advanced Visual Interfaces, 154--161, 2008. Google ScholarDigital Library
- K. Dandekar, B. I. Raju, and M. A. Srinivasan. 3-D finite-element models of human and monkey fingertips to investigate the mechanics of tactile sense. Journal of Biomechanical Engineering, 125(5):682--691, 2003.Google ScholarCross Ref
- C. T. Dang, M. Straub, and E. André. Hand distinction for multi-touch tabletop interaction. In Proceedings of the ACM Conference on Interactive Tabletops and Surfaces, 101--108, 2009. Google ScholarDigital Library
- P. Dietz and D. Leigh. DiamondTouch: a multi-user touch technology. In Proceedings of the ACM Symposium on User interface Software and Technology, 219--226, 2001. Google ScholarDigital Library
- A. Dippon and G. Klinker. KinectTouch: accuracy test for a very low-cost 2.5D multitouch tracking system. In Proceedings of the ACM International Conference on Interactive Tabletops and Surfaces, 49--52, 2011. Google ScholarDigital Library
- D. Freeman, H. Benko, M. R. Morris, and D. Wigdor. ShadowGuides: visualizations for in-situ learning of multi-touch and whole-hand gestures. In Proceedings of the ACM Conference on Interactive Tabletops and Sur- faces, 165--172, 2009. Google ScholarDigital Library
- M. Frisch, J. Heydekorn, and R. Dachselt. Investigating multi-touch and pen gestures for diagram editing on interactive surfaces. In Proceedings of ACM Conference on Interactive Tabletops and Surfaces, 149--156, 2009. Google ScholarDigital Library
- Y. Guiard. Asymmetric division of labor in human skilled bimanual action: The kinematic chain as a model. J Mot Behav, 19(4):486--517, 1987.Google ScholarCross Ref
- J. Y. Han. Low-cost multi-touch sensing through frustrated total internal reflection. In Proceedings of the ACM Symposium on User Interface Software and Technology, 115--118, 2005. Google ScholarDigital Library
- C. Harrison, H. Benko, and A. D. Wilson. OmniTouch: wearable multitouch interaction everywhere. In Proceedings of the ACM Symposium on User Interface Software and Technology, 441--450, 2011. Google ScholarDigital Library
- C. Harrison, J. Schwarz, and S. E. Hudson. TapSense: enhancing finger interaction on touch surfaces. In Proceedings of the ACM Symposium on User Interface Software and Technology, 627--636, 2011. Google ScholarDigital Library
- K. Hinckley, K. Yatani, M. Pahud, N. Coddington, J. Rodenhouse, A. Wilson, H. Benko, and B. Buxton. Manual deskterity: an exploration of simultaneous pen + touch direct input. In Extended Abstracts of the ACM Conference on Human Factors in Computing Systems, 2793--2802, 2010. Google ScholarDigital Library
- C. Holz and P. Baudisch. The generalized perceived input point model and how to double touch accuracy by extracting fingerprints. In Proceedings of ACM Conference on Human Factors in Computing Systems, 581-- 590, 2010. Google ScholarDigital Library
- W. Javed, K. Kim, S. Ghani, and N. Elmqvist. Evalu- ating physical/virtual occlusion management techniques for horizontal displays. In Human-Computer Interaction - Proceedings of INTERACT, volume 6948 of Lecture Notes in Computer Science, 391--408. Springer, 2011. Google ScholarDigital Library
- M. Kaltenbrunner. reacTIVision and TUIO: A tangible tabletop toolkit. In Proceedings of the ACM Conference on Interactive Tabletops and Surfaces, 9--16, 2009. Google ScholarDigital Library
- P. Lopes, R. Jota, and J. A. Jorge. Augmenting touch interaction through acoustic sensing. In Proceedings of the ACM International Conference on Interactive Tabletops and Surfaces, 53--56, 2011. Google ScholarDigital Library
- P. Lopes, D. Mendes, B. Araújo, and J. A. Jorge. Combining bimanual manipulation and pen-based input for 3D modelling. In Proceedings of the Eurographics Symposium on Sketch-Based Interfaces and Modeling, 15--22, 2011. Google ScholarDigital Library
- S. Malik and J. Laszlo. Visual touchpad: a two-handed gestural input device. In Proceedings of the ACM International Conference on Multimodal Interfaces, 289--296, 2004. Google ScholarDigital Library
- N. Marquardt, J. Kiemer, D. Ledo, S. Boring, and S. Greenberg. Designing user-, hand-, and handpart- aware tabletop interactions with the TouchID toolkit. In Proceedings of the ACM International Conference on Interactive Tabletops and Surfaces, 21--30, 2011. Google ScholarDigital Library
- M. R. Morris, K. Ryall, C. Shen, C. Forlines, and F. Vernier. Beyond 'social protocols': multi-user coordination policies for co-located groupware. In Proceedings of the ACM Conference on Computer Supported Cooperative Work, 262--265, 2004. Google ScholarDigital Library
- D. J. Sturman and D. Zeltzer. A survey of glove- based input. IEEE Computer Graphics and Applications, 14(1):30--39, Jan. 1994. Google ScholarDigital Library
- D. Vogel and R. Balakrishnan. Occlusion-aware inter- faces. In Proceedings of the ACM Conference on Human Factors in Computing Systems, 263--272, 2010. Google ScholarDigital Library
- D. Vogel and G. Casiez. Hand occlusion on a multi- touch tabletop. In Proceedings of ACM Conference on Human Factors in Computing Systems, 2307--2316, 2012. Google ScholarDigital Library
- F. Wang, X. Cao, X. Ren, and P. Iran. Detecting and leveraging finger orientation for interaction with direct-touch surfaces. In Proceedings of ACM Symposium on User Interface Software and Technology, 23--32, 2009. Google ScholarDigital Library
- R. Y. Wang and J. Popović. Real-time hand-tracking with a color glove. ACM Transactions on Graphics, 28(3), 2009. Google ScholarDigital Library
- M. Weiser. The computer for the twenty-first century. Scientific American, 3(265):94--104, Sept. 1991.Google ScholarCross Ref
- D. Wigdor, H. Benko, J. Pella, J. Lombardo, and S. Williams. Rock & rails: extending multi-touch interactions with shape gestures to enable precise spatial manipulations. In Proceedings of ACM Conference on Human Factors in Computing Systems, 1581--1590, 2011. Google ScholarDigital Library
- A. D. Wilson. TouchLight: an imaging touch screen and display for gesture-based interaction. In Proceedings of the ACM International Conference on Multimodal interfaces, 69--76, 2004. Google ScholarDigital Library
- A. D. Wilson. Using a depth camera as a touch sensor. In Proceedings of the ACM Conference on Interactive Tabletops and Surfaces, 69--72, 2010. Google ScholarDigital Library
- A. D. Wilson and H. Benko. Combining multiple depth cameras and projectors for interactions on, above and between surfaces. In Proceedings of the ACM Symposium on User Interface Software and Technology, 273--282, 2010. Google ScholarDigital Library
- J. O. Wobbrock, M. R. Morris, and A. D. Wilson. User-defined gestures for surface computing. In Proceedings of the ACM Conference on Human Factors in Computing Systems, 1083--1092, 2009. Google ScholarDigital Library
- K.-P. Yee. Two-handed interaction on a tablet display. In Extended Abstracts of the ACM Conference on Human Factors in Computing Systems, 1493--1496, 2004. Google ScholarDigital Library
- R. Zeleznik, A. Bragdon, F. Adeputra, and H.-S. Ko. Hands-on math: a page-based multi-touch and pen desk-top for technical work and problem solving. In Proceedings of the ACM Symposium on User Interface Software and Technology, 17--26, 2010 . Google ScholarDigital Library
- H. Zhang, X.-D. Yang, B. Ens, H.-N. Liang, P. Boulanger, and P. Irani. See me, see you: a lightweight method for discriminating user touches on tabletop displays. In Proceedings of the ACM Conference on Human Factors in Computing Systems, 2327--2336, 2012 Google ScholarDigital Library
Index Terms
- Extended multitouch: recovering touch posture and differentiating users using a depth camera
Recommendations
dSensingNI: a framework for advanced tangible interaction using a depth camera
TEI '12: Proceedings of the Sixth International Conference on Tangible, Embedded and Embodied InteractionThis work presents an approach to overcome the disadvantages of existing interaction frameworks and technologies for touch detection and object interaction. The robust and easy to use framework dSensingNI (Depth Sensing Natural Interaction) is described,...
Determining the benefits of direct-touch, bimanual, and multifinger input on a multitouch workstation
GI '09: Proceedings of Graphics Interface 2009Multitouch workstations support direct-touch, bimanual, and multifinger interaction. Previous studies have separately examined the benefits of these three interaction attributes over mouse-based interactions. In contrast, we present an empirical user ...
Face-to-Face Collaboration Points in Storytelling using Multitouch Tabletop Systems
NHT '15: Proceedings of the 2015 Workshop on Narrative & HypertextThis paper introduces and presents application areas for face-to-face collaboration points in storytelling for adults using multitouch tabletop solutions. The multitouch interaction appears natural and can nicely be used in a group. Furthermore, this ...
Comments