WeGaze: Large-Space Gaze Analysis via Multi-camera System for Social Interactions
Ocular metrics are essential for understanding human cognition, behavior, and interpersonal dynamics. Tracking multi-person ocular dynamics in large spaces in a non-invasive manner is a challenging technical problem. Current methodologies often rely on intrusive wearable trackers that can disrupt natural behavior and authentic interactions among individuals. While multi-camera setups hold significant promise for improving accuracy and robustness in complex settings, gaze estimation with multi-camera systems has yet to be thoroughly investigated in research. The ultimate objective of WeGaze is to expand the possibilities of gaze and ocular dynamic research by developing a novel system for detecting and analyzing ocular dynamics of multiple humans with multi-cameras in a large space for naturalistic settings. Our steps involve: (1) Leveraging our unique expertise in the multi-view RGB camera system and incorporating the professional eye tracker, we will collect an advanced dataset from multi-view cameras with ground-truth. (2) With the collected data, we will develop a new gaze estimation algorithm for multi-view camera settings. (3) We design a new setup to capture interactions involving multiple participants in large-spaces. (4) We extend the developed algorithm to accommodate multi-person scenarios, enabling high-level eye movement analysis, including shared and mutual gaze behavior, in dynamic, room-scale environments. (5) Finally, we will evaluate the developed model in an environment, such as a room equipped with multi-cameras where multiple people are having natural conversations. The developed algorithm will be open source and distributed initially to the UGaze network. To ensure its usability, we will establish accuracy criteria focusing on gaze direction, eye blinks and interaction-based labels such as mutual gaze and gaze-sharing. This project will help expand the current limits of eye tracking technology and enable application of gaze studies in multi-person, naturalistic settings.
Deutsche Forschungsgemeinschaft (DFG) - Project number 563112506
Dr. Senya Polikovsky
Optics and Sensing Lab (OSLab), MPI for Intelligent Systems, Tübingen
Dr. Senya Polikovsky has been a research engineer at the Optics and Sensing Laboratory of the Max Planck Institute for Intelligent Systems in Tübingen since 2014. He develops algorithms and sensor systems for multimodal data acquisition, image processing and precise motion analysis, which are used in robotic perception and human sensing applications. He studied Electronic and Communication Engineering at the Holon Institute of Technology and subsequently obtained an M.Sc. and a PhD in Computer Vision from the University of Tsukuba. Before joining the MPI, he worked as a Research Fellow at the Centre for Computational Sciences in Tsukuba. He has also worked on several industrial development projects in image processing, photogrammetry and sensor technologies.
Team members
Gökce Ergün
Optics and Sensing Lab (OSLab), MPI for Intelligent Systems, Tübingen
Gökce Ergün is a research engineer at the Optics and Sensing Lab of the Max-Planck-Institute for Intelligent Systems. She works on the development of multi-sensor and multi-camera acquisition systems and the implementation of software solutions for computer-assisted motion and behaviour analysis. Previously, she worked at the MPI as a software programmer and as an intern in the field of pose estimation. She studied psychology, earning an M.A. in Clinical Psychology from Doğuş University in 2011 and a M.S. in Applied Information and Data Science from Lucerne University of Applied Sciences and Arts in 2022. Since 2017, she has also been pursuing a Bachelor's degree in Mathematics and Statistics.
Anastasios Yiannakidis
Perceiving Systems Department, MPI for Intelligent Systems, Tübingen
Anastasios Yiannakidis is a Research Software Engineer in the Perceiving Systems department at the Max-Planck-Institute for Intelligent Systems in Tübingen. Previously, he worked as a Research Scientist at the University of Cyprus and as a Research Intern at the CYENS Centre of Excellence. He studied Computer Science at the University of Cyprus. His work focuses on computer vision, 3D pose and motion reconstruction, and synthetic data generation. He has been involved in various projects on realistic human and animal motion models and large-scale datasets for pose estimation. He is currently developing scalable pipelines for motion synthesis, systems for 3D digitisation of animal movements, and data-driven virtual environments.
Xucong Zhang
Intelligent Systems Department, Faculty of EEMCS, Delft University of Technology (TU Delft)
Dr. Xucong Zhang has been an assistant professor at TU Delft since 2021, where he conducts research on human-centred computer vision, gaze behaviour and whole-body motion modelling. Previously, he was a postdoctoral researcher at the Advanced Interaction Technologies Lab at ETH Zurich. He received his PhD from the Max-Planck-Institute for Informatics. His research focuses on embodied visual intelligence, in particular the holistic modelling of human motion using AI. The aim is to enable realistic avatars, robust human-robot interaction and comprehensive human-centred AI systems. He also works at the interface of computer vision, medicine and health research to develop technological tools that support clinical applications and create societal benefits. Zhang is also a Guest Scientist at the MPI for Intelligent Systems, an ELLIS Scholar and an IEEE Senior Member.