Imagine if you could enter a play by putting on a virtual headset. Now imagine that the characters in the play are shaking your hand or giving you a hug, and that you can feel them do these things.
This is the kind of experience that researchers at Queenâs University are developing through the âMeta-Physical Theatre: Designing Physical Interactions in Virtual Reality Live Performances Using Robotics and Smart Textilesâ project. In a nutshell, the project integrates physical touch into virtual reality (VR) live performances.
is the lead researcher on the project. Dr. Pan is an Assistant Professor in the Faculty of Engineering and Applied Science and a member of Ingenuity Labs Research Institute at Queenâs. He was one of six inaugural recipients of a in 2024, supporting community-focused research that pushes boundaries in technology and society.
Virtual Reality Beyond the Visual
In association with intersectional arts organizations, this project aims to build immersive narratives where participants can not only see and hear virtual characters but can also physically interact with them. It pushes boundaries on VR environments to build immersive environments where touch becomes a part of the narrative structure.
Dr. Panâs idea began years earlier during his time at Disney. While working on Star Wars: Galaxyâs Edge, he developed an immersive experience where visitors could feel an iconic âforce grabâ (when a Jedi summons a lightsabre through the air). âYou would put on a VR headset, and you would see, in the distance, this lightsabre that you can reach out to with your hand and it would start zooming toward you,â he explains. âYou would actually see the lightsabre come into your hand in VR. At the same time, a robot in the real world would deliver a lightsabre prop with the exact same timing and force.â Though the project was ultimately shelved by Disney, Dr. Pan didnât give up on the idea. âI thought there was a lot left on the table by shelving that project.â
Making Touch Feel Real in VR
Of course, there’s no VR theatre without theatre, and Dr Pan’s collaboration with is essential to the project. Wheeler is a fellow Ingenuity Labs and Connected Minds member, Assistant Professor in the DAN School of Drama and Music, and Director of Artistic Research at SpiderWebShow Performance. “Shortly after arriving at Queenâs, I was introduced to Michael⊠we thought it would be really cool to actually have a theatrical narrative that uses interpersonal interactions in VR,â Dr Pan says. Supported by community organizations, Dr. Pan and Wheeler co-created a live VR theatre experience that integrates physical touch. âItâs a high risk, high reward project that Connected Minds was willing to fund.
â[We are] creating this narrative that involves physical interactions with virtual characters. [We are] starting out simple, weâre looking at simple interactions like high fives, or fist bumps, and handovers of objects where you donât necessarily need a lot of fidelity in terms of physical interactions.â
To make these moments feel real, the team uses haptic proxies. As Dr. Pan explains, haptic proxies are physical props that âstand in for haptic interactions you would normally feel in the real world.â For example, a robot-mounted hand can simulate a high five at the exact moment the participant sees it in VR.
However, matching physical actions in the world and virtual actions in VR creates a major technical challenge. The system must align spatial coordinates using motion capture and high-fidelity 3D pose tracking, so that the location of the proxy in the real world matches the location where the VR headset thinks you should be.
Timing matters too. The team must also synchronize physical and virtual actions on the scale of milliseconds. âFor dynamic experiences, itâs even more complicated,â Dr. Pan explains. âParticularly for handovers or high fives, there needs to be not only a physical correlation, but also a temporal correlation. You canât have the high five happen in VR first, followed by it happening 500 milliseconds later in the physical world. It breaks the illusion.â To avoid this lag, the team uses a system that shares information between the VR environment and robotic devices to keep latency low and synchronization precise.
Collaboration Across Disciplines
The project is supported by two arts organizations: and . The former is a Kingston-based arts organization and Canadaâs first live-to-digital performance company. It focuses on exploring the intersection of live performance and digital technology. âWith SpiderWebShow, we work with Adrienne Wong, who is contributing to the dramaturgingâ says Dr. Pan. bCurrent is a Toronto-based company that supports the work of Black and intersectional artists and plays a role in shaping the narrative voice of the theatrical experience. Together, these collaborators ensure that the narrative experience is inclusive and culturally relevant.
For Dr. Pan, itâs important that the creative process among engineers and artists is authentic. âWe are emphasizing the co-creative nature of this project … [Michael and I] talk about these experiences at length and we have many ideas on what we eventually want to do with this technology, but one of the most important steps is weâre not leaving each other out in the dark.â
Beyond Theatre: Next Steps
Dr. Pan has big ideas on where the project and technology could eventually go.
âWe already have inquiries into sports training,â he says. âThereâs lots of implications for being able to customize training regimens for athletes.â For instance, being able to train a hockey goalie in a safe and replicable environment without needing live opponents or expensive setups would be helpful to coaches.
The technology could also support hands-on training for skilled trades, with the potential to lower barriers to technical training and to improve safety. âWe could use a robot to mimic a lathe, and then do operator training in VR, especially when there is a shortage of machine equipment or safety concerns, we could have novices training with haptic proxies before moving on to the physical machine.â
Beyond performance and training, Dr. Pan is also excited about its application to care for the elderly and combatting loneliness. Heâs been speaking to , a Connected Minds researcher at York University, who studies VR in palliative and geriatric care. âLoneliness is a huge issue with the elderly population,â he explains. â[We are exploring] if we can use this technology to make social connections.â
Conclusion
In a world where screens mediate much of our social world, Dr. Pan asks: what if we could reclaim a sense of touch, even through a headset? His project brings together engineering, art, performance and community, showing that immersive technology isnât about tricking the eye, itâs about restoring presence and human connection.
