Between the real and imaginary worlds: Interactive Mixed Reality in real time
Dr Fang-Lue Zhang from Victoria University of Wellington will develop a novel framework for enriched immersive experiences in Mixed Reality
Mixed Reality (MR) is a blend of the physical and digital worlds. A given MR experience can sit anywhere on the MR continuum between physical and virtual reality. Using a combination of wearable screens, cameras and physical controls, this technology provides tremendous potential for opportunities in entertainment, education and other enriched experiences. However, there are still important technological limitations to overcome.
Recently, researchers have developed 360 degree video technology that allows for the capture and augmentation of entire 3D environments. To build an authentic MR experience, it is necessary to reconstruct physical scenes in a fully interactive virtual reality. However, current approaches are poorly adapted to rapidly evolving scenes containing moving objects captured from a moving camera - all features required of a practical MR application. Dr Zhang proposes to use a Marsden Fund Fast-Start grant to develop a framework for scene reconstruction, which goes beyond these restrictions, allowing for MR experiences that synthesise interactive dynamic content in real time.
An illustration of a panoramic scene that contains moving objects, and is combined with a virtual reality head-mounted-device. Dr Zhang will be dealing with these types of moving scenes in his mixed reality project. Phot supplied
The proposed research will bring together many strands of cutting-edge technology and innovation, such as deep learning, spherical image synthesis and MR into a single coherent project for the first time. This will inspire a cross-fertilisation of ideas and enhance Aotearoa New Zealand’s international reputation in computer graphics.