Interactive Realities Lab

While many examples of multi-modal virtual environments exist, they are usually limited in number (perhaps visual coupled with aural). There is a need for research in expanding and unifying modalities and their understanding. Bringing together four modalities (visual, aural, haptic and olfactory) presents a unique challenge in creating a flexible yet simple software infrastructure. Some of the modalities fit in easily while others pose a significant challenge.

Virtual environment systems, especially those supporting multi-modal interactions require a robust and flexible software infrastructure that supports a wide range of devices, interaction techniques, and target applications. In addition to interactivity needs, a key factor of robustness of the software is the minimization of latency and more importantly, reduction of jitter (the variability of latency). We are researching a flexible software infrastructure that has demonstrated robustness in initial prototyping. The infrastructure, based on the VESS Library, simplifies the task of creating multi-modal virtual environments. Our extensions to VESS include numerous features to support new input and output devices for new sensory modalities and interaction techniques, as well as some control over latency and jitter.

In addition, we have also been performing research in collaborative environments and the interaction present in multimodal environments within a collaborative setting.  We are building a network architecture (called GEMINI) for collaborative, multimodal environments with a focus on interaction.  Within a dynamic world three types of human interaction are possible: user-to-user, world-to-user, and user-to-world.  The first two form what are fundamentally multimodal environments; the user receives feedback using various modalities both from other users and the world.  The last one forms the area of dynamic environments where the user can affect the world itself.  The architecture we are building provides a flexible mechanism for studying multimodal environments in this distributed sense.