The Augmented Reality Music Ensemble project allows users to practise their instruments alongside virtual musicians.
The Augmented Reality Music Ensemble (ARME) project was first ideated during the pandemic when musicians were unable to practise together. Even without a pandemic getting in the way, scheduling, isolation, and spatial constraints can be prohibitive in musical collaboration. ARME was conceived as a platform to enable musicians to rehearse virtually together.
With research funding from the Engineering and Physical Sciences Research Council (EPSRC), the ARME project group began to explore the feasibility of creating immersive rehearsal environments. They used pre-recorded audio and video that were time-stretched during replay to create multiple virtual ensemble members with whom musicians could rehearse.
ARME’s development has been marked by collaboration between multiple universities including the University of Birmingham, Birmingham City University, and the University of Warwick combining expertise in music, technology, psychology, and computer science. While the project is a collaborative effort, its core development and experimentation take place primarily in the University of Birmingham’s Virtual Reality Lab, a state-of-the-art facility open to researchers and students interested in advancing AR and VR technologies. This collaborative environment fosters interdisciplinary research and innovation, making it a hub for cutting-edge developments in immersive experiences.
As the project progressed, additional funding streams provided critical support for scaling ARME’s scope and impact, broadening into the inclusion of new technologies such as motion capture and AI and the development of an app and a museum installation.
When asked about the development of ARME, lead researcher Dr Massimiliano Di Luca, Associate Professor in the School of Psychology, said: “The significant advancements of ARME are achieved by tackling the technical challenge of real-time synchronisation. As even small delays can disrupt the synchronicity required for musical performance the local replay of pre-recorded or reconstructed avatars allows for an immersive and interactive music rehearsal. The AI-driven avatars provide a sufficiently realistic simulation of a group rehearsal, capturing the dynamics of live collaboration. Future developments may include allowing avatars to deviate from pre-recorded performances and even the ability to improvise and jam together.”
ARME will continue to evolve into a platform that can supports a range of musical setups, including pop and rock bands, to cater to a broader audience. The platform will integrate cutting-edge technologies such as:
ARME's potential extends beyond individual rehearsal to larger educational and public engagement contexts. For instance, the project includes a museum installation that demonstrates interactive metronomes adapting to user actions, showcasing ARME's broader potential for creative and interactive displays.
Immersive experiences hold promise for reshaping the music industry. By providing tools for interactive rehearsals and performances, ARME paves the way for new creative possibilities.
ARME exemplifies how immersive technologies can redefine musical collaboration. By breaking down geographical barriers, the platform allows musicians worldwide to rehearse together, fostering greater inclusivity and collaboration. This capability is particularly transformative in educational settings, where students can practice alongside virtual representations of professional musicians or receive real-time feedback on their performances.
Looking to the future, Dr Di Luca commented: “Immersive experiences hold promise for reshaping the music industry. By providing tools for interactive rehearsals and performances, ARME paves the way for new creative possibilities. Musicians can experiment with ensemble configurations, explore different genres, and even involve audiences in virtual performances, redefining how music is experienced and shared.”
Associate Professor
Staff profile for Dr Massimiliano (Max) Di Luca, Lecturer in Computational Neuroscience in the Centre for Computational Neuroscience and Cognitive Robotics (CNCR).
