University of Birmingham

We are pleased to announce that Dr Jianbo Jiao and Prof Ales Leonardis have been successful in a prestigious Amazon Research Award, which will explore PCo3D: Physically Plausible Controllable 3D Generative Models.

Along with awardees from 51 universities in 15 countries, they will have access to Amazon public datasets, in addition to AWS AI/ML services and tools.

We asked Jianbo and Ales to reflect on the award in a short interview:

What are the main goals of your research?

• The primary objectives of this proposed research focus on developing 3D generative deep models that are both physically plausible and controllable. Specifically, we aim to achieve the following:
• Developing 3D Generative Models: Create 3D generative models using simple inputs, such as images captured by smartphones or natural language descriptions (text or speech).
• Utilise state-of-the-art stable diffusion models and incorporate physical properties learned from physics simulators.
• Achieving Precise Control and Manipulation: Ensure physically plausible control and manipulation of the 3D models. Consider intuitive interactions with the environment, adhering to basic deterministic physical laws.
• Integration of control mechanisms during model training or application to pre-trained generative models.
• Comprehensive Analysis and Evaluation: Conduct thorough analysis and evaluation of the proposed generative models. Potentially showcase proof-of-concept demos.

What will the research award enable to you to do?

This prestigious research award will enable us to achieve the aforementioned goals, assist in setting up the experimental system, and facilitate the recruitment of researchers to work on the problem. Another valuable aspect of this award is the access to extensive Amazon resources, including datasets and AWS computational resources. Given the data-driven, deep learning-based nature of this project, it will greatly benefit from such support.

What are the significant impacts of your research?

The impact of this research could be substantial for both academia and industry. As mentioned earlier, controllability and physical understanding are critical challenges in 3D generative models. However, the exploration of physically plausible 3D generative models remains notably lacking in the literature. Existing generative models struggle to produce reasonable results when given instructions related to physical properties. While recent works have delved into physically-aware 3D generation and human generation, they are limited in scale and scope. This proposed research introduces a potential new direction, encouraging follow-up studies within the research community. Furthermore, due to its strong ties to smart mobile applications, the resulting research holds promising applications in competitive industrial settings (e.g. novel apps and smart wearable devices). For example, the ability to generate a physically plausible 3D scene with controllable instructions and interact with the generated objects, as if in the real physical world, holds significant potential. Ultimately, this research has the potential to further dissolve the barrier between the real and virtual worlds.

What are you particularly excited about?