3D-PhysNet: Learning the intuitive physics of non-rigid object deformations

Zhihua Wang, Stefano Rosa, Bo Yang, Sen Wang, Niki Trigoni, Andrew Markham

Research output: Chapter in Book/Report/Conference proceedingConference contribution

7 Citations (Scopus)
45 Downloads (Pure)


The ability to interact and understand the environment is a fundamental prerequisite for a wide range of applications from robotics to augmented reality. In particular, predicting how deformable objects will react to applied forces in real time is a significant challenge. This is further confounded by the fact that shape information about encountered objects in the real world is often impaired by occlusions, noise and missing regions e.g. a robot manipulating an object will only be able to observe a partial view of the entire solid. In this work we present a framework, 3D-PhysNet, which is able to predict how a three-dimensional solid will deform under an applied force using intuitive physics modelling. In particular, we propose a new method to encode the physical properties of the material and the applied force, enabling generalisation over materials. The key is to combine deep variational autoencoders with adversarial training, conditioned on the applied force and the material properties. We further propose a cascaded architecture that takes a single 2.5D depth view of the object and predicts its deformation. Training data is provided by a physics simulator. The network is fast enough to be used in real-time applications from partial views. Experimental results show the viability and the generalisation properties of the proposed architecture.

Original languageEnglish
Title of host publicationProceedings of the Twenty-Seventh International Joint Conference on Artificial Intelligence
PublisherInternational Joint Conferences on Artificial Intelligence
Number of pages7
ISBN (Electronic)9780999241127
Publication statusPublished - 2018
Event27th International Joint Conference on Artificial Intelligence 2018 - Stockholm, Sweden
Duration: 13 Jul 201819 Jul 2018

Publication series

NameProceedings of the International Joint Conference on Artificial Intelligence IJCAI
ISSN (Electronic)1045-0823


Conference27th International Joint Conference on Artificial Intelligence 2018
Abbreviated titleIJCAI 2018

ASJC Scopus subject areas

  • Artificial Intelligence


Dive into the research topics of '3D-PhysNet: Learning the intuitive physics of non-rigid object deformations'. Together they form a unique fingerprint.

Cite this