Visual system identiﬁcation: learning physical parameters and latent spaces from pixels
View/Open
Date
20/01/2023Author
Jaques, Miguel
Metadata
Abstract
In this thesis, we develop machine learning systems that are able to leverage the knowledge
of equations of motion (scenespecific or sceneagnostic) to perform object discovery,
physical parameter estimation, position and velocity estimation, camera pose
estimation, and learn structured latent spaces that satisfy physical dynamics rules.
These systems are unsupervised, learning from unlabelled videos, and use as inductive
biases the general equations of motion followed by objects of interest in the scene.
This is an important task as in many complex real world environments groundtruth
states are not available, although there is physical knowledge of the underlying system.
Our goals with this approach, i.e. integration of physics knowledge with unsupervised
learning models, are to improve visionbased prediction, enable new forms of control,
increase dataefficiency and provide model interpretability, all of which are key areas
of interest in machine learning. With the above goals in mind, we start by asking the
following question: given a scene in which the objects’ motions are known up to some
physical parameters (e.g. a ball bouncing off the floor with unknown restitution coefficient),
how do we build a model that uses such knowledge to discover the objects in the
scene and estimate these physical parameters?
Our first model, PAIG (PhysicsasInverseGraphics), approaches this problem from a
visionasinversegraphics perspective, describing the visual scene as a composition of
objects defined by their location and appearance, which are rendered onto the frame in
a graphics manner. This is a known approach in the unsupervised learning literature,
where the fundamental problem then becomes that of derendering, that is, inferring and
discovering these locations and appearances for each object. In PAIG we introduce a
key rendering component, the CoordinateConsistent Decoder, which enables the integration
of the known equations of motion with an inversegraphics autoencoder architecture
(trainable endtoend), to perform simultaneous object discovery and physical
parameter estimation. Although trained on simple simulated 2D scenes, we show that
knowledge of the physical equations of motion of the objects in the scene can be used
to greatly improve future prediction and provide physical scene interpretability.
Our second model, VSysId, tackles the limitations shown by the PAIG architecture,
namely the training difficulty, the restriction to simulated 2D scenes, and the need for
noiseless scenes without distractors. Here, we approach the problem from rst principles
by asking the question: are neural networks a necessary component to solve this
problem? Can we use simpler ideas from classical computer vision instead? With V
SysId, we approach the problem of object discovery and physical parameter estimation
from a keypoint extraction, tracking and selection perspective, composed of 3 separate
stages: proposal keypoint extraction and tracking, 3D equation tting and camera pose
estimation from 2D trajectories, and entropybased trajectory selection. Since all the
stages use lightweight algorithms and optimisers, VSysId is able to perform joint object
discovery, physical parameter and camera pose estimation from even a single video,
drastically improving dataefficiency. Additionally, due to the fact that it does not use a
rendering/derendering approach, it can be used in real 3D scenes with many distractor
objects. We show that this approach enables a number of interest applications, such as
visionbased robot endeffector localisation and remote breath rate measurement.
Finally, we move into the area of structured recurrent variational models from vision,
where we are motivated by the following observation: in existing models, applying a
force in the direction from a start point and an end point (in latent space), does not
result in a movement from the start point towards the end point, even on the simplest
unconstrained environments. This means that the latent space learned by these models
does not follow Newton’s law, where the acceleration vector has the same direction
as the force vector (in pointmass systems), and prevents the use of PID controllers,
which are the simplest and most well understood type of controller. We solve this problem
by building inductive biases from Newtonian physics into the latent variable model,
which we call NewtonianVAE. Crucially, Newtonian correctness in the latent space brings
about the ability to perform proportional (or PID) control, as opposed to the more computationally
expensive model predictive control (MPC). PID controllers are ubiquitous
in industrial applications, but had thus far lacked integration with unsupervised vision
models. We show that the NewtonianVAE learns physically correct latent spaces in simulated
2D and 3D control systems, which can be used to perform goalbased discovery
and control in imitation learning, and path following via Dynamic Motion Primitives.
Collections
Related items
Showing items related by title, author, creator and subject.

From learning to elearning: mining educational data. A novel, datadriven approach to evaluate individual differences in students’ interaction with learning technology
Vigentini, Lorenzo (The University of Edinburgh, 20101125)In recent years, learning technology has become a very important addition to the toolkit of instructors at any level of education and training. Not only offered as a substitute in distance education, but often complementing ... 
Tracing learning strategies in online learning environments: a learning analytics approach
Matcha, Wannisa (The University of Edinburgh, 20200728)Learning has expanded beyond formal education; yet, students continue to face the challenge of how to effectively direct their learning. Among the processes of learning, the selection and application of learning tactics ... 
Learning to play Scots fiddle: an adult learning perspective
Campbell, Katherine (The University of Edinburgh, 1999)This thesis examines how adults learn to play Scots fiddle. It focuses on a group of adults who are members of an intermediatestandard fiddle class in the city of Edinburgh, Scotland. The class is part of the wider ...