Talk for Vector Institute

Physically Structured Neural Networks

for Smooth and Contact Dynamics

Alexander Terenin

https://avt.im/ · @avt_im

This talk: 

Physically Structured Neural Networks

Most of my work: Gaussian processes and decision-making systems

Physically Structured Neural Networks

$$ \htmlData{fragment-index=1,class=fragment}{ x_0 } \qquad \htmlData{fragment-index=2,class=fragment}{ x_1 = x_0 + f(x_0)\Delta t } \qquad \htmlData{fragment-index=3,class=fragment}{ x_2 = x_1 + f(x_1)\Delta t } \qquad \htmlData{fragment-index=4,class=fragment}{ .. } $$

Model using recurrent neural networks

Neural Ordinary Differential Equations

$$ \htmlData{fragment-index=0,class=fragment}{ x_{t+1} = x_t + f(x_t,\theta) } \htmlData{fragment-index=1,class=fragment}{ \mathllap{\ubr{\phantom{x_{t+1} = x_t + f(x_t,\theta)}\vphantom{\frac{\d x_t}{\d t}}}{\t{residual network}}} } \qquad \htmlData{fragment-index=2,class=fragment}{ \leadsto } \qquad \htmlData{fragment-index=3,class=fragment}{ \frac{\d x_t}{\d t} = f(x_t, \theta) } \htmlData{fragment-index=4,class=fragment}{ \mathllap{\ubr{\phantom{\frac{\d x_t}{\d t} = f(x_t, \theta)}}{\t{neural ODE}}} } $$

Continuous-time recurrent neural networks:
neural ordinary differential equations

Physically Structured Recurrent Neural Networks

$$ \htmlData{fragment-index=0,class=fragment}{ \frac{\d x_t}{\d t} = f(x_t, \theta) } \htmlData{fragment-index=1,class=fragment}{ \mathllap{\ubr{\phantom{\frac{\d x_t}{\d t} = f(x_t, \theta)}\vphantom{\frac{\pd L_\theta}{\pd q}}}{\t{black-box ODE}}} } \qquad \htmlData{fragment-index=2,class=fragment}{ \leadsto } \qquad \htmlData{fragment-index=3,class=fragment}{ \frac{\d}{\d t} \frac{\pd L_\theta}{\pd \dot{q}} - \frac{\pd L_\theta}{\pd q} = 0 } \htmlData{fragment-index=4,class=fragment}{ \mathllap{\ubr{\phantom{\frac{\d}{\d t} \frac{\pd L_\theta}{\pd \dot{q}} - \frac{\pd L_\theta}{\pd q} = 0}}{\t{Euler-Lagrange equations}}} } $$

Change the ODE to introduce inductive biases

Physically Structured Recurrent Neural Networks

$$ \footnotesize \htmlData{fragment-index=0,class=fragment}{ \begin{gathered} q_0 \\ p_0 \end{gathered} } \qquad \begin{gathered} \htmlData{fragment-index=1,class=fragment}{ q_1 = q_0 + {\textstyle\frac{\pd H}{\pd q}}(q_0,p_0) \Delta t } \\ \htmlData{fragment-index=2,class=fragment}{ p_1 = p_0 - {\textstyle\frac{\pd H}{\pd p}}(q_1,p_0) \Delta t } \end{gathered} \qquad \begin{gathered} \htmlData{fragment-index=3,class=fragment}{ q_2 = q_1 + {\textstyle\frac{\pd H}{\pd q}}(q_1,p_1) \Delta t } \\ \htmlData{fragment-index=4,class=fragment}{ p_2 = p_1 - {\textstyle\frac{\pd H}{\pd p}}(q_2,p_1) \Delta t } \end{gathered} \qquad \htmlData{fragment-index=5,class=fragment}{ .. } $$

Use inductive biases to improve performance and efficiency

Variational Integrator Networks

for Physically Structured Embeddings

Steindór Sæmundsson, Alexander Terenin, Katja Hofmann, and Marc Deisenroth

AISTATS 2020

Variational Integrator Networks

$$ \htmlData{fragment-index=0,class=fragment}{ \frac{\d}{\d t} \frac{\pd L_\theta}{\pd \dot{q}} - \frac{\pd L_\theta}{\pd q} = 0 } \htmlData{fragment-index=1,class=fragment}{ \mathllap{\ubr{\phantom{\frac{\d}{\d t} \frac{\pd L_\theta}{\pd \dot{q}} - \frac{\pd L_\theta}{\pd q} = 0}\vphantom{\int_{t_0}^{t_1}}}{\t{Euler-Lagrange equations}}} } \qquad \htmlData{fragment-index=2,class=fragment}{ \leadsto } \qquad \htmlData{fragment-index=3,class=fragment}{ S(\v{q},\v{\dot{q}}) = \int_{t_0}^{t_1} L_\theta(\v{q}_t,\v{\dot{q}}_t) \d t } \htmlData{fragment-index=4,class=fragment}{ \mathllap{\ubr{\phantom{S(\v{q},\v{\dot{q}}) = \int_{t_0}^{t_1} L_\theta(\v{q}_t,\v{\dot{q}}_t) \d t}}{\t{Principle of Least Action}}} } $$

Trajectories of motion are stationary points of action integral

Variational Integrator Networks

$$ L^d_\theta(\v{q}_t,\v{q}_{t+1}, h) \approx \int_t^{t+h} L_\theta(\v{q}(\tau),\v{\dot{q}}(\tau)) \d \tau $$

Discretize ODE by discretizing its underlying action integral

Variational Integrator Networks

Discrete dynamics lead to neural network architectures

Conservation of Physical Quantities

Network learns what quantities to conserve

Physically Structured Autoencoders

Use inductive biases to efficiently learn from pixels

Predicting the motion of a pendulum from pixels

black-box neural network
black: ground truth

physical structure

physical structure
and correct geometry

Inductive biases drive data-efficiency

Learned Latent Spaces

(a) VAE

(b) Dynamic VAE

(c) Lie Group VAE

(d) VIN-$\footnotesize SO(2)$

(e) VIN-$\footnotesize SO(2)$
(true $\footnotesize\m{M}$)

(f) Ground Truth

Physical structure yields smoothness in space and time

Learning 

Contact Dynamics

using Physically Structured Neural Networks

Andreas Hochlehnert, Alexander Terenin, Steindór Sæmundsson, and Marc Deisenroth

AISTATS 2021

Contact Dynamics

(a) Find the contact time $\small t_c$

(b) Calculate true trajectory

Collisions lead to discontinuous jumps in velocity

Contact Dynamics

$$ \htmlClass{fragment}{ \delta S(\v{q},\v{\dot{q}}) = 0 } \qquad \htmlClass{fragment}{ S(\v{q},\v{\dot{q}}) = \int_{t_0}^{t_1} L_\theta(\v{q}_t,\v{\dot{q}}_t) \d t } $$

Action integral leads to two components:

(i) smooth dynamics between contact times

(ii) transfer of momentum at contact events

Contact Dynamics

Use neural ODEs to design networks that can model contacts

Contact Dynamics

Principled and accurate handling of discontinuity

Contact Dynamics

Principled and accurate handling of discontinuity

Contact Dynamics

Idealized touch feedback sensor critical for performance

Touch Feedback

Ambiguity: motion can be explained by contact,
or, alternatively, by smooth dynamics

Towards Physically Structured World Models

Lots of future extensions:

  • Learning contact dynamics from vision
  • Multiple objects and disentangled latent spaces
  • Three-dimensional environments
  • Connections with neural radiance fields
  • Planning with world models

World Models: computer vision with physics

Lots of work to do!

Thank you!

https://avt.im/· @avt_im

S. Sæmundsson, A. Terenin, K. Hofmann, M. P. Deisenroth. Variational Integrator Networks for Physically Structured Embeddings. Artificial Intelligence and Statistics, 2020.

A. Hochlehnert, A. Terenin, S. Sæmundsson, M. P. Deisenroth. Learning Contact Dynamics using Physically Structured Neural Networks. Artificial Intelligence and Statistics, 2021.