Talk for Vector Institute
Physically Structured Neural Networks
for Smooth and Contact Dynamics
This talk:
Physically Structured Neural Networks
Most of my work: Gaussian processes and decision-making systems
Physically Structured Neural Networks
$$ \htmlData{fragment-index=1,class=fragment}{ x_0 } \qquad \htmlData{fragment-index=2,class=fragment}{ x_1 = x_0 + f(x_0)\Delta t } \qquad \htmlData{fragment-index=3,class=fragment}{ x_2 = x_1 + f(x_1)\Delta t } \qquad \htmlData{fragment-index=4,class=fragment}{ .. } $$
Model using recurrent neural networks
Neural Ordinary Differential Equations
$$ \htmlData{fragment-index=0,class=fragment}{ x_{t+1} = x_t + f(x_t,\theta) } \htmlData{fragment-index=1,class=fragment}{ \mathllap{\ubr{\phantom{x_{t+1} = x_t + f(x_t,\theta)}\vphantom{\frac{\d x_t}{\d t}}}{\t{residual network}}} } \qquad \htmlData{fragment-index=2,class=fragment}{ \leadsto } \qquad \htmlData{fragment-index=3,class=fragment}{ \frac{\d x_t}{\d t} = f(x_t, \theta) } \htmlData{fragment-index=4,class=fragment}{ \mathllap{\ubr{\phantom{\frac{\d x_t}{\d t} = f(x_t, \theta)}}{\t{neural ODE}}} } $$
Continuous-time recurrent neural networks:
neural ordinary differential equations
Physically Structured Recurrent Neural Networks
$$ \htmlData{fragment-index=0,class=fragment}{ \frac{\d x_t}{\d t} = f(x_t, \theta) } \htmlData{fragment-index=1,class=fragment}{ \mathllap{\ubr{\phantom{\frac{\d x_t}{\d t} = f(x_t, \theta)}\vphantom{\frac{\pd L_\theta}{\pd q}}}{\t{black-box ODE}}} } \qquad \htmlData{fragment-index=2,class=fragment}{ \leadsto } \qquad \htmlData{fragment-index=3,class=fragment}{ \frac{\d}{\d t} \frac{\pd L_\theta}{\pd \dot{q}} - \frac{\pd L_\theta}{\pd q} = 0 } \htmlData{fragment-index=4,class=fragment}{ \mathllap{\ubr{\phantom{\frac{\d}{\d t} \frac{\pd L_\theta}{\pd \dot{q}} - \frac{\pd L_\theta}{\pd q} = 0}}{\t{Euler-Lagrange equations}}} } $$
Change the ODE to introduce inductive biases
Physically Structured Recurrent Neural Networks
$$ \footnotesize \htmlData{fragment-index=0,class=fragment}{ \begin{gathered} q_0 \\ p_0 \end{gathered} } \qquad \begin{gathered} \htmlData{fragment-index=1,class=fragment}{ q_1 = q_0 + {\textstyle\frac{\pd H}{\pd q}}(q_0,p_0) \Delta t } \\ \htmlData{fragment-index=2,class=fragment}{ p_1 = p_0 - {\textstyle\frac{\pd H}{\pd p}}(q_1,p_0) \Delta t } \end{gathered} \qquad \begin{gathered} \htmlData{fragment-index=3,class=fragment}{ q_2 = q_1 + {\textstyle\frac{\pd H}{\pd q}}(q_1,p_1) \Delta t } \\ \htmlData{fragment-index=4,class=fragment}{ p_2 = p_1 - {\textstyle\frac{\pd H}{\pd p}}(q_2,p_1) \Delta t } \end{gathered} \qquad \htmlData{fragment-index=5,class=fragment}{ .. } $$
Use inductive biases to improve performance and efficiency
Variational Integrator Networks
for Physically Structured Embeddings
Steindór Sæmundsson, Alexander Terenin, Katja Hofmann, and Marc Deisenroth
AISTATS 2020
Variational Integrator Networks
$$ \htmlData{fragment-index=0,class=fragment}{ \frac{\d}{\d t} \frac{\pd L_\theta}{\pd \dot{q}} - \frac{\pd L_\theta}{\pd q} = 0 } \htmlData{fragment-index=1,class=fragment}{ \mathllap{\ubr{\phantom{\frac{\d}{\d t} \frac{\pd L_\theta}{\pd \dot{q}} - \frac{\pd L_\theta}{\pd q} = 0}\vphantom{\int_{t_0}^{t_1}}}{\t{Euler-Lagrange equations}}} } \qquad \htmlData{fragment-index=2,class=fragment}{ \leadsto } \qquad \htmlData{fragment-index=3,class=fragment}{ S(\v{q},\v{\dot{q}}) = \int_{t_0}^{t_1} L_\theta(\v{q}_t,\v{\dot{q}}_t) \d t } \htmlData{fragment-index=4,class=fragment}{ \mathllap{\ubr{\phantom{S(\v{q},\v{\dot{q}}) = \int_{t_0}^{t_1} L_\theta(\v{q}_t,\v{\dot{q}}_t) \d t}}{\t{Principle of Least Action}}} } $$
Trajectories of motion are stationary points of action integral
Variational Integrator Networks
$$ L^d_\theta(\v{q}_t,\v{q}_{t+1}, h) \approx \int_t^{t+h} L_\theta(\v{q}(\tau),\v{\dot{q}}(\tau)) \d \tau $$
Discretize ODE by discretizing its underlying action integral
Variational Integrator Networks
Discrete dynamics lead to neural network architectures
Conservation of Physical Quantities
Network learns what quantities to conserve
Physically Structured Autoencoders
Use inductive biases to efficiently learn from pixels
Predicting the motion of a pendulum from pixels
black-box neural network
black: ground truth
physical structure
physical structure
and correct geometry
Inductive biases drive data-efficiency
Learned Latent Spaces
(a) VAE
(b) Dynamic VAE
(c) Lie Group VAE
(d) VIN-$\footnotesize SO(2)$
(e) VIN-$\footnotesize SO(2)$
(true $\footnotesize\m{M}$)
(f) Ground Truth
Physical structure yields smoothness in space and time
Learning
Contact Dynamics
using Physically Structured Neural Networks
Andreas Hochlehnert, Alexander Terenin, Steindór Sæmundsson, and Marc Deisenroth
AISTATS 2021
Contact Dynamics
(a) Find the contact time $\small t_c$
(b) Calculate true trajectory
Collisions lead to discontinuous jumps in velocity
Contact Dynamics
$$ \htmlClass{fragment}{ \delta S(\v{q},\v{\dot{q}}) = 0 } \qquad \htmlClass{fragment}{ S(\v{q},\v{\dot{q}}) = \int_{t_0}^{t_1} L_\theta(\v{q}_t,\v{\dot{q}}_t) \d t } $$
Action integral leads to two components:
(i) smooth dynamics between contact times
(ii) transfer of momentum at contact events
Contact Dynamics
Use neural ODEs to design networks that can model contacts
Contact Dynamics
Principled and accurate handling of discontinuity
Contact Dynamics
Principled and accurate handling of discontinuity
Contact Dynamics
Idealized touch feedback sensor critical for performance
Touch Feedback
Ambiguity: motion can be explained by contact,
or, alternatively, by smooth dynamics
Towards Physically Structured World Models
Lots of future extensions:
- Learning contact dynamics from vision
- Multiple objects and disentangled latent spaces
- Three-dimensional environments
- Connections with neural radiance fields
- Planning with world models
World Models: computer vision with physics
Lots of work to do!
Thank you!
https://avt.im/· @avt_im
Thank you!
https://avt.im/· @avt_im
S. Sæmundsson, A. Terenin, K. Hofmann, M. P. Deisenroth. Variational Integrator Networks for Physically Structured Embeddings. Artificial Intelligence and Statistics, 2020.
A. Hochlehnert, A. Terenin, S. Sæmundsson, M. P. Deisenroth. Learning Contact Dynamics using Physically Structured Neural Networks. Artificial Intelligence and Statistics, 2021.