The Finite–Time Turnpike Property in Machine Learning

1. Introduction

We consider a system that is governed by a neural ODE that can be considered as a continuous-time ResNet. The system $\mathbf{S}$ is defined as follows:

$$\mathbf{S}\left\{\begin{array}{c}x\left(0\right)=x_0\in {\mathbb{R}}^d,\hfill \\ x^{\prime }\left(t\right)=\sum _{i=1}^p\sigma (a_i{\left(t\right)}^{\top }x\left(t\right)+b_i\left(t\right))w_i\left(t\right)\hfill \end{array}\right.$$

(see for example [1,2]). For $i\in \{1,...,p\}$ we have $w_i\left(t\right)\in {\mathbb{R}}^d$. The $w_i\left(t\right)$ are the columns of the matrix $W\left(t\right)\in {\mathbb{R}}^{d\times p}$. We have $a_i\left(t\right)\in {\mathbb{R}}^d$ and the $a_i\left(t\right)$ are the columns of the matrix $A\left(t\right)\in {\mathbb{R}}^{d\times p}$. The bias vector $b\left(t\right)$ is in ${\mathbb{R}}^p$ and has the components $b_i\left(t\right)$.

The motivation to study $\mathbf{S}$ is that a time-discrete version can be considered as a residual neural networks (ResNets) that has been very used in many applications, see [3] for example in image registration and classification problems. A time-discrete version can be obtained for example by an explicit Euler discretization of $\mathbf{S}$.

The activation function $\sigma$ is assumed to be Lipschitz continuous with a Lipschitz constant that is less than or equal to 1 and differentiable, for example

$$\sigma \left(z\right)=tanh\left(z\right),$$

$\sigma \left(z\right)={\displaystyle \frac{1}{1+exp(-z)}}$. It acts on vectors componentwise.

For a given time horizon $T>0$, we study an optimal control problem on the time interval $[0,T]$, where the desired state at T is prescribed by the terminal condition $x\left(T\right)=x_T$, where $x_T\in {\mathbb{R}}^d$ denotes the given desired output of the system. Let $t_0\in (0,T)$ be given. For the training of the system, we study the loss function with a tracking term

$$Q(W,A,b)={\int }_{t_0}^T|x\left(t\right)-x_T|+|x^{\prime }\left(t\right)|dt.$$

with the non–smooth norm $\left|z\right|={\sum }_{i=1}^d\left|z_i\right|$.

We define the control cost (regularization term)

$$R(W,A,b)={\int }_0^T\frac{1}{2}{\parallel W\left(t\right)\parallel }^2+\frac{1}{2}{\parallel A\left(t\right)\parallel }^2+\frac{1}{2}{\parallel b\left(t\right)\parallel }^{2}dt.$$

Here ($\parallel W\left(t\right)\parallel$) denotes the Frobenius norm of $W\left(t\right)$. We introduce the space

$X\left(T\right)=\left\{\mathrm{measurable\; functions}\right(W\left(t\right),A\left(t\right),b\left(t\right)\left)\mathrm{defined\; on}\right(0,T)$

$\mathrm{such\; that}{\int }_0^T{\parallel W\left(t\right)\parallel }^2+{\parallel A\left(t\right)\parallel }^2+{\parallel b\left(t\right)\parallel }^{2}dt<\infty \}.$

Lemma 10 in [4] states that system S is exactly controllable, that is the terminal condition

$$x\left(t_0\right)=z$$

(1)

can be satisfied for all $t_0>0$. To be precise, for all $t_0>0$ there exists a constant $C_e>0$ such that for all $z\in {\mathbb{R}}^d$ we can find a control $u_{exact}$ such that for the state $\tilde{x}$ that is generated by S with the initial condition $\tilde{x}\left(0\right)=x_0$ we have $\tilde{x}\left(t_0\right)=z$ and

$$\parallel u_{exact}{\parallel }_{L^2(0,t_0)}\le C_e\parallel z-x_0\parallel .$$

(2)

Also the linearized system is exactly controllable in the sense that for all $t_0>0$ there exists a constant $C_e>0$ such that for all $z\in {\mathbb{R}}^d$ we can find a control $\tilde{v}$ such that for the state $\tilde{x}$ that is generated by the linearized system that is stated below with the initial condition $\tilde{x}\left(0\right)=0$ we have $\tilde{x}\left(t_0\right)=z$ and

$$\parallel \tilde{v}{\parallel }_{L^2(0,t_0)}\le C_e\parallel z\parallel .$$

(3)

The linearized system at a given $u=(W,A,b)$ for the variation $\delta x$ of the state that is generated by a variation $\delta u=(\delta W,\delta A,\delta b)$ of the control is

$$\delta x^{\prime }\left(t\right)=\sum _{i=1}^p\sigma (a_i{\left(t\right)}^{\top }x\left(t\right)+b_i\left(t\right))\delta w_i\left(t\right)+\sum _{i=1}^p{\sigma }^{\prime }(a_i{\left(t\right)}^{\top }x\left(t\right)+b_i\left(t\right))w_i\left(t\right)x{\left(t\right)}^{\top }\delta a_i\left(t\right)$$

$$+\sum _{i=1}^p{\sigma }^{\prime }(a_i{\left(t\right)}^{\top }x\left(t\right)+b_i\left(t\right))w_i\left(t\right)\delta b_i\left(t\right)+\sum _{i=1}^p{\sigma }^{\prime }(a_i{\left(t\right)}^{\top }x\left(t\right)+b_i\left(t\right))w_i\left(t\right)a_i{\left(t\right)}^{\top }\delta x\left(t\right)$$

with the initial condition $\delta x\left(0\right)=0$.

A universal approximation theorem for the corresponding time-discrete case with recurrent neural networks can be found in the seminal paper [5] by Cybenko, see also [6,7,8,9].

For a parameter $\gamma >0$ define

$$J(W,A,b)=\gamma Q(W,A,b)+R(W,A,b).$$

(4)

We study the minimization (training) problem

$$\mathbf{P}(T,\gamma ):\underset{(W,A,b)\in X\left(T\right)}{min}J(W,A,b)$$

Our main result is that the optimal control problem $\mathbf{P}(T,\gamma )$ has the finite-time turnpike property, that is the desired state is already reached in the interior of the time-interval $[0,T]$ and remains there until the end of the time interval. The finite-time turnpike property has been studied for example in [10,11] and [12]. In the first two references, the finite time-turnpike property is achieved by the non-smoothness of the objective functional. In this paper, we use a similar approach adapted to the framework of neural ordinary differential equations.

The finite-time turnpike property is an extremal case of the celebrated turnpike property that has originally been studied in economics. The turnpike analysis investigates how the solutions of dynamic optimal control problems with a time evolution are related to the solutions of the corresponding static problems where the time-derivatives are set to zero and the initial conditions are cancelled. It turns out that often for large time horizons on large parts of the time interval the solution of the dynamic problems is very close to the solution of the corresponding static problem. For an overview about the turnpike property, see [13,14,15,16] and the numerous references therein.

In the case of the finite-time turnpike property, after finite time the solution of the dynamic problem coincides with the solution of the static problem. The exponential turnpike property for ResNets and beyond has been studied for example in [17], but not the finite-time turnpike property.

2. The Finite-Time Turnpike Property

The following Theorem contains our main result, which states that the control cost entails the finite-time turnpike property.

Theorem 1.

For each sufficiently large $\gamma >0$ each optimal trajectory for $\mathbf{P}(T,\gamma )$ satisfies

$$x\left(t\right)=x_T,t\in [t_0,T],$$

that is $\mathbf{P}(T,\gamma )$ has the finite-time turnpike property. For $t\ge t_0$ for the optimal parameters we have $W\left(t\right)=0$, $A\left(t\right)=0$ and $b\left(t\right)=0$. The optimal parameters remain unchanged if γ is further enlarged or if T is further enlarged.

For the proof of Theorem 1 we need a result about the embedding of the continuous functions in the Sobolev space $W^{1,1}$: Let

$$L^1(0,T)=\{f:[0,T]\to \mathbb{R},f\mathrm{is\; measurable},\mathrm{i.e.}{\int }_0^T\left|f\left(t\right)\right|dt<\infty \}.$$

Consider the embedding of the space of continuous functions in the space

$$W^{1,1}(0,T)=\{f\in L^1(0,T):f^{\prime }\in L^1(0,T)\}.$$

Lemma 1.

Let $t_0\in [0,T)$. For all $x\in W^{1,1}(t_0,T)$ we have

$$\underset{t\in [t_0,T]}{max}\left|x\left(t\right)\right|\le \left({\displaystyle \frac{1}{T-t_0}}+1\right){\int }_{t_0}^T\left|x\left(t\right)\right|+\left|x^{\prime }\left(t\right)\right|dt.$$

(5)

Proof of Lemma 1.

For $t_1$, $t_2\in [t_0,T]$ we have

$$\begin{array}{ccc}\hfill |x\left(t_1\right)-x\left(t_2\right)|& =& \left|{\int }_{t_1}^{t_2}{x}^{\prime }\left(t\right)dt\right|\hfill \\ & \le & \left|{\int }_{t_1}^{t_2}\left|x^{\prime }\left(t\right)\right|dt\right|.\hfill \end{array}$$

Thus x is continuous on $[t_0,T]$. Hence there exists a point $t_{*}\in [t_0,T]$ with

$$|x\left(t_{*}\right)|=\underset{t\in [t_0,T]}{min}|x\left(t\right)|\le {\displaystyle \frac{1}{T-t_0}}{\int }_{t_0}^T\left|x\left(t\right)\right|dt.$$

Thus for all $\tau \in [t_0,T]$ the following inequality holds:

$$\begin{array}{ccc}\hfill \left|x\right(\tau \left)\right|& \le & |x\left(t_{*}\right)|+|x\left(t_{*}\right)-x\left(\tau \right)|\hfill \\ & \le & {\displaystyle \frac{1}{T-t_0}}{\int }_{t_0}^T\left|x\left(t\right)\right|dt+{\int }_{t_0}^T\left|x^{\prime }\left(t\right)\right|dt\hfill \\ & \le & \left({\displaystyle \frac{1}{T-t_0}}+1\right){\int }_{t_0}^T\left|x\left(t\right)\right|+\left|x^{\prime }\left(t\right)\right|dt.\hfill \end{array}$$

□

Now we are prepared for the proof of Theorem 1.

Proof of Theorem 1.

Case 1: If $x_0=x_T$, the parameters $u_{*}=(W_{*},A_{*},b_{*})=(0,0,0)$ generate the constant state $x\left(t\right)=x_T$. Hence $u_{*}=0$ solves $\mathbf{P}(T,\gamma )$ and the assertion follows.

Case 2: Now we assume that $x_0\ne x_T$. For $u=(W,A,b)\in X\left(T\right)$ define the cost

$$C_{(0,t_0)}\left(u\right)={\int }_0^{t_0}\frac{1}{2}{\parallel W\left(t\right)\parallel }^2+\frac{1}{2}{\parallel A\left(t\right)\parallel }^2+\frac{1}{2}{\parallel b\left(t\right)\parallel }^{2}dt.$$

Define the non-smooth tracking term

$$I_{non}\left(u\right)={\int }_{t_0}^T|x\left(t\right)-x_T|+|x^{\prime }\left(t\right)|dt.$$

Define the objective functional

$$K_T\left(u\right)=C_{(0,t_0)}\left(u\right)+\gamma I_{non}\left(u\right).$$

We consider the auxiliary problem

$$\mathbf{Q}\left(T\right):\underset{u\in X\left(T\right)}{min}{K}_T\left(u\right).$$

We show that for solution $u_{*}$ of $\mathbf{Q}\left(T\right)$ we have

$$I_{non}\left(u_{*}\right)=0$$

by an indirect proof. Suppose that there exists a solution $u_{*}=(W_{*},A_{*},b_{*})$ of $\mathbf{Q}\left(T\right)$ such that $I_{non}\left(u_{*}\right)>0$. Then for the corresponding optimal state $x_{*}$ that is generated by $\mathbf{S}$ we have $x_{*}\left(t_0\right)\ne x_T$; otherwise we could switch off the control at $t_0$ and continue with the zero control $(0,0,0)$ for $t\in (t_0,T]$ that generates the constant state $x_T$ on $(t_0,T]$ to strictly improve the performance.

Define the auxiliary state

$$\tilde{x}\left(t_0\right)=x_T+{\displaystyle \frac{1}{I_{non}\left(u_{*}\right)}}\left(x_{*}\left(t_0\right)-x_T\right).$$

The exact controllability of the linearized system implies that we can find a control $\tilde{v}\in L^2(0,t_0)$ that due to (3) satisfies the inequality

$$\parallel \tilde{v}{\parallel }_{L^2(0,t_0)}\le C_e\parallel \tilde{x}\left(t_0\right)-x_T\parallel =C_e{\displaystyle \frac{1}{I_{non}\left(u_{*}\right)}}\parallel x_{*}\left(t_0\right)-x_T\parallel$$

that generates the state $\tilde{V}$ with $\tilde{V}(0,·)=0$ and $\tilde{V}\left(t_0\right)=\tilde{x}\left(t_0\right)-x_T$.

Due to (5) we have

$$\parallel x_{*}\left(t_0\right)-x_T\parallel \le \left({\displaystyle \frac{1}{T-t_0}}+1\right){\int }_{t_0}^T\left|x_{*}\left(t\right)-x_T\right|+\left|x_{*}^{\prime }\left(t\right)\right|dt$$

(6)

$$=\left({\displaystyle \frac{1}{T-t_0}}+1\right)I_{non}\left(u_{*}\right).$$

Thus we have

$$\parallel \tilde{v}{\parallel }_{L^2(0,t_0)}\le C_e\left({\displaystyle \frac{1}{T-t_0}}+1\right).$$

For a step-size $\epsilon \in (0,I_{non}\left(u_{*}\right))$ define

$$\lambda =1-{\displaystyle \frac{\epsilon }{I_{non}\left(u_{*}\right)}}\in (0,1).$$

Consider the control u with

$$u\left(t\right)=u_{*}\left(t\right)-\epsilon \tilde{v}\left(t\right)$$

for $t\in (0,t_0]$ and for $t\in (t_0,T)$ we defined $\tilde{v}=(\delta W,\delta A,\delta b)$ with $\delta W\left(t\right)=-{\displaystyle \frac{\epsilon }{I_{non}\left(u_{*}\right)}}{u}_{*}\left(t\right)$, $\delta A\left(t\right)=-{\displaystyle \frac{\epsilon }{I_{non}\left(u_{*}\right)}}{A}_{*}\left(t\right)$, $\delta b\left(t\right)=-{\displaystyle \frac{\epsilon }{I_{non}\left(u_{*}\right)}}{b}_{*}\left(t\right)$.

Then if $\gamma >0$ is sufficiently large, $-\tilde{v}$ is a descent direction in the sense that by a little step in the direction $-\tilde{v}$ we can improve the performance of the control $u_{*}$. This can be seen as follows.

For the state $x=x_{*}+\delta x$ that is generated with the solution $\delta x$ of the linearized system with the initial condition $\delta x\left(0\right)=0$ we have at $t_0$

$$x\left(t_0\right)-x_T=(x_{*}\left(t_0\right)-x_T)-\epsilon (\tilde{x}\left(t_0\right)-x_T)=\left(1-{\displaystyle \frac{\epsilon }{I_{non}\left(u_{*}\right)}}\right)(x_{*}\left(t_0\right)-x_T)$$

$$=\lambda (x_{*}\left(t_0\right)-x_T).$$

Hence on $[t_0,T]$ the state $x=x_{*}+\delta x$ that is generated with the solution $\delta x$ of the linearized system with the initial condition $\delta x\left(t_0\right)=-{\displaystyle \frac{\epsilon }{I_{non}\left(u_{*}\right)}}\left(x_{*}\left(t_0\right)-x_T\right)$ is

$$x=x_T+\lambda (x_{*}\left(t\right)-x_T).$$

Thus for the tracking term we have the bound

$$I_{non}\left(u\right)=\lambda I_{non}\left(u_{*}\right)+{\mathcal{O}(\parallel \delta u\parallel }^2)=\left(1-{\displaystyle \frac{\epsilon }{I_{non}\left(u_{*}\right)}}\right)I_{non}\left(u_{*}\right)+{\mathcal{O}(\parallel \delta u\parallel }^2).$$

For the control cost we have

$$C_{(0,t_0)}\left(u\right)={\langle u_{*}-\epsilon \tilde{v},u_{*}-\epsilon \tilde{v}\rangle }_{L^2(0,t_0)}=C_{(0,t_0)}\left(u_{*}\right)-2\epsilon {\langle u_{*},\tilde{v}\rangle }_{L^2(0,t_0)}+{\epsilon }^2{C}_{(0,t_0)}\left(\tilde{v}\right).$$

Define

$$p\left(\epsilon \right)=K_T(u_{*}-\epsilon \tilde{v})$$

$$=C_{(0,t_0)}\left(u_{*}\right)-2\epsilon {\langle u_{*},\tilde{v}\rangle }_{L^2(0,t_0)}+{\epsilon }^2{C}_{(0,t_0)}\left(\tilde{v}\right)+\gamma \left(1-{\displaystyle \frac{\epsilon }{I_{non}\left(u_{*}\right)}}\right)I_{non}\left(u_{*}\right)+{\mathcal{O}(\parallel \delta u\parallel }^2).$$

Then we have

$$p^{\prime }\left(\epsilon \right)=-2{\langle u_{*},\tilde{v}\rangle }_{L^2(0,t_0)}+2\epsilon C_{(0,t_0)}\left(\tilde{v}\right)-\gamma +\mathcal{O}\left(\epsilon \right).$$

This yields

$$p^{\prime }\left(0\right)=-2{\langle u_{*},\tilde{v}\rangle }_{L^2(0,t_0)}-\gamma .$$

The exact controllability of S implies that there is a control $u_{exact}\in L^2(0,t_0)$ with (due to (2))

$$\parallel u_{exact}{\parallel }_{L^2(0,t_0)}\le C_e\parallel {\tilde{x}}_0-x_T\parallel$$

that generates the state $V_{exact}$ with $V_{exact}(0,·)=x_0$ and $V_{exact}(t_0,·)=x_T$. For $t>t_0$, let $u_{exact}\left(t\right)=0$. Since $u_{exact}$ is feasible for $\mathbf{Q}\left(T\right)$, this yields the inequality

$$C_{(t_0,T)}\left(u_{*}\right)\le K_T\left(u_{*}\right)\le K_T\left(u_{exact}\right)=\parallel u_{exact}{\parallel }_{L^2(0,t_0)}^2\le C_e^2{\parallel x_0-x_T\parallel }_{L^2(0,L)}^2.$$

Hence we have

$${\langle u_{*},\tilde{v}\rangle }_{L^2(0,t_0)}\le C_e\parallel x_0-x_T{\parallel }_{L^2(0,L)}{\parallel \tilde{v}\parallel }_{L^2(0,t_0)}$$

$$\le \parallel x_0-x_T{\parallel }_{L^2(0,L)}{C}_e^2\left({\displaystyle \frac{1}{T-t_0}}+1\right).$$

Thus if

$$\gamma >2\parallel x_0-x_T{\parallel }_{L^2(0,L)}{C}_e^2\left({\displaystyle \frac{1}{T-t_0}}+1\right),$$

we have $p^{\prime }\left(0\right)\le -\gamma +2{\parallel x_0-x_T\parallel }_{L^2(0,L)}{C}_e^2\left({\displaystyle \frac{1}{T-t_0}}+1\right)<0$. This implies that for $\epsilon >0$ sufficiently small we have

$$K_T(u_{*}-\epsilon \tilde{v})<K_T\left(u_{*}\right),$$

which is a contradiction to the optimality of $u^{*}$.

Hence for any optimal control of $\mathbf{Q}\left(T\right)$ we have $I_{non}\left(u_{*}\right)=0$. With inequality (6) this implies that for the optimal state we have $x_{*}\left(t_0\right)=x_T$.

Now we come back to problem

$$\mathbf{P}(T,\gamma ):\underset{u}{min}J\left(u\right)$$

with J defined in (4). Let $v_P\left(T\right)$ denote the optimal value of $\mathbf{P}(T,\gamma )$ and $v_Q\left(T\right)$ denote the optimal value of $\mathbf{Q}\left(T\right)$. Since $K_T\left(u\right)\le J\left(u\right)$, we have $v_Q\left(T\right)\le v_P\left(T\right).$

Moreover, any optimal control $u_{*}$ for $\mathbf{Q}\left(T\right)$ is feasible for $\mathbf{P}(T,\gamma )$. Since $x_{*}\left(t_0\right)=x_T$, we have $C_{(t_0,T)}\left(u_{*}\right)=0$. Hence $v_P\left(T\right)\le J\left(u_{*}\right)=K_T\left(u_{*}\right)=v_Q\left(T\right)$, and thus $v_P\left(T\right)\le v_Q\left(T\right).$ Therefore we have

$$v_P\left(T\right)=v_Q\left(T\right).$$

This implies that parameters that are optimal for $\mathbf{P}\left(T\right)$ are also optimal for $\mathbf{Q}\left(T\right)$ and the assertion follows. Thus we have proved Theorem 1. □

3. Existence of Solutions of $\mathbf{P}(T,\gamma )$ for Fixed A

For the sake of completeness of the analysis, we also state an existence result. However we can only prove the existence of a solution for the problem where the matrix A is fixed an not an optimization parameter for $\mathbf{P}(T,\gamma )$. Thus for a given matrix-valued function $A\left(t\right)$, we consider the problem

$$\mathbf{P}(T,\gamma ,A):\underset{(·,A,·)\in X\left(T\right)}{min}J(·,A,·)$$

In order to show the existence of a solution of $\mathbf{P}(T,\gamma ,A)$, we assume that there exists a number $M>0$ such that for $t\in [0,T]$ almost everywhere we have ${max}_{i\in \{1,...,p\}}\parallel \left(a_i\right)\left(t\right)\parallel \le M$. This is the case if the $a_i$ are elements of the function space $L^{\infty }(0,T)$, for example if they are step functions over $(0,T)$.

Theorem 2.

Assume that ${sup}_x\left|\sigma \left(x\right)\right|\le 1$ and the Lipschitz constant of σ is less than or equal to 1. Assume that $A\left(t\right)$ is given such that we have

$$\mathrm{ess}\underset{i\in \{1,...,p\},s\in [0,T]}{sup}\parallel \left(a_i\right)\left(s\right)\parallel <\infty .$$

Then each $T>0$ and $\gamma >0$, problem $\mathbf{P}(T,\gamma ,A)$ has a solution $W,b$ such that in $(W,A,b)\in X\left(T\right)$.

If $A\left(t\right)=0$ for $t\ge t_0$, for sufficiently large γ each solution has the finite-time turnpike property stated in Theorem 1.

The proof of Theorem 2 uses Gronwall’s Lemma (see for example [18]). For the convenience of the reader we state it here:

Lemma 2

(Gronwall’s Lemma). Let $L>0$, $U_0\ge 0$, $\epsilon \ge 0$ and an integrable function U on $[0,T]$ be given.

Assume that for $t\in [0,T]$ almost everywhere the integral inequality

$$0\le U\left(t\right)\le U_0+{\int }_0^{t}LU\left(\tau \right)+\epsilon d\tau$$

hold. Then for $t\in [0,T]$ almost everywhere the function U satisfies the inequality

$$U\left(t\right)\le U_0{\mathrm{e}}^{Lt}+\epsilon {\displaystyle \frac{{\mathrm{e}}^{Lt}-1}{L}}.$$

Now we present the proof of Theorem 2.

Proof of Theorem 2.

Consider a minimizing sequence ${\left(u_n\right)}_{n=1}^{\infty }$ with $u_n=(W_n,A,b_n)\in X\left(T\right)$ for all $n\in \{1,2,3,...\}$. Define the norm

$$\parallel u_n{\parallel }_{X\left(T\right)}=\sqrt{{\int }_0^T\parallel W_n{\left(t\right)\parallel }^2+{\parallel A\left(t\right)\parallel }^2+{\parallel b_n\left(t\right)\parallel }^{2}dt}$$

and the corresponding inner product that gives a Hilbert space structure to $X\left(T\right)$. Due to the definition of J, there exists a number $M>0$ such that for all $n\in \{1,2,...\}$ we have

$$\parallel u_n{\parallel }_{X\left(T\right)}\le M,$$

(7)

that is the sequence is bounded in $X\left(T\right)$.

Hence there exists a weakly-converging subsequence with a limit

$$u_{*}=(W_{*},A,b_{*})\in X\left(T\right).$$

Let $x_{*}$ denote the state generated by $u_{*}$. For the states $x_n$ generated by the $u_n$ as a solution of $\mathbf{S}$ due to the definition of the tracking term R we can assume by increasing M if necessary that we have

$$\underset{s\in [0,T],n\in \{1,2,3,...\}}{sup}\parallel x_n\left(s\right)\parallel \le M.$$

Due to Mazur’s Lemma (see for example [19,20]), there exists a subsequence of convex combinations that converges strongly. To be precise, there exist convex combinations

$$v_k=\sum _{m=k}^{N\left(k\right)}{\lambda }_m^{\left(k\right)}{u}_m,with{\lambda }_m^{\left(k\right)}\ge 0,k\le m\le N\left(k\right)and\sum _{m=k}^{N\left(k\right)}{\lambda }_m^{\left(k\right)}=1$$

such that

$$\underset{k\to \infty }{lim}{\parallel v_k-u_{*}\parallel }_X\left(T\right)=0.$$

This implies

$$\underset{k\to \infty }{lim}{\int }_0^T\parallel W_n\left(t\right)-W_{*}\left(t\right)\parallel +\parallel b_n\left(t\right)-b_{*}\left(t\right)\parallel dt=0.$$

Since $\sigma$ is Lipschitz continuous with a Lipschitz constant that is less than or equal to 1, this implies for $i\in \{1,...,p\}$

$$\left|\sigma \left(\sum _{m=k}^{N\left(k\right)}{\lambda }_m^{\left(k\right)}[\left(a_i\right){\left(t\right)}^{\top }{x}_m\left(t\right)+{\left(b_i\right)}_m\left(t\right))\right]-\sigma (\left(a_i\right){\left(t\right)}^{\top }{x}_{*}\left(t\right)+{\left(b_i\right)}_{*}\left(t\right))\right|$$

$$\le \left|\sum _{m=k}^{N\left(k\right)}{\lambda }_m^{\left(k\right)}[\left(a_i\right){\left(t\right)}^{\top }{x}_m\left(t\right)+{\left(b_i\right)}_m\left(t\right))\left)\right]-(\left(a_i\right){\left(t\right)}^{\top }{x}_{*}\left(t\right)+{\left(b_i\right)}_{*}\left(t\right))\right|.$$

(8)

Thus for $t\in [0,T]$ almost everywhere we have

$$∥\sum _{m=k}^{N\left(k\right)}{\lambda }_m^{\left(k\right)}{x}_m\left(t\right)-x_{*}\left(t\right)∥$$

$$\le \sum _{i=1}^p{\int }_0^t∥\sum _{m=k}^{N\left(k\right)}{\lambda }_m^{\left(k\right)}{\left(w_i\right)}_m\left(s\right)-{\left(w_i\right)}_{*}\left(s\right)∥\left|\sigma (\left(a_i\right){\left(s\right)}^{\top }{x}_{*}\left(s\right)+{\left(b_i\right)}_{*}\left(s\right))\right|$$

$$+∥\sum _{m=k}^{N\left(k\right)}{\lambda }_m^{\left(k\right)}{\left(w_i\right)}_m\left(s\right)∥\left|\sigma (\sum _{m=k}^{N\left(k\right)}{\lambda }_m^{\left(k\right)}\left(a_i\right){\left(s\right)}^{\top }{x}_m\left(s\right)+{\left(b_i\right)}_m\left(s\right)))-\sigma (\left(a_i\right){\left(s\right)}^{\top }{x}_{*}\left(s\right)+{\left(b_i\right)}_{*}\left(s\right))\right|ds.$$

Then the fact that ${sup}_x\left|\sigma \left(x\right)\right|\le 1$, the Cauchy-Schwarz inequality , (7) and (8) yield

$$∥\sum _{m=k}^{N\left(k\right)}{\lambda }_m^{\left(k\right)}{x}_m\left(t\right)-x_{*}\left(t\right)∥\le \sum _{i=1}^p{\int }_0^t∥\sum _{m=k}^{N\left(k\right)}{\lambda }_m^{\left(k\right)}{\left(w_i\right)}_m\left(s\right)-{\left(w_i\right)}_{*}\left(s\right)∥ds$$

$$+\sqrt{{\int }_0^t{∥\sum _{m=k}^{N\left(k\right)}{\lambda }_m^{\left(k\right)}{\left(w_i\right)}_m\left(s\right)∥}^{2}ds}\sqrt{{\int }_0^t{\left|\sigma (\sum _{m=k}^{N\left(k\right)}{\lambda }_m^{\left(k\right)}\left(a_i\right){\left(s\right)}^{\top }{x}_m\left(s\right)+{\left(b_i\right)}_m\left(s\right)))-\sigma (\left(a_i\right){\left(s\right)}^{\top }{x}_{*}\left(s\right)+{\left(b_i\right)}_{*}\left(s\right))\right|}^{2}ds}$$

$$\le \sum _{i=1}^p{\int }_0^t∥\sum _{m=k}^{N\left(k\right)}{\lambda }_m^{\left(k\right)}{\left(w_i\right)}_m\left(s\right)-{\left(w_i\right)}_{*}\left(s\right)∥ds$$

$$+M\sqrt{{\int }_0^t{\left|(\sum _{m=k}^{N\left(k\right)}{\lambda }_m^{\left(k\right)}\left(a_i\right){\left(s\right)}^{\top }{x}_m\left(s\right)+{\left(b_i\right)}_m\left(s\right))-(\left(a_i\right){\left(s\right)}^{\top }{x}_{*}\left(s\right)+{\left(b_i\right)}_{*}\left(s\right))\right|}^{2}ds}$$

$$\le \sum _{i=1}^p{\int }_0^t∥\sum _{m=k}^{N\left(k\right)}{\lambda }_m^{\left(k\right)}{\left(w_i\right)}_m\left(s\right)-{\left(w_i\right)}_{*}\left(s\right)∥ds$$

$$+M\sqrt{{\int }_0^t{\left|\left(a_i\right){\left(s\right)}^{\top }\left[\sum _{m=k}^{N\left(k\right)}{\lambda }_m^{\left(k\right)}{x}_m\left(s\right)-x_{*}\left(s\right)\right]\right|}^{2}ds}+M\sqrt{{\int }_0^t{\left|\sum _{m=k}^{N\left(k\right)}{\lambda }_m^{\left(k\right)}{\left(b_i\right)}_m\left(s\right)-{\left(b_i\right)}_{*}\left(s\right)\right|}^{2}ds.}$$

Due to Mazur’s Lemma, this yields the existence of a sequence ${\left({ϵ}_k\right)}_k$ with ${ϵ}_k\ge 0$ and ${lim}_{k\to \infty }{ϵ}_k=0$ such that

$$∥\sum _{m=k}^{N\left(k\right)}{\lambda }_m^{\left(k\right)}{x}_m\left(t\right)-x_{*}\left(t\right)∥\le {\epsilon }_k+\sum _{i=1}^{p}M\sqrt{{\int }_0^t\mathrm{ess}{sup}_{s\in (0,T)}{\parallel \left(a_i\right)\left(s\right)\parallel }^2{∥\sum _{m=k}^{N\left(k\right)}{\lambda }_m^{\left(k\right)}{x}_m\left(t\right)-x_{*}\left(t\right)∥}^{2}dt}.$$

Thus by increasing the value of M if necessary, we obtain for $t\in [0,T]$ almost everywhere

$$∥\sum _{m=k}^{N\left(k\right)}{\lambda }_m^{\left(k\right)}{x}_m\left(t\right)-x_{*}\left(t\right)∥\le {\epsilon }_k+\sum _{i=1}^{p}M\sqrt{\underset{0}{\overset{t}{\int }}{M}^2{∥\sum _{m=k}^{N\left(k\right)}{\lambda }_m^{\left(k\right)}{x}_m\left(s\right)-x_{*}\left(s\right)∥}^{2}ds}.$$

Since $\left(\right|u|+{\left|v\right|)}^2\le {2\left|u\right|}^2+2{\left|v\right|}^2$ and

${\left(\sum _{i=1}^p\sqrt{|z_i|}\right)}^2\le p\sum _{i=1}^p|z_i|$ this yields the integral inequality

$${∥\sum _{m=k}^{N\left(k\right)}{\lambda }_m^{\left(k\right)}{x}_m\left(t\right)-x_{*}\left(t\right)∥}^2\le 2{\left({\epsilon }_k\right)}^2+2p{M}^4\sum _{i=1}^p\underset{0}{\overset{t}{\int }}{∥\sum _{m=k}^{N\left(k\right)}{\lambda }_m^{\left(k\right)}{x}_m\left(s\right)-x_{*}\left(s\right)∥}^{2}ds.$$

Now Gronwall’s Lemma yields for $t\in [0,T]$ almost everywhere

$$∥\sum _{m=k}^{N\left(k\right)}{\lambda }_m^{\left(k\right)}{x}_m\left(t\right)-x_{*}\left(t\right)∥=\mathcal{O}\left({\epsilon }_k\right).$$

This yields

$$\underset{k\to \infty }{lim}\underset{t\in \left[0T\right]}{max}∥\sum _{m=k}^{N\left(k\right)}{\lambda }_m^{\left(k\right)}{x}_m\left(t\right)-x_{*}\left(t\right)∥=0.$$

For the time derivatives we obtain again by increasing the value of M if necessary

$${\int }_0^T∥\sum _{m=k}^{N\left(k\right)}{\lambda }_m^{\left(k\right)}{x}_m^{\prime }\left(t\right)-x_{*}^{\prime }\left(t\right)∥dt$$

$$\le \sum _{i=1}^p{\int }_0^T∥\sum _{m=k}^{N\left(k\right)}{\lambda }_m^{\left(k\right)}{\left(w_i\right)}_m\left(t\right)-{\left(w_i\right)}_{*}\left(t\right)∥\left|\sigma (\left(a_i\right){\left(t\right)}^{\top }{x}_{*}\left(t\right)+{\left(b_i\right)}_{*}\left(t\right))\right|$$

$+∥\sum _{m=k}^{N\left(k\right)}{\lambda }_m^{\left(k\right)}{\left(w_i\right)}_m\left(t\right)∥\left|\sigma (\sum _{m=k}^{N\left(k\right)}{\lambda }_m^{\left(k\right)}\left(a_i\right){\left(t\right)}^{\top }{x}_m\left(t\right)+{\left(b_i\right)}_m\left(t\right)))-\sigma (\left(a_i\right){\left(t\right)}^{\top }{x}_{*}\left(t\right)+{\left(b_i\right)}_{*}\left(t\right))\right|dt$

$$\le \sum _{i=1}^p{\int }_0^T∥\sum _{m=k}^{N\left(k\right)}{\lambda }_m^{\left(k\right)}{\left(w_i\right)}_m\left(t\right)-{\left(w_i\right)}_{*}\left(t\right)∥dt$$

$+\sqrt{{\int }_0^T{∥\sum _{m=k}^{N\left(k\right)}{\lambda }_m^{\left(k\right)}{\left(w_i\right)}_m\left(t\right)∥}^{2}dt}\sqrt{{\int }_0^T{\left|\sum _{m=k}^{N\left(k\right)}{\lambda }_m^{\left(k\right)}\left(a_i\right){\left(t\right)}^{\top }{x}_m\left(t\right)+{\left(b_i\right)}_m\left(t\right)-\left(\left(a_i\right){\left(t\right)}^{\top }{x}_{*}\left(t\right)+{\left(b_i\right)}_{*}\left(t\right)\right)\right|}^{2}dt}$

$$\le {\epsilon }_k+\sum _{i=1}^{p}M\sqrt{{\int }_0^T{\left|\sum _{m=k}^{N\left(k\right)}{\lambda }_m^{\left(k\right)}\left(a_i\right){\left(t\right)}^{\top }{x}_m\left(t\right)-\left(a_i\right){\left(t\right)}^{\top }{x}_{*}\left(t\right)\right|}^{2}dt}$$

$$+M\sqrt{{\int }_0^T{\left|\sum _{m=k}^{N\left(k\right)}{\lambda }_m^{\left(k\right)}{\left(b_i\right)}_m\left(t\right)-{\left(b_i\right)}_{*}\left(t\right)\right|}^{2}dt}$$

$$\le {\epsilon }_k(1+M)+M\sum _{i=1}^p\sqrt{{\int }_0^T{\left|{\left(a_i\right)}_m{\left(t\right)}^{\top }\left[\sum _{m=k}^{N\left(k\right)}{\lambda }_m^{\left(k\right)}{x}_m\left(t\right)-x_{*}\left(t\right)\right]\right|}^{2}dt}$$

$$\le {\epsilon }_k(1+M)+M\sum _{i=1}^p\sqrt{{\int }_0^T\parallel \left(a_i\right)\left(t\right){\parallel }^2{\parallel \sum _{m=k}^{N\left(k\right)}{\lambda }_m^{\left(k\right)}{x}_m\left(t\right)-x_{*}\parallel }^{2}dt}$$

$$\le {\epsilon }_k(1+M)$$

$$+M\sum _{i=1}^p\mathrm{ess}\underset{s\in [0,T]}{sup}\parallel \left(a_i\right)\left(s\right)\parallel \sqrt{{\int }_0^T{∥\sum _{m=k}^{N\left(k\right)}{\lambda }_m^{\left(k\right)}{x}_m\left(t\right)-x_{*}\left(t\right)∥}^{2}dt}$$

$$\le {\epsilon }_k(1+M)+M\sum _{i=1}^p\mathrm{ess}\underset{s\in [0,T]}{sup}\parallel \left(a_i\right)\left(s\right)\parallel {\epsilon }_k=\mathcal{O}\left({\epsilon }_k\right).$$

Thus we have

$$\underset{k\to \infty }{lim\; inf}Q\left(v_k\right)\ge Q\left(u_{*}\right),\underset{k\to \infty }{lim\; inf}R\left(v_k\right)\ge R\left(u_{*}\right).$$

This yields

$$\underset{k\to \infty }{lim\; inf}J\left(u_k\right)\ge J\left(u_{*}\right).$$

Hence $u_{*}$ is a solution of $\mathbf{P}(T,\gamma ,A)$. This shows that solution of $\mathbf{P}(T,\gamma ,A)$ exist.

The exact controllability properties that have been used for the construction in the proof of Theorem 1 still hold if the matrix A is fixed. Hence the assertion follows. □

4. Discussion

We have shown that with a suitable non-smooth loss function each solution of a learning problem has the finite-time turnpike property which means that it reaches the desired state exactly after finite time. Since the finite time $t_0$ can be considered as a problem parameter, this situation allows to choose $t_0$ in a convenient way. Thus $t_0$ arises as an additional design parameter in the design of optimal neural networks, that corresponds to the number of layers. Since for $t\in [t_0,T]$ the optimal parameters are zero, System $\mathbf{S}$ does not change the state on $[t_0,T]$ and thus the time horizon can be cut off at $t_0$.

Hence the problem to find the optimal number of layers in a neural network corresponds in the setting of neural ODEs to the problem of time-optimal control where the task is to find a minimal value of $t_0$ subject to the constraint that $x\left(t_0\right)=x_T$ and for the optimal parameters $u\left(t\right)$ the constraint ${\parallel u\left(t\right)\parallel }_{X\left(t_0\right)}^2\le \rho$ is satisfied. Here the number $\rho$ is prescribed as a problem parameter. Let $\omega (T,\gamma )$ denote the optimal value of $\mathbf{P}(T,\gamma )$. Then for optimal parameters $u\left(t\right)$ that solve $\mathbf{P}(T,\gamma )$ we have ${\parallel u\left(t\right)\parallel }_{X\left(t_0\right)}^2\le \omega (T,\gamma )$. Since Theorem 1 implies that for the optimal state we have $x\left(t_0\right)=x_T$, we conclude that optimal parameters for $\mathbf{P}(T,\gamma )$ also solve the time-optimal control problem with parameter $\rho =\omega (T,\gamma )$ and the optimal time is $t_0$.

We have shown the existence of a solution of the nonlinear optimization problem for the case that one of the parameters, namely the matrix $A\left(t\right)$ is fixed. In order to show that a solution also exists with A as an additional optimization parameter, we expect that an additional regularization term in the objective functional (for example ${\int }_0^T\parallel A^{\prime }\left(t\right){\parallel }^{2}dt)$ is necessary. This is a topic for future research. We expect that the finite-time turnpike property also holds in the case $t_0=0$. However, the proof that is presented here does not apply to this case so this is another topic for future research. As a possible application of our results we have in mind the numerical solution of shape inverse problems as described in [21].