On Constructive Approximation of Nonlinear Operators

1. Introduction

A purpose of the proposed methodology is to provide an effective way to transform large data sets. The methodology is motivated by the problems arising in signal processing where the nonlinear operator $ℱ$: $K_Y\to K_X$ is interpreted as a nonlinear system (or a nonlinear filter) transforming a set of stochastic signals $K_Y$ to the set of stochastic signals $K_X$. Therefore, below we refer to this terminology.

The device behind the proposed method is based on a special extension of the piecewise linear interpolation technique to the case of stochastic signal sets. The device is not straightforward and requires the careful substantiation presented in Section 2.3, Section 3.4, Section 4.2 and Section 4.4 below.

1.1. Motivations

The problem under consideration is motivated by the following observations.

1.1.1. Transformation of Large Sets of Signals

Suppose we need to transform a set of signals $K_Y$ to another set of signals $K_X$. The signals are represented by finite stochastic vectors1. A major associated difficulty and inconvenience which is common to many known filtering methodologies (see, for example, [1]–9,11,13,22,23,25]) is that they require a prior information on each reference signal to be estimated2. In particular, the filters in [22,23,25] are based on the use of either the reference signal $\mathbf{x}\in K_X$ itself, as in [22,23], or its estimate, as in [25]. The Wiener filtering approach (see, for example, [1]–13,23,25]) assumes that covariance matrices formed from a reference signal, $\mathbf{x}\in K_X$, and an observed signal, $\mathbf{y}\in K_Y$, are known or can be estimated. The latter can be done, for instance, from samples of $\mathbf{x}$ and $\mathbf{y}$. In particular, this means that the reference signal $\mathbf{x}$ can be measured.

In the case of processing large signal sets, such restrictions become much more inconvenient.

The major motivating question for this work is as follows. Let $ℱ$: $K_Y\to K_X$ denote a filter that estimates a large set of reference signals, $K_X$, from a large set of observed signals, $K_Y$. Each set contains N signals. Is it possible to construct a filter $ℱ$ that requires a prior information only on few signals, p ≪ N, from K_X but performs better than the known filters based on a prior information on every reference signal from K_X? We denote such a filter by $ℱ$^(p−1).

It is shown in Section 2.3 and Section 4.4 that the positive answer is achievable under quite unrestrictive assumptions. The required features of filter $ℱ$$(p-1)$ are satisfied by its special structure described in Section 2.3, Section 3.1 and Section 3.4. The related conditions are also considered in those Sections.

1.1.2. Filtering Based on Idea of Piecewise Function Interpolation

The specific structure of the proposed filter follows from the extension of piecewise function interpolation [14]. This is because the technique of piecewise function interpolation [14] has significant advantages over the methods of linear and polynomial approximation used in known filtering techniques (such as, for example, those in [5,9]).

The structure of the proposed filter is presented in Section 2.3, Section 3.1 and Section 4.2 below.

1.1.3. Exploiting Pseudo-Inverse Matrices in the Filter Model

Most of the known filtering techniques, for example, those ones in [1]–3,6]–8,11,23,25], are based on exploiting inverse matrices in their mathematical models. In the cases of grossly corrupted signals or erroneous measurements those inverse matrices may not exist and, thus, those filters cannot be applied. The examples in Section 5 illustrate this case.

The filter proposed here avoids this drawback since its model is based on exploiting pseudo-inverse matrices. As a result, the proposed filter always exist. That is, it processes any kind of noisy signals. An extension of the filtering techniques to the case of implementation of the pseudo-inverse matrices is done on the basis of theory presented in [5].

1.1.4. Computational Work

Let m and n be the number of components of $\mathbf{x}\in K_X$ and of $\mathbf{y}\in K_Y$, respectively, where $K_X$ and $K_Y$ each contains N signals. The known filtering techniques (e.g. see [1]–8,11,23,25]), applied to $\mathbf{x}$ and $\mathbf{y}$, require computation of a product of an $m\times n$ matrix and an $n\times n$ matrix, as well as computation of an $n\times n$ inverse or pseudo-inverse matrix for each pair of signals $\mathbf{x}\in K_X$ and $\mathbf{y}\in K_Y$. This requires $O\left(2m{n}^2\right)$ and $O\left(26n^3\right)$ flops, respectively [26]. Thus, for the processing of all signals in $K_X$ and $K_Y$, the filters in [1]–8,11,23,25] require $O\left(2m{n}^{2}N\right)+O\left(26n^{3}N\right)$ operations.

Alternatively, $K_X$ and $K_Y$ can be represented by vectors, $\mathbf{\chi }$ and $\mathbf{\gamma }$, each with $mN$ and $nN$ components, respectively. In such a case, the techniques in [1]–8,11,23,25] can be applied to $\mathbf{\chi }$ and $\mathbf{\gamma }$ as opposed to each signals in $K_X$ and $K_Y$. computational requirement is then $O\left(2m{n}^2{N}^2\right)$ and $O\left(26n^3{N}^3\right)$ operations, respectively [26].

In both cases, but especially when N is large, computational work associated with the approaches [1]–8,11,23,25] becomes unreasonable hard.

For the filter ${ℱ}^{(p-1)}$ to be introduced below, the associated computational work is substantially less. This is because ${ℱ}^{(p-1)}$ requires computation of only p pseudo-inverse matrices associated with p selected signals in $K_X$, where p is much less than the number of signals in $K_X$. Therefore, for processing of the signal sets, $K_X$ and $K_Y$, ${ℱ}^{(p-1)}$ requires only $O\left(2m{n}^{2}p\right)+O\left(26n^{3}p\right)$ flops where $p\ll N$. This comparison is illustrated in Section 5.

1.2. Relevant works

Some particular filtering techniques relevant to the method proposed below are as follows.

1.2.1. Generic Optimal Linear (GOL) Filter [5]

The generic optimal linear (GOL) filter in [5] is a generalization of the Wiener filter to the case when covariance matrix is not invertible and observable signal is arbitrarily noisy (i.e. when, in particular, noise is not necessarily additive and Gaussian). The GOL filter has been developed for processing an individual stochastic signal. Some ideas from [5] are used in the proof of Theorem 1 below.

1.2.2. Simplicial Canonical Piecewise Linear Filter [23]

A complex Wiener adaptive filter was developed in [23] from the two-dimensional complex-valued simplicial canonical piecewise linear filter [24]. The filter in [23] was developed for the processing of an individual stochastic signal and can be exploited when the reference signal is known and a ‘covariance-like’ matrix is invertible. The latter precludes an application to the signal types considered, for example, in Section 5: the matrices used in [23] are not invertible for the signals as those in Section 5. Similarly, the filters studied in [8,11] were developed for the processing of a single signal when the covariance matrices are invertible.

For the filter proposed here, these restrictions are removed.

1.2.3. Adaptive Piecewise Linear Filter [22]

A piecewise linear filter in [22] was proposed for a fixed image denoising (given by a matrix), corrupted by an additive Gaussian noise. That is, the method involved a non stochastic reference signal and required its knowledge. No theoretical justification for the filter was given in [22].

1.2.4. Averaging Polynomial Filter [10,12]

The averaging polynomial filter proposed in [10,12] was developed for the purpose of processing infinite signal sets. The filter was based on an argument involving the ‘averaging’ over sets of signals under consideration. This device allows one to determine a single filter for the processing of infinite signal sets. At the same time, it leads to an increase in the associated error when signals differ considerably from each other. This effect is illustrated in Section 5 below.

1.2.5. Other Relevant Filters

The technique developed in [13] is an extension of the GOL filter to the constraint problem with respect to the filter rank. It concerns data compression.

The methods in [6,7,15,16] have been developed for deterministic signals. Motivated by the results achieved in [15,16], adaptive filters were elaborated in [17]. A theoretical basis for the device proposed in [15,16] is provided in [18].

We note that the idea of piecewise linear filtering has been used in the literature in several very different conceptual frameworks, despite exploiting some very similar terms (as in [15]–24]). At the same time, a common feature of those techniques is that they were developed for the processing of a single signal, not of large signal sets as in this paper. In particular, piecewise linear filters in [19] have been obtained by arranging linear filters and thresholds in a tree structure. Piecewise linear filters discussed in [20] were developed using so-called threshold decomposition, which is a segmentation operator exploited to split a signal into a set of multilevel components. Filter design methods for piecewise linear systems proposed in [21] were based on a piecewise Lyapunov function.

1.3. Difficulties Associated with the Known Filtering Techniques

Basic difficulties associated with applying the known filtering techniques to the case under consideration (i.e. to processing of large signal sets, $K_X$ and $K_Y$) are that:

(i) they require an information on each reference signal (in the form of a sample, for example),

(ii) matrices used in the known filters can be not invertible (as in the simulations considered below in Section 5) and then the filter does not exist, and

(ii) the associated computation work may require a very long time. For example, in simulations (Section 5), MATLAB was out of memory for computing the GOL filter [5] when each of sets $K_X$ and $K_Y$ was represented by a long vector (this option has been discussed in Section 1.1.4 above).

1.4. Differences from the Known Filtering Techniques

The differences from the known filtering techniques discussed above are as follows.

(i) We consider a single filter that processes arbitrarily large input-output sets of stochastic signal-vectors. The known filters [1]–9,11,13,15]–25] have been developed for the processing of an individual signal-vector only. In the case of their application to arbitrarily large signal sets, they imply difficulties described in Section 1.1 and Section 1.3 above.

(ii) As a result, our piecewise linear filter model (Section 3), the statement of the problem (Section 3.3 below) and consequently, the device of its solution (Section 4 below) are different from those considered in [15]–24]. In this regard, see also Section 1.2.5.

(iii) The above naturally leads to a new structure of the filter (presented in Section 3.4 and Section 4.2 below) which is very different from the known ones.

1.5. Contribution

In general, for the processing of large data sets, the proposed filter allows us to achieve better results in comparison with the known techniques in [1]–25]. In particular, it allows us to

(i) achieve a desired accuracy in signal estimation3,

(ii) exploit a prior information only on few reference signals, p, from the set $K_X$ that contains $N\gg p$ signals or even infinite number of signals,

(iii) find a single filter to process any signal from the arbitrarily large signal set,

(vi) determine the filter in terms of pseudo-inverse matrices so that the filter always exists, and

(v) decrease the computational load compared to the related known techniques.

2. Some Preliminaries

2.1. Notation

The signal sets we consider are, in fact, special representations of time series.

Let $(\Omega ,\Sigma ,\mu )$ be a probability space4, and $K_X$ and $K_Y$ be arbitrarily large sets of signals such that

$$K_X=\{\mathbf{x}(t,·)\in L^2(\Omega ,{\mathbb{R}}^m)|t\in T\}and{K}_Y=\{\mathbf{y}(t,·)\in L^2(\Omega ,{\mathbb{R}}^n)|t\in T\}$$

where $T:=[a,b]\subseteq \mathbb{R}.$ We interpret $\mathbf{x}(t,·)$ as a reference signal and $\mathbf{y}(t,·)$ as an observable signal, an input to the filter $ℱ$ studied below5. The variable $t\in T\subseteq \mathbb{R}$ represents time6. Then, for example, the stochastic signal x(t, ·) can be interpreted as an arbitrary stationary time series.

Let ${\left\{t_k\right\}}_1^p\subset T$ be a sequence of fixed time-points such that

$$a=t_1<\dots <t_p=b.$$

(1)

Because of the partition (1), the sets $K_Y$ and $K_X$ are divided in `smaller’ subsets $K_{X,1},\dots ,K_{X,p-1}$ and $K_{Y,1},\dots ,K_{Y,p-1}$, respectively, so that, for each $j=1,\dots ,p$,

$$K_{X,j}=\left\{\mathbf{x}(t,·)\right|t_j\le t\le t_{j+1}\}and{K}_{Y,j}=\left\{\mathbf{y}(t,·)\right|t_j\le t\le t_{j+1}\}.$$

(2)

Therefore, $K_Y$ and $K_X$ can now be represented as

$$K_X=\bigcup _{j=1}^{p-1}{K}_{X,j}and{K}_Y=\bigcup _{j=1}^{p-1}{K}_{Y,j}.$$

(3)

2.2. Brief Description of the Problem

Given two arbitrarily large sets of stochastic signals, $K_Y$ and $K_X$, find a single filter $ℱ$:$K_Y\to K_X$ that estimates the signal $\mathbf{x}\in K_X$ with a controlled, associated error. Note that in our formulation the set $K_Y$ can be finite or infinite.

2.3. Brief description of the method

The solution of the above problem is based on the representation of the proposed filter in the form of a sum with $p-1$ terms ${ℱ}_1,\dots ,{ℱ}_{p-1}$ where each term, $ℱ$_j, is interpreted as a particular sub-filter (see (4) and (5) below). Such a filter is denoted by ${ℱ}^{(p-1)}:K_Y\to K_X$.

The sub-filter $ℱ$_j transforms signals that belong to ‘piece’ $K_{Y,j}$ of set $K_Y$ to signals in ‘piece’ $K_{X,j}$ of $K_X$, i.e. ${ℱ}_j:K_{Y,j}\to K_{X,j}$. Each sub-filter $ℱ$_j depends on two parameters, ${\alpha }_j$ and ${\mathcal{B}}_j$.

The prime idea is to determine $ℱ$_j (i.e. ${\alpha }_j$ and ${\mathcal{B}}_j$) separately, for each $j=1,\dots ,p-1$. The required ${\alpha }_j$ and ${\mathcal{B}}_j$ follow from the solutions of the equation (11) and an associated minimization problem (11) (see Section 3.4 and Section 4.2 below). This procedure adjusts $ℱ$_j so that the error associated with the estimation of $\mathbf{x}(t,·)\in K_{X,j}$ is minimal.

A motivation for such a structure of the filter ${ℱ}^{(p-1)}$ is as follows. The method of determining ${\alpha }_j$ and ${\mathcal{B}}_j$ provides an estimate ${ℱ}_j\left[\mathbf{y}\right(t,·\left)\right]$ that interpolates $\mathbf{x}(t,·)\in K_{X,j}$ at $t=t_j$ and $t=t_{j+1}$. In other words, the filter is flexible to variations in the sets of observed and reference signals $K_Y$ and $K_X$, respectively. Due to this way of determining $ℱ$j, it is natural to expect that the processing of a ‘smaller’ signal set, $K_{Y,j}$, may lead to a smaller associated error than that for the processing of the whole set $K_Y$ by a filter which is not specifically adjusted to each particular piece $K_{Y,j}$.

As a result, ${ℱ}^{(p-1)}\left[\mathbf{y}\right(t,·\left)\right]$ represents a special piecewise interpolation procedure and, thus, should be attributed with the associated advantages such as, for example, the high accuracy of estimation.

In Section 4.4, this observation is confirmed. In Section 2 and Section 5, it is also shown that the proposed technique allows us to avoid the difficulties discussed in Section 1.3 above.

3. Description of the Problem

3.1. Piecewise Linear Filter Model

Let ${ℱ}^{(p-1)}:K_Y\to K_X$ be a filter such that, for each $t\in T$,

$${ℱ}^{(p-1)}\left[\mathbf{y}(t,·)\right]=\sum _{j=1}^{p-1}{\delta }_j{ℱ}_j\left[\mathbf{y}(t,·)\right],$$

(4)

where

$${ℱ}_j\left[\mathbf{y}(t,·)\right]={\alpha }_j+{\mathcal{B}}_j\left[\mathbf{y}(t,·)\right]and{\delta }_j=\left\{\begin{array}{cc}1,& if{t}_j\le t\le t_{j+1},\hfill \\ 0,& otherwise.\hfill \end{array}\right.$$

(5)

Here, $ℱ$_j is a sub-filter defined for $t_j\le t\le t_{j+1}$. In (5), ${\alpha }_j={[{\alpha }_j^{\left(1\right)},\dots ,{\alpha }_j^{\left(m\right)}]}^T\in {\mathbb{R}}^m$ and ${\mathcal{B}}_j:L^2(\Omega ,{\mathbb{R}}^n)\to L^2(\Omega ,{\mathbb{R}}^m)$ is a linear operator given by a matrix $B_j\in {\mathbb{R}}^{m\times n}$, so that

$$\left[{\mathcal{B}}_j\left(\mathbf{y}\right)\right](t,\omega )=B_j\left[\mathbf{y}(t,\omega )\right].$$

Thus, $ℱ$_j is defined by an operator $F_j:{\mathbb{R}}^n\to {\mathbb{R}}^m$ such that

$$F_j\left[\mathbf{y}(t,\omega )\right]={\alpha }_j+B_j\left[\mathbf{y}(t,\omega )\right].$$

(6)

Filter ${ℱ}^{(p-1)}$ defined by (4)-(6) is called the piecewise filter7.

3.2. Assumptions

In the known approaches related to filtering of stochastic signals (e.g. see [1]–13,23,25]), it is assumed that covariance matrices formed from the reference signal and observed signal are known or can be estimated.

The assumption used here is similar. The covariance matrices that are assumed to be known or can be estimated, are formed from selected signal pairs $\{\mathbf{x}(t_j,·),\mathbf{y}(t_j,·)\}$ with $j=1,\dots ,p$ and p to be a small number8, $p\ll N$, where N is the number of signals in $K_X$ or $K_Y$.

3.3. The Problem

In (4)-(6), parameters of the filter ${ℱ}^{(p-1)}$, i.e. vector ${\alpha }_j$ and matrix $B_j$, for $j=1,\dots ,p-1$, are unknown. Therefore, under the assumptions described in Section 3.2, the problem is to determine ${\alpha }_j$ and $B_j$, for $j=1,\dots ,p-1$. The related problem is to estimate an error associated with the filter ${ℱ}^{(p-1)}$.

Solutions to the both problems are given in Section 4.2 and Section 4.4, respectively. In particular, in the following Section 3.4, interpolation conditions (8) and (11) are introduced that lead to a determination of ${\alpha }_j$ and $B_j$.

3.4. Interpolation Conditions

Let us denote

$$\parallel \mathbf{x}(t_j,·){\parallel }_{\Omega }^2={\int }_{\Omega }{\parallel \mathbf{x}(t_j,\omega )\parallel }_2^{2}d\mu \left(\omega \right)$$

(7)

where $\parallel \mathbf{x}(t_j,\omega ){\parallel }_2$ is the Euclidean norm of $\mathbf{x}(t_j,\omega )\in {\mathbb{R}}^m$.

For $t=t_1$, let $\widehat{\mathbf{x}}(t_1,·)$ be an estimate of $\mathbf{x}(t_1,·)$ determined by known methods [1]–13,23,25]. This is the initial condition of the proposed technique.

For $j=1,\dots ,p-1$, each sub-filter $F_j$ in (5)-(6) is defined so that ${\alpha }_j$ and ${\mathcal{B}}_j$ satisfy the conditions as follows.

Sub-filter $ℱ$₁: For $j=1$, ${\alpha }_1$ and ${\mathcal{B}}_1$ solve

$$\begin{array}{c}\hfill \widehat{\mathbf{x}}(t_1,·)={\alpha }_1+{\mathcal{B}}_1\left[\mathbf{y}(t_1,·)\right]\mathrm{and}\underset{{\mathcal{B}}_1}{min}{∥[\mathbf{x}(t_2,·)-{\alpha }_1]-{\mathcal{B}}_1\left[\mathbf{y}(t_2,·)\right]∥}_{\Omega }^2,\end{array}$$

(8)

respectively. Then an estimate of $\mathbf{x}(t,·)$, $\widehat{\mathbf{x}}(t,·)$, for $t\in [t_1,t_2]$, is determined as

$$\begin{array}{c}\hfill \widehat{\mathbf{x}}(t,·)={ℱ}_1\left[\mathbf{y}(t,·)\right]={\alpha }_1+{\mathcal{B}}_1\left[\mathbf{y}(t,·)\right]=\widehat{\mathbf{x}}(t_1,·)+{\mathcal{B}}_1[\mathbf{y}(t,·)-\mathbf{y}(t_1,·)]\end{array}$$

(9)

where ${\alpha }_1$ and ${\mathcal{B}}_1$ satisfy (8). In particular, ${\alpha }_1=\widehat{\mathbf{x}}(t_1,·)-{\mathcal{B}}_1\left[\mathbf{y}(t_1,·)\right]$ and

$$\widehat{\mathbf{x}}(t_2,·)={ℱ}_1\left[\mathbf{y}(t_2,·)\right].$$

Extending this procedure up to $j=k-1$, where $k=3,\dots ,p$, we set the following. Let $\widehat{\mathbf{x}}(t_{k-1},·)$ be an estimate of $\mathbf{x}(t_{k-1},·)$ defined by the preceding steps as

$$\begin{array}{c}\hfill \widehat{\mathbf{x}}(t_{k-1},·)={ℱ}_{k-2}\left[\mathbf{y}(t_{k-1},·)\right].\end{array}$$

(10)

Then sub-filter ${ℱ}_{k-1}$ is defined as follows.

Sub-filter ${ℱ}_{k-1}$: For $j=k-1$, ${\alpha }_{k-1}$ and ${\mathcal{B}}_{k-1}$ solve

$$\begin{array}{c}\hfill \widehat{\mathbf{x}}(t_{k-1},·)={\alpha }_{k-1}+{\mathcal{B}}_{k-1}\left[\mathbf{y}(t_{k-1},·)\right]\mathrm{and}\underset{{\mathcal{B}}_{k-1}}{min}{∥[\mathbf{x}(t_k,·)-{\alpha }_{k-1}]-{\mathcal{B}}_{k-1}\left[\mathbf{y}(t_k,·)\right]∥}_{\Omega }^2,\end{array}$$

(11)

respectively. Then an estimate of $\mathbf{x}(t,·)$, $\widehat{\mathbf{x}}(t,·)$, for $t\in [t_{k-1},t_k]$, is determined as

$$\begin{array}{c}\hfill \widehat{\mathbf{x}}(t,·)={ℱ}_{k-1}\left[\mathbf{y}(t,·)\right]={\alpha }_{k-1}+{\mathcal{B}}_{k-1}\left[\mathbf{y}(t,·)\right]=\widehat{\mathbf{x}}(t_{k-1},·)+{\mathcal{B}}_1[\mathbf{y}(t,·)-\mathbf{y}(t_{k-1},·)].\end{array}$$

(12)

The conditions (8) and (11) are motivated by the device of piecewise function interpolation and associated advantages [14].

Filter $(p-1)$ of the form (4)-(5) with ${\alpha }_j$ and ${\mathcal{B}}_j$ satisfying (8) and (11) is called the piecewise linear interpolation filter. The pair of signals $\{\mathbf{x}(t_k,·),\mathbf{y}(t_k,·)\}$ associated with time $t_k$ defined by (1) is called the interpolation pair.

4. Main Results

4.1. General Device

In accordance withe the scheme presented in Section 3.1 and Section 3.4 above, an estimate of the reference signal $\mathbf{x}(t,·)$, for any $t\in T=[a,b]$, by the piecewise linear interpolation filter ${ℱ}^{(p-1)}$, is given by

$$\begin{array}{c}\hfill \widehat{\mathbf{x}}(t,·)={ℱ}^{(p-1)}\left[\mathbf{y}(t,·)\right]=\sum _{j=1}^{p-1}{\delta }_j{ℱ}_j\left[\mathbf{y}(t,·)\right],\end{array}$$

(13)

where, for each $j=1,\dots ,p-1,$ the sub-filter $ℱ$_j is given by (5), and is defined from the interpolation conditions (8) and (11).

Below, we show how to determine $ℱ$_j to satisfy the conditions (8) and (11).

4.2. Determination of Piecewise Linear Interpolation Filter

Let us denote

$$\begin{array}{ccc}\hfill & & \mathbf{z}(t_j,t_{j+1},·)=\mathbf{x}(t_{j+1},·)-\widehat{\mathbf{x}}(t_j,·)and\mathbf{w}(t_j,t_{j+1},·)=\mathbf{y}(t_{j+1},·)-\mathbf{y}(t_j,·).\hfill \end{array}$$

(14)

We need to represent $\mathbf{z}(t_j,t_{j+1},·)$ and $\mathbf{w}(t_j,t_{j+1},·)$ in terms of their components as follows:

$$\begin{array}{ccc}& & \mathbf{z}(t_j,t_{j+1},·)={[{\mathbf{z}}^{\left(1\right)}(t_j,t_{j+1},·),\dots ,{\mathbf{z}}^{\left(m\right)}(t_j,t_{j+1},·)]}^T\hfill \\ \hfill \mathrm{and}& & \mathbf{w}(t_j,t_{j+1},·)={[{\mathbf{w}}^{\left(1\right)}(t_j,t_{j+1},·),\dots ,{\mathbf{w}}^{\left(n\right)}(t_j,t_{j+1},·)]}^T,\hfill \end{array}$$

where ${\mathbf{z}}^{\left(j\right)}(t_j,t_{j+1},·)\in L^2(\Omega ,\mathbb{R})$ and ${\mathbf{w}}^{\left(i\right)}(t_j,t_{j+1},·)\in L^2(\Omega ,\mathbb{R})$ are stochastic variables, for all $j=1,\dots ,m$.

Then we can introduce the covariance matrix

$$\begin{array}{c}\hfill E_{z_j{w}_j}={\left\{〈{\mathbf{z}}^{\left(i\right)}(t_j,t_{j+1},·),{\mathbf{w}}^{\left(k\right)}(t_j,t_{j+1},·)〉\right\}}_{i,k=1}^{m,n},\end{array}$$

(15)

where $〈{\mathbf{z}}^{\left(i\right)}(t_j,t_{j+1},·),{\mathbf{w}}^{\left(k\right)}(t_j,t_{j+1},·)〉={\int }_{\Omega }{\mathbf{z}}^{\left(i\right)}(t_j,t_{j+1},\omega ){\mathbf{w}}^{\left(k\right)}(t_j,t_{j+1},\omega )d\mu \left(\omega \right).$

Below, $M^{†}$ is the Moor-Penrose generalized inverse of a matrix M.

Now, we are in a position to establish the main results.

Theorem 1. 

Let

$$K_X=\{\mathbf{x}(t,·)\in L^2(\Omega ,{\mathbb{R}}^m)|t\in T=[a,b]\}and{K}_Y=\{\mathbf{y}(t,·)\in L^2(\Omega ,{\mathbb{R}}^n)|t\in T=[a,b]\}$$

be sets of reference signals and observed signals, respectively. Let $t_j\in [a,b]$, for $j=1,\dots ,p$, be such that

$$a=t_1<\dots <t_p=b.$$

For $t=t_1,$ let $\widehat{\mathbf{x}}(t_1,·)$ be a known estimate of $\mathbf{x}(t_1,·)$9. Then, for any $t\in [a,b]$, the proposed piecewise linear interpolation filter ${ℱ}^{(p-1)}:L^2(\Omega ,{\mathbb{R}}^n)\to L^2(\Omega ,{\mathbb{R}}^m)$ transforming any signal $\mathbf{y}(t,·)\in L^2(\Omega ,{\mathbb{R}}^m)$ to an estimate of $\mathbf{x}(t,·)$, $\widehat{\mathbf{x}}(t,·)$, is given by

$$\widehat{\mathbf{x}}(t,·)={ℱ}^{(p-1)}\left[\mathbf{y}(t,·)\right]=\sum _{j=1}^{p-1}{\delta }_j{F}_j\left[\mathbf{y}(t,·)\right]$$

(16)

where

$${ℱ}_j\left[\mathbf{y}(t,·)\right]=\widehat{\mathbf{x}}(t_j,·)+{\mathcal{B}}_j[\mathbf{y}(t,·)-\mathbf{y}(t_j,·)],$$

(17)

$$\hat{\mathbf{x}}(t_j,·)={ℱ}_{j-1}\left[\mathbf{y}(t_j,·)\right](for j=2,\dots ,p-1),$$

(18)

$$B_j=E_{z_j{w}_j}{E}_{w_j{w}_j}^{†}+M_{B_j}[I_n-E_{w_j{w}_j}{E}_{w_j{w}_j}^{†}],$$

(19)

and where I_n is n × n identity matrix and M_Bj is an m × n arbitrary matrix.

Proof: The proof of Theorem 1 is given in Section A.

It is worthwhile to observe that, due to an arbitrary matrix $M_{B_j}$ in (19), the filter ${ℱ}^{(p-1)}$ is not unique. In particular, $M_{B_j}$ can be chosen as the zero matrix $\mathbb{O}$ similarly to the generic optimal linear [5] (which is also not unique by the same reason).

4.3. Numerical Realization of Filter ${ℱ}^{(p-1)}$ and Associated Algorithm

4.3.1. Numerical Realization

In practice, the set $T=[a,b]$ (see Section 2.1) is represented by a finite set ${\left\{{\tau }_k\right\}}_{k=1}^N$, i.e. $[a,b]=[{\tau }_1,{\tau }_2,\dots ,{\tau }_N]$ where $a\le {\tau }_1<{\tau }_2<\dots <{\tau }_N\le b$.

For $k=1,\dots ,N$, the estimate of $\mathbf{x}({\tau }_k,·)$, $\widehat{\mathbf{x}}({\tau }_k,·)$, and observed signal $\mathbf{y}({\tau }_k,·)$ are represented by $m\times q$ and $n\times q$ matrices

$$\begin{array}{c}\hfill {\widehat{X}}^{\left(k\right)}=[\widehat{\mathbf{x}}({\tau }_k,{\omega }_1),\dots ,\widehat{\mathbf{x}}({\tau }_k,{\omega }_q)]and{Y}^{\left(k\right)}=[\mathbf{y}({\tau }_k,{\omega }_1),\dots ,\mathbf{y}({\tau }_k,{\omega }_q)].\end{array}$$

(20)

The sequence of fixed time-points ${\left\{t_k\right\}}_1^p\subset [a,b]$ introduced in (1) is such that

$${\tau }_1=t_1<\dots <t_p={\tau }_N,$$

(21)

where $t_1={\tau }_{n_0},t_2={\tau }_{n_0+n_1},\dots ,t_p={\tau }_{n_0+n_1\dots +n_{p-1}},$ and where $n_0=1$ and $n_1,\dots ,n_{p-1}$ are positive integers such that $N=n_0+n_1+\dots +n_{p-1}$.

For $j=1,\dots ,p$, vectors $\widehat{\mathbf{x}}(tj,·)$ and $\mathbf{y}(tj,·)$ associated with $t_j$ in (21) are represented, respectively, by

$${\widehat{X}}_j^{\left(k\right)}=[\widehat{\mathbf{x}}(t_j,{\omega }_1),\dots ,\widehat{\mathbf{x}}(t_j,{\omega }_N)]\mathrm{and}{Y}_j=[\mathbf{y}(t_j,{\omega }_1),\dots ,\mathbf{y}(t_j,{\omega }_N)].$$

4.3.2. Algorithm

As it has been mentioned in Section 3.4, it is supposed that, for $t=t_1,$ an estimate of $X_1$, ${\widehat{X}}_1$, is known and can be determined by the known methods. This is the initial condition of the proposed technique.

On the basis of the results obtained in Section 3.4 and Section 4.2, the performance algorithm of the proposed filter consists of the following steps. For $j=1\dots ,p$, we write $N_j=n_0+n_1+\dots +n_{j-1}$.

Initial parameters: $Y^{\left(1\right)},\dots ,Y^{\left(N\right)}$, ${\left\{t_j\right\}}_{j=1}^p$ (see (21)), ${\left\{E_{z_j{w}_j}\right\}}_{j=1}^p$, ${\left\{E_{w_j{w}_j}\right\}}_{j=1}^p$ (see (14) and (15)), ${\widehat{X}}_1$, $n_0=1$ , $N_0=0$ and $M_{B_j}=\mathbb{O}$, for $j=1,\dots ,p-1$.

(Possible ways to get estimates of $E_{z_j{w}_j}$ and $E_{w_j{w}_j}$ are discussed below in Section 4.5.)

Final parameters:${\widehat{X}}^{\left(2\right)}$, ${\widehat{X}}^{\left(3\right)}$,…, ${\widehat{X}}^{\left(N\right)}$.

Algorithm:

• for $j=1$ to p do

    begin

$$B_j=E_{z_j{w}_j}{E}_{w_j{w}_j}^{†};$$

• for $k=N_{j-1}+1$ to $N_j$ do

     begin

$${\widehat{X}}^{\left(k\right)}={\widehat{X}}_j+B_j(Y^{\left(k\right)}-Y_j);$$

$${\widehat{X}}_{j+1}={\widehat{X}}^{\left(N_j\right)};$$

      end

    end

4.4. Error Analysis

It is natural to expect that the error associated with the piecewise interpolating filter ${ℱ}^{(p-1)}$ decreases when ${\displaystyle \underset{j=1,\dots ,p-1}{max}\Delta t_j}$ decreases. Below, in Theorem 3, we justify that this observation is true. To this end, first, in the following Theorem 2, we establish an estimate of the error associated with the filter F.

Let us introduce the norm by

$$\begin{array}{c}\hfill {\parallel \mathbf{x}(t,·)\parallel }_{T,\Omega }^2={\displaystyle \frac{1}{b-a}}{\int }_T{\parallel \mathbf{x}(t,·)\parallel }_{\Omega }^{2}dt.\end{array}$$

(22)

We also denote ${\parallel \mathbf{x}(t,\omega )\parallel }_{T,\Omega }^2={\parallel \mathbf{x}(t,·)\parallel }_{T,\Omega }^2$.

Let us suppose that $\mathbf{x}(·,\omega )$ and $\mathbf{y}(·,\omega )$ are Lipschitz continuous signals, i.e. that there exist real non-negative constants ${\lambda }_j$ and ${\gamma }_j$, with $j=1,\dots ,p$, such that, for $t\in [t_j,t_{j+1}]$,

$$\begin{array}{c}\hfill \parallel \mathbf{x}(t,\omega )-\mathbf{x}(t_j,\omega ){\parallel }_{T,\Omega }^2\le {\lambda }_j\Delta t_{j}and{\parallel \mathbf{y}(t,\omega )-\mathbf{y}(t_{j+1},\omega )\parallel }_{T,\Omega }^2\le {\gamma }_j\Delta t_j\end{array}$$

(23)

where $\Delta t_j=|t_{j+1}-t_j|$.

Theorem 2. 

Under the conditions (23) the error associated with the piecewise interpolation filter, $\parallel \mathbf{x}(t,\omega )-F^{(p-1)}{\left[\mathbf{y}(t,\omega )\right]\parallel }_{T,\Omega }^2$, is estimated as follows:

$$\begin{array}{ccc}\hfill & & \parallel \mathbf{x}(t,\omega )-F^{(p-1)}{\left[\mathbf{y}(t,\omega )\right]\parallel }_{T,\Omega }^2\hfill \\ & & \le \underset{j=1,\dots ,p-1}{max}\left[({\lambda }_j+{\gamma }_j\parallel B_j{\parallel }^2)\Delta t_j+\parallel E_{z_j{z}_j}^{1/2}{\parallel }^2-{\parallel E_{z_j{w}_j}{\left(E_{w_j{w}_j}^{1/2}\right)}^{†}\parallel }^2\right].\hfill \end{array}$$

(24)

Proof: The proof of Theorem 2 is given in Section A. □

Further, to show that the error of the reference signal estimate tends to the zero, we need to assume that, for $t\in [t_1,t_2]$, the known estimate $\widehat{\mathbf{x}}(t_1,\omega )$ differs from $\mathbf{x}(t,\omega )$ for the value of the order $\Delta t_1$, i.e. that, for some constant $c_1\ge 0$,

$$\begin{array}{c}\hfill \parallel \mathbf{x}(t,\omega )-\widehat{\mathbf{x}}(t_1,\omega ){\parallel }_{\Omega }^2\le c_1\Delta t_1,fort\in [t_1,t_2].\end{array}$$

(25)

Theorem 3. 

Let the conditions (23) and (25) be true. Then the error associated with the piecewise interpolating filter F, $\parallel \mathbf{x}(t,\omega )-F^{(p-1)}{\left[\mathbf{y}(t,\omega )\right]\parallel }_{T,\Omega }^2$, decreases in the following sense:

$$\begin{array}{c}\hfill {\displaystyle \parallel \mathbf{x}(t,\omega )-F^{(p-1)}{\left[\mathbf{y}(t,\omega )\right]\parallel }_{T,\Omega }^2\to 0as\underset{j=1,\dots ,p-1}{max}\Delta t_j\to 0andp\to \infty .}\end{array}$$

(26)

Proof: The proof of Theorem 3 is given in Section A. □

Remark 1. 

We would like to emphasize that the statement of Theorem 3 is fulfilled only under assumptions (23) and (25). At the same time, the assumptions (23) and (25) are not restrictive from a practical point of view. The condition (23) is true for Lipschitz continuous signals $\mathbf{x}$ and $\mathbf{y}$, i.e. for very wide class of signals. The condition (25) is achieved by a choosing an appropriate known method (e.g. see [1]–13,23,25]) to find the estimate $\widehat{\mathbf{x}}(t_1,\omega )$ used in the proposed filter ${ℱ}^{(p-1)}$ (see (8) and Theorem 1).

4.5. Some Remarks Related to the Assumptions of the Method

As it has been mentioned in Section 3.2, for $j=1,\dots ,p$, matrices $E_{z_j{w}_j}$ and $E_{w_j{w}_j}$ in (19) are assumed to be known or can be estimated. Here, p is a chosen number of selected interpolation signal pairs (see Section 3.4). We note that normally p is much smaller than the number of input-output signals $\mathbf{x}(t,·)$ and $\mathbf{y}(t,·)$. Therefore, to estimate any signal $\mathbf{x}(t,·)$ from an arbitrarily large set $K_X$, only a small number, p, of matrices $E_{z_j{w}_j}$ and $E_{w_j{w}_j}$ should be estimated (or be known). This issue has also been discussed in Section 1.1.1 and Section 1.1.4.

By the proposed method, $\mathbf{x}(t,·)$ is estimated for $t\in [t_j,t_{j+1}]$. While $E_{w_j{w}_j}$ in (19) can be directly estimated from observed signals $\mathbf{y}(t_{j+1},·)$ and $\mathbf{y}(t_j,·)$, an estimate of matrix $E_{z_j{w}_j}$ depends on the reference signal $\mathbf{x}(t_{j+1},·)$ (see (14) and (15)) which is unknown (because the estimate is considered for $t\in [t_j,t_{j+1}]$).

Some possible approaches to an estimation of matrix $E_{z_j{w}_j}$ could be as follows.

1. In the general case, when $\mathbf{x}(t,·)$ and $\mathbf{y}(t,·)$ are arbitrary signals as discussed in Section 2.1 above, matrix $E_{z_j{w}_j}$ can be estimated as proposed, for example, in [27], from samples of $z_j$ and $w_j$.

2. In the case of incomplete observations, the method proposed in [28,29] can be used.

3. Let $E_{{\widehat{z}}_j{w}_j}$ be a matrix obtained from matrix $E_{z_j{w}_j}$ where the term $\mathbf{x}(t_{j+1},·)$ is replaced by $\widehat{\mathbf{x}}(t,·)$ with $t\in [t_{j-1},t_j]$. Since $\widehat{\mathbf{x}}(t,·)$ with $t\in [t_{j-1},t_j]$ is known, matrix $E_{{\widehat{z}}_j{w}_j}$ can be considered as an estimate of $E_{z_j{w}_j}$.

4. In the important case of an additive noise, $E_{z_j{w}_j}$ can be represented in the explicit form. Indeed, if

$$\mathbf{y}(t,·)=\mathbf{x}(t,·)+\xi (t,·)$$

where $\xi (t,·)\in L^2(\Omega ,{\mathbb{R}}^m)$ is a random noise, then $\mathbf{z}(t_j,t_{j+1},·)=\mathbf{y}(t_{j+1},·)-\xi (t_{j+1},·)-\widehat{\mathbf{x}}(t_j,·)$ and matrix $E_{z_j{w}_j}$ can be represented as follows:

$$\begin{array}{c}\hfill E_{z_j{w}_j}=E_{(y_{j+1}-{\xi }_{j+1})(y_{j+1}-y_j)}-E_{{\widehat{x}}_j(y_{j+1}-y_j)}\end{array}$$

(27)

We note that the RHS of (27) depends only on observed signals $\mathbf{y}(t_j,·)$, $\mathbf{y}(t_{j+1},·)$, estimated signal $\widehat{\mathbf{x}}(t_j,·)$, and noise $\xi (t_{j+1},·)$, not on the reference signal $\mathbf{x}(t_{j+1},·)$. In particular, in (27), the term $E_{{\xi }_{j+1}(y_{j+1}-y_j)}$ can be estimated as $\pm {\left(E\left[{\xi }_{j+1}^2\right]\right)}^{1/2}{\left(E\left[{(y_{j+1}-y_j)}^2\right]\right)}^{1/2}$ where ${\displaystyle E\left[{\xi }_{j+1}^2\right]={\int }_{\Omega }{\left[\xi (t_{j+1},\omega )\right]}^{2}d\mu \left(\omega \right)}$. It is motivated by the Holder’s inequality for integrals. The second term in (27), $E_{{\widehat{x}}_j(y_{j+1}-y_j)}$, can be estimated from the samples of $\widehat{\mathbf{x}}(t_{j+1},·)$ and $\mathbf{y}(t_{j+1},·)-\mathbf{y}(t_j,·)$.

We also note that the first term in the RHS of (27), $E_{(y_{j+1}-{\xi }_{j+1})(y_{j+1}-y_j)}$, is similar to the related covariance matrix in the Wiener filtering approach [5].

5. Other known ways to estimate $E_{{\xi }_{j+1}(y_{j+1}-y_j)}$ can be found in [5], Section 5.3.

In general, an estimation of covariance matrices is a special research topic which is not a subject of this paper. The relevant references can be found, for example, in [5,29].

5. Simulations

5.1. General Consideration

In these simulations, in accordance with Section 4.3.1, signal sets $K_X$ and $K_Y$ (see Section 2.1) are given by

$$K_X=\{\mathbf{x}({\tau }_1,·),\mathbf{x}({\tau }_2,·),\dots ,\mathbf{x}({\tau }_N,·)\}and{K}_Y=\{\mathbf{y}({\tau }_1,·),\mathbf{y}({\tau }_2,·,\dots ,\mathbf{y}({\tau }_N,·)\},$$

where, for $k=1,\dots ,N$, $\mathbf{x}({\tau }_k,·)\in L^2(\Omega ,{\mathbb{R}}^m)$ and $\mathbf{y}({\tau }_k,·)\in L^2(\Omega ,{\mathbb{R}}^n)$. In many practical problems (arising, for example, in a DNA analysis the number N is quite large, for instance, $N=\mathcal{O}\left({10}^4\right)$.

We set $N=141$ and $m=n=116$. Thus, in these simulations, the interval $T=[a,b]$ (see Section 2.1 and Section 4.3.1) is modelled as 141 points ${\tau }_k$ with $k=1,\dots ,141$ so that $[a,b]=[{\tau }_1,{\tau }_2,\dots ,{\tau }_{141}]$.

The sequence of fixed time-points ${\left\{t_k\right\}}_1^p\subset T$ in (1) is now such that

$${\tau }_1=t_1<\dots <t_p={\tau }_{141}.$$

(28)

Below, in Examples 1-12, four particular choices of the specific interpolation signal pairs $\{\mathbf{x}(t_j,·),$$\mathbf{y}(t_j,·){\}}_1^p$ (introduced in Section 3.4) are considered, for $p=5,8,15$ and 28. Points $t_1,\dots ,t_p$ are as follows.

For $p=5,8,15$, if $j=1,\dots ,p$, then $t_j=t_j^{\left(p\right)}={\tau }_1+(j-1){\Delta }_p$, respectively, where ${\Delta }_5=35,$ ${\Delta }_8=20$ and ${\Delta }_{15}=10$.

For $p=28$, if $j=1,\dots ,p-1$, then $t_j=t_j^{\left(p\right)}={\tau }_1+(j-1){\Delta }_{28}$, and if $j=p$, then $t_{28}=t_{28}^{\left(28\right)}=t_{27}+6=141$, where ${\Delta }_{28}=5$.

Signals $\mathbf{x}({\tau }_k,·)$ and $\mathbf{y}({\tau }_k,·)$ have been simulated as digital images represented by $116\times 256$ matrices

$$\begin{array}{c}\hfill X^{\left(k\right)}=[\mathbf{x}({\tau }_k,{\omega }_1),\dots ,\mathbf{x}({\tau }_k,{\omega }_{256})]\mathrm{and}{Y}^{\left(k\right)}=[\mathbf{y}({\tau }_k,{\omega }_1),\dots ,\mathbf{y}({\tau }_k,{\omega }_{256})],\end{array}$$

(29)

respectively, for $k=1,\dots ,141$, so that $X^{\left(k\right)}$ represents an image that should be estimated from an observed image $Y^{\left(k\right)}$. A column of matrices $X^{\left(k\right)}$ and $Y^{\left(k\right)}$, $\mathbf{x}({\tau }_k,{\omega }_i)\in {\mathbb{R}}^{116}$ and $\mathbf{y}({\tau }_k,{\omega }_i)\in {\mathbb{R}}^{116}$, for $i=1,\dots ,256$, represents a realization of signals $\mathbf{x}({\tau }_k,·)$ and $\mathbf{y}({\tau }_k,·)$, respectively.

Note that $X^{\left(1\right)},\dots ,X^{\left(141\right)}$ did not used in the piecewise linear filter $F^{(p-1)}$ below since they are not supposed to be known. They are represented here for illustration purposes only. In particular, $X^{\left(1\right)},\dots ,X^{\left(141\right)}$ are used to compare their estimates by different filters.

Observed noisy signals $Y^{\left(1\right)},\dots ,Y^{\left(141\right)}$ have been simulated in different forms presented by (40), (49), (50) and (51) in the Examples 1-12 below. We note that the considered observed signals are grossly corrupted.

To estimate the signals $X^{\left(1\right)}$, ..., $X^{\left(141\right)}$ from the observed signals $Y^{\left(1\right)}$, ..., $Y^{\left(141\right)}$, the proposed piecewise linear filter $F^{(p-1)}$, the generic optimal linear (GOL) filters [5] and the averaging polynomial filter [12] have been used.

The filters proposed in [12,13,22,23] have not been applied here by the reasons discussed in Section 1. In particular, the filter in [23] cannot be applied to signals represented by $Y^{\left(1\right)}$, ..., $Y^{\left(141\right)}$ in the form (40), (49), (50) and (51) below because the associated inverse matrices used in [23] do not exist.

For signals under consideration (given by matrices $X^{\left(k\right)}$ and $Y^{\left(k\right)}$ with $k=1,\dots ,141$), the filter $F^{(p-1)}$, the generic optimal linear (GOL) filters [5] and the averaging polynomial filter [10,12] are represented as follows.

(i) Piecewise linear filter $F^{(p-1)}$.For $j=1,\dots ,p$, $\{X_j,Y_j\}$ designates an interpolation pair defined similarly to that in Section 3.4. Each $X_j$ and $Y_j$ is associated with $t_j$ in (25) so that

$$X_j=[\mathbf{x}(t_j,{\omega }_1),\dots ,\mathbf{x}(t_j,{\omega }_{256})]and{Y}_j=[\mathbf{y}(t_j,{\omega }_1),\dots ,\mathbf{y}(t_j,{\omega }_{256})].$$

The estimate ${\widehat{X}}^{\left(k\right)}$ of $X^{\left(k\right)}$ by the filter $F^{(p-1)}$ is given by

$${\widehat{X}}^{\left(k\right)}=F^{(p-1)}\left[Y^{\left(k\right)}\right],$$

(30)

where, by (16)-(19) in Section 4.2,

$$\begin{array}{ccc}\hfill & & F^{(p-1)}\left[Y^{\left(k\right)}\right]=\sum _{j=1}^{p-1}{\delta }_j{F}_j\left[Y^{\left(k\right)}\right],{\delta }_j=\left\{\begin{array}{cc}1,& if{t}_j\le {\tau }_k\le t_{j+1},\hfill \\ 0,& otherwise,\hfill \end{array}\right.\hfill \end{array}$$

(31)

$$\begin{array}{ccc}& & F_j^{(p-1)}\left[Y^{\left(k\right)}\right]={\widehat{X}}_j+B_j[Y^{\left(k\right)}-Y_j],\hfill \end{array}$$

(32)

$$\begin{array}{ccc}& & {\widehat{X}}_j=F_{j-1}\left[Y_j\right],{\widehat{X}}_{1}isgiven,\hfill \end{array}$$

(33)

$$\begin{array}{ccc}& & B_j=E_{ZjWj}{\left(E_{WjWj}\right)}^{†},\hfill \end{array}$$

(34)

and where $E_{ZjWj}$ and $E_{WjWj}$ are estimates of matrices $E_{z_j{w}_j}$ and $E_{w_j{w}_j}$ in (19), respectively. In particular, $E_{W_j{W}_j}$ can be represented in the form

$$E_{WjWj}=W_j{W}_j^T,where{W}_j=Y_{j+1}-Y_j.$$

(35)

Further, matrix $E_{ZjWj}$ depends on $Z_j=X_{j+1}-{\widehat{X}}_j$ where $X_{j+1}$ is unknown. Therefore a determination of $E_{ZjWj}$ is reduced, in fact, to finding an estimate of $X_{j+1}$. Since it is customary to find $E_{ZjWj}$ in terms of signal samples [5], $E_{ZjWj}$ has been presented as

$$E_{ZjWj}={\tilde{Z}}_j{W}_j^T,where{\tilde{Z}}_j={\tilde{X}}_{j+1}-{\widehat{X}}_j$$

(36)

and ${\tilde{X}}_{j+1}$ has been constructed from a sample of $X_{j+1}$ as follows. The sample of $X_{j+1}$ is a $116\times 128$ matrix presented by odd columns of $X_{j+1}$. Then an estimate of $X_{j+1}$ is chosen as a $116\times 256$ matrix ${\tilde{X}}_{j+1}$ where each odd column is a related odd column of $X_{j+1}$, and each even column is an average of two adjacent columns. The last column in ${\tilde{X}}_{j+1}$ is the same as its preceding column.

This way of estimating $E_{ZjWj}$ was chosen for illustration purposes only. Other related methods have been considered in Section 4.5.

The errors associated with the filter $F^{(p-1)}$ are given by

$${\epsilon }_{k,F}^{(p-1)}={∥X^{\left(k\right)}-F^{(p-1)}\left[Y^{\left(k\right)}\right]∥}_F^2,fork=1,\dots ,141.$$

(37)

(ii) Generic optimal linear (GOL) filters [5]. To each signal $Y^{\left(k\right)}$, an individual GOL filter $W_k$ has also been applied, so that $W_k$ estimates $X^{\left(k\right)}$ from $Y^{\left(k\right)}$ in the form

$$W_k{Y}^{\left(k\right)}=E_{X^{\left(k\right)}{Y}^{\left(k\right)}}{E}_{Y^{\left(k\right)}{Y}^{\left(k\right)}}^{†}{Y}^{\left(k\right)},$$

for each $k=1,\dots ,141$. Thus, the GOL filter $W_k$ requires an estimate of 141 matrices $E_{X^{\left(k\right)}{Y}^{\left(k\right)}}$, for each $k=1,\dots ,141$.

Similarly to matrix $E_{Z_j{W}_j}$ in the filter $F^{(p-1)}$ above, the matrix $E_{X^{\left(k\right)}{Y}^{\left(k\right)}}$ has been estimated from samples of each $X^{\left(k\right)}$, ${\tilde{X}}^{\left(k\right)}$, for each $k=1,\dots ,141$.

One of the advantages of the proposed filter$F^{(p-1)}$ is that $F^{(p-1)}$ requires a smaller number, p, of samples of $X_j$, ${\tilde{X}}_j$, to be known (where $j=1,\dots ,p$).

The errors associated with filters $W_k$ are given by

$${\displaystyle {ϵ}_{k,w}={\parallel X^{\left(k\right)}-W_k{Y}^{\left(k\right)}\parallel }_F^2.}$$

(38)

(iii) Averaging polynomial filters [10,12]. By the methodology in [10], the averaging polynomial filter W is based on the use of the estimates of the covariance matrices, $E_{XY}$ and $E_{YY}$, in the form

$$E_{XY}={\displaystyle \frac{1}{141}}\sum _{k=1}^{141}{\tilde{X}}^{\left(k\right)}{\left(Y^{\left(k\right)}\right)}^T\mathrm{and}{E}_{YY}={\displaystyle \frac{1}{141}}\sum _{k=1}^{141}{Y}^{\left(k\right)}{\left(Y^{\left(k\right)}\right)}^T.$$

Then, for each, $k=1,\dots ,141$, the estimate of $X^{\left(k\right)}$ is given by

$$W{Y}^{\left(k\right)}=E_{XY}{E}_{YY}^{†}{Y}^{\left(k\right)}.$$

The errors associated with the filter W are given by

$${\displaystyle {\epsilon }_{k_W}={\parallel X^{\left(k\right)}-W{Y}^{\left(k\right)}\parallel }_F^2,fork=1,\dots ,141.}$$

(39)

5.2. Simulations with Signals Modelled from Images ‘Plant’: Application of Piecewise Interpolation Filter and GOL Filters

Here, results of simulations for reference signals represented by matrices $X^{\left(1\right)},$ $\dots ,$$X^{\left(141\right)}$ (see (29) above) formed from images ‘plant’10 are considered. Typical selected images $X^{\left(k\right)}$ are shown in Figure 9.

Observed noisy images $Y^{\left(1\right)},\dots ,Y^{\left(141\right)}$ have been simulated in the form

$$Y^{\left(k\right)}=X^{\left(k\right)}•{\mathtt{randn}}_{\left(k\right)}•{\mathtt{rand}}_{\left(k\right)},$$

(40)

for each $k=1,\dots ,141.$ Here, • means the Hadamard product, and ${\mathtt{randn}}_{\left(k\right)}$ and ${\mathtt{rand}}_{\left(k\right)}$ are $116\times 256$ matrices with random entries. The entries of ${\mathtt{randn}}_{\left(k\right)}$ are normally distributed with mean zero, variance one and standard deviation one. The entries of ${\mathtt{rand}}_{\left(k\right)}$ are uniformly distributed in the interval $(0,1)$. A typical example of such images is given in Figure 10 (a).

To demonstrate the effectiveness of the proposed filter $F^{(p-1)}$, sub-filters $F_j^{(p-1)}$ and associated interpolation signal pairs ${\{X_j,Y_j\}}_{j=1}^p$ have been chosen in four different ways as follows.

Example 1. First, for $p=5$, the interpolation signal pairs are

$$\begin{array}{ccc}\hfill & & \{X_1,Y_1\}=\{X^{\left(1\right)},Y^{\left(1\right)}\},\{X_2,Y_2\}=\{X^{\left(35\right)},Y^{\left(35\right)}\},\{X_3,Y_3\}=\{X^{\left(70\right)},Y^{\left(70\right)}\},\hfill \end{array}$$

(41)

$$\begin{array}{ccc}& & \{X_4,Y_4\}=\{X^{\left(105\right)},Y^{\left(105\right)}\},\{X_5,Y_5\}=\{X^{\left(141\right)},Y^{\left(141\right)}\}.\hfill \end{array}$$

(42)

The error values ${\left\{{\epsilon }_{k,F}^{\left(4\right)}\right\}}_1^{141}$ associated with filter $F^{\left(4\right)}$ are evaluated by (37). The graph of ${\left\{{\epsilon }_{k,F}^{\left(4\right)}\right\}}_1^{141}$ is presented in Figure 11 (a).

Example 2. For $p=8$, the interpolation signal pairs are

$$\begin{array}{ccc}\hfill & & \{X_1,Y_1\}=\{X^{\left(1\right)},Y^{\left(1\right)}\},\{X_j,Y_j\}=\{X^{\left(20\right(j-1\left)\right)},Y^{\left(20\right(j-1\left)\right)}\},forj=2,\dots ,7;\hfill \end{array}$$

(43)

$$\begin{array}{ccc}& & and\{X_8,Y_8\}=\{X^{\left(141\right)},Y^{\left(141\right)}\}.\hfill \end{array}$$

(44)

The error magnitudes ${\left\{{\epsilon }_{k,F}^{\left(7\right)}\right\}}_1^{141}$ associated with the piecewise interpolation filter $F^{\left(7\right)}$ constructed by (31)-(36) with the interpolation signal pairs given by (43)-(44) are diagrammatically shown in Figure 11 (b).

It follows from Figure 11 (b) that the errors associated with filter $F^{\left(7\right)}$ is less than those of filter $F^{\left(4\right)}$. This is a confirmation of Theorem 3.

Example 3. Further, for $p=15$, the interpolation pairs are

$$\begin{array}{ccc}\hfill & & \{X_1,Y_1\}=\{X^{\left(1\right)},Y^{\left(1\right)}\},\{X_j,Y_j\}=\{X^{\left(10\right(j-1\left)\right)},Y^{\left(10\right(j-1\left)\right)}\}forj=2,\dots ,14;\hfill \end{array}$$

(45)

$$\begin{array}{ccc}& & \mathrm{and}\{X_{15},Y_{15}\}=\{X^{\left(141\right)},Y^{\left(141\right)}\}.\hfill \end{array}$$

(46)

In Figure 11 (c), the errors ${\left\{{\epsilon }_{k,F}^{\left(15\right)}\right\}}_1^{141}$ associated with the piecewise interpolation filter $F^{\left(15\right)}$ are presented. The Figure 11 (c) demonstrates a further confirmation of Theorem 3: the errors associated with the piecewise interpolation filter diminishes as p increases.

Example 4. Finally, the number of interpolation signal pairs ${\{X_j,Y_j\}}_{j=1}^p$ is $p=29$ so that

$$\begin{array}{ccc}\hfill & & \{X_1,Y_1\}=\{X^{\left(1\right)},Y^{\left(1\right)}\},\{X_j,Y_j\}=\{X^{\left(5\right(j-1\left)\right)},Y^{\left(5\right(j-1\left)\right)}\}forj=2,\dots ,28;\hfill \end{array}$$

(47)

$$\begin{array}{ccc}& & \mathrm{and}\{X_{29},Y_{29}\}=\{X^{\left(141\right)},Y^{\left(141\right)}\}.\hfill \end{array}$$

(48)

In this case, when p is grater than in the previous Examples 1-3, the errors ${\left\{{\epsilon }_{k,F}^{\left(29\right)}\right\}}_1^{141}$ associated with the piecewise interpolation filter $F^{\left(29\right)}$ are smaller than those associated with filters $F^{\left(4\right)}$, $F^{\left(8\right)}$ and $F^{\left(15\right)}$ - see Figure 11 (d).

The diagrams of errors associated with the GOL filters [5] are also presented in Figure 11. It follows from Figure 11 that proposed filters $F^{\left(4\right)}$, $F^{\left(8\right)}$, $F^{\left(15\right)}$ and $F^{\left(29\right)}$ provide the better accuracy then that of the GOL filters.

At the same time, the filter $F^{(p-1)}$ is easer to implement since it requires less initial information compared to GOL filters, as it has been discussed in Section 1.1.1 and Section 1.1.4.

Results of the application of the averaging polynomial filter [10,12] are discussed in Section 5.4 below.

5.3. Simulations with Signals Modelled from Images ‘Boat’: Application of Piecewise Interpolation Filter and GOL Filters

In this section, results of the simulations for a different type of signals than those considered in Section 5.2 above are presented. Here, the reference signals $X^{\left(1\right)},$$\dots ,$$X^{\left(141\right)}$ are formed from images ‘boat’11.

Observed noisy signals $Y^{\left(1\right)},\dots ,Y^{\left(141\right)}$ have been simulated in the form

$$Y^{\left(k\right)}=X^{\left(k\right)}•{\mathtt{randn}}_{\left(k\right)},$$

(49)

for each $k=1,\dots ,141.$ The noise term is different from that in (40).

Typical selected images $X^{\left(k\right)}$ and $Y^{\left(k\right)}$ are shown in Figure 12 and Figure 13, respectively.

As in Section 5.2, the piecewise interpolation filter $F^{(p-1)}$ is constructed by (31)-(36). In Examples 5-8 below, the number $p-1$ of sub-filters $F_j^{(p-1)}$ and associated interpolation signal pairs ${\{X_j,Y_j\}}_{j=1}^p$ have been chosen in four different ways.

Example 5. First, similar to Example 1, the number of interpolation signal pairs ${\{X_j,Y_j\}}_{j=1}^p$ has been chosen as $p=5$, and $X_j$ and $Y_j$ have been presented as in (41)-(42).

The error values ${\left\{{\epsilon }_{k,F}^{\left(4\right)}\right\}}_1^{141}$ associated with the piecewise interpolation filter $F^{\left(4\right)}$ applied to these data are presented in Figure 14 (a).

Example 6. For the grater number of interpolation signal pairs than that in Example 5, $p=8$, and for $X_j$ and $Y_j$ ($j=1,\dots ,8$) chosen as in (43)-(44), the error magnitudes ${\left\{{\epsilon }_{k,F}^{\left(7\right)}\right\}}_1^{141}$ associated with the piecewise interpolation filter $F^{\left(7\right)}$ are diagrammatically shown in Figure 14 (b). A comparison between Figure 14 (a) and (b) demonstrates that the increase in p implies the decrease in the errors associated with the filter $F^{(p-1)}$.

Example 7. For $p=15$, and for $X_j$ and $Y_j$ ($j=1,\dots ,15$) chosen as in (45)-(46), the errors ${\left\{{\epsilon }_{k,F}^{\left(14\right)}\right\}}_1^{141}$ associated with the piecewise interpolation filter $F^{\left(14\right)}$ are further less than those for filters $F^{\left(4\right)}$ and $F^{\left(7\right)}$. See Figure 14 (c) in this regard.

Example 8. The further increase in p to $p=29$, confirms this tendency. The piecewise interpolation filter $F^{\left(28\right)}$ with $X_j$ and $Y_j$ ($j=1,\dots ,29$) chosen similar to (47)-(48) produces the associated errors ${\left\{{\epsilon }_{k,F}^{\left(28\right)}\right\}}_1^{141}$ represented in Figure 14 (d). They are, clearly, less than the errors associated with filters $F^{\left(4\right)}$, $F^{\left(7\right)}$ and $F^{\left(15\right)}$.

The errors associated with the GOL filters are also presented in Figure 14 (a)-(d). The figures clearly demonstrate the advantage of the piecewise interpolation filter $F^{(p-1)}$.

Results of the application of the averaging polynomial filter [12] are discussed in Section 5.4 below.

5.4. Results of Simulations for Averaging Polynomial Filter [10,12]

To further illustrate the effectiveness of the proposed piecewise interpolation filter, in this Section, results of simulations for the averaging polynomial filter [10,12] are presented. The filter has been applied to two different types of data considered in Section 5.2 and Section 5.3.

Example 9. The filter [10,12] applied to signals considered in Section 5.2 gives the associated errors ${\left\{{ϵ}_{k_W}\right\}}_{k=1}^{141}$ (see (39)) represented in Figure 15 (a). For a comparison, the errors associated with the piecewise interpolation filter $F^{\left(28\right)}$ and the GOL filters [5] are also given in Figure 15 (a).

A typical example of the estimated signal by the averaging polynomial filter [10,12] is presented in Figure 10 (d) above.

Example 10. The averaging polynomial filter [12] applied to signals considered in Section 5.3 produces the associated errors ${\left\{{ϵ}_{k_W}\right\}}_{k=1}^{141}$ shown in Figure 15 (b). The errors associated with the piecewise interpolation filter $F^{\left(28\right)}$ are much smaller and they are not discerned in Figure 15 (b).

Together with Figure 10, Figure 11, Figure 13 and Figure 14, Figure 15 (a) and (b) illustrate the advantage of the piecewise interpolation filter.

Figure 1. Examples of selected signals to be estimated from observed data.

Figure 1. Examples of selected signals to be estimated from observed data.

Figure 2. Examples of the observed signal and the estimates obtained by different filters.

Figure 2. Examples of the observed signal and the estimates obtained by different filters.

Figure 3. Illustration of the errors associated with the piecewise interpolation filters $F^{(p-1)}$ and the GOL filters [5] applied to signals described in Examples 1–4.

Figure 3. Illustration of the errors associated with the piecewise interpolation filters $F^{(p-1)}$ and the GOL filters [5] applied to signals described in Examples 1–4.

Figure 4. Examples of selected signals to be estimated from observed data considered in Example 5-9.

Figure 4. Examples of selected signals to be estimated from observed data considered in Example 5-9.

Figure 5. Examples of the observed signal and the estimates obtained by different filters.

Figure 5. Examples of the observed signal and the estimates obtained by different filters.

5.5. Further Simulations with Different Type of Noise

In Examples 11 and 12 below, a different type of noise is considered. Unlike the multiplicative noise in (40) and (49), here, the noise is additive.

Example 11. First, the piecewise interpolation filter $F^{\left(28\right)}$, the GOL filters [5] and the averaging polynomial filter [12] have been applied to the observed signals given by

$$Y^{\left(k\right)}=X^{\left(k\right)}+900\times {\mathtt{randn}}_{\left(k\right)},\mathrm{for} k=1,\dots ,141.$$

(50)

where $X^{\left(k\right)}$ is as in Section 5.2, i.e. $X^{\left(k\right)}$ is formed from the images ‘plant’. In Figure 8 (a), the diagrams of the errors associated with filter $F^{\left(28\right)}$ and the GOL filters [5] are given. The errors associated with the averaging polynomial filter [12], ${\left\{{ϵ}_{k_W}\right\}}_{k=1}^{141}$, are much grater (of order $\mathcal{O}\left({10}^9\right)$) and they are not presented in Figure 8 (a).

Figure 6. Illustration of errors associated with the piecewise interpolation filter $F^{(p-1)}$ of order p and the generic optimal linear (GOL) filters [5] applied to signals described in Examples 5–8.

Figure 6. Illustration of errors associated with the piecewise interpolation filter $F^{(p-1)}$ of order p and the generic optimal linear (GOL) filters [5] applied to signals described in Examples 5–8.

Figure 7. Illustration of errors associated with the averaging polynomial filters [10,12] in Examples 9 and 10.

Figure 7. Illustration of errors associated with the averaging polynomial filters [10,12] in Examples 9 and 10.

Example 12. In this example, the reference signals $X^{\left(1\right)},\dots ,X^{\left(141\right)}$ are as those in Section 5.3, i.e. they are formed from the image ‘boat’. The observed signals are given by

$$Y^{\left(k\right)}=X^{\left(k\right)}+1000\times {\mathtt{randn}}_{\left(k\right)},\mathrm{for} k=1,\dots ,141.$$

(51)

The piecewise interpolation filter $F^{\left(28\right)}$ and the GOL filters [5] estimate the reference signals with the associated errors represented in Figure 8 (b). As in Example 11 above, in this case, the errors associated with the averaging polynomial filter [10,12] are much grater (of order $\mathcal{O}\left({10}^{10}\right)$) and they are not presented in Figure 8 (b).

Examples 11 and 12 further demonstrate the advantages of the proposed piecewise interpolation filter.

Figure 8. Illustration of errors associated with the piecewise interpolation filter $F^{(p-1)}$ and the generic optimal linear (GOL) filters [5] applied to signals described in Examples 11 and 12.

Figure 8. Illustration of errors associated with the piecewise interpolation filter $F^{(p-1)}$ and the generic optimal linear (GOL) filters [5] applied to signals described in Examples 11 and 12.

Figure 9. Examples of selected signals to be estimated from observed data.

Figure 9. Examples of selected signals to be estimated from observed data.

Figure 10. Examples of the observed signal and the estimates obtained by different filters.

Figure 10. Examples of the observed signal and the estimates obtained by different filters.

Figure 11. Illustration of the errors associated with the piecewise interpolation filters $F^{(p-1)}$ and the generic optimal linear (GOL) filters [5] applied to signals described in Examples 1–4.

Figure 11. Illustration of the errors associated with the piecewise interpolation filters $F^{(p-1)}$ and the generic optimal linear (GOL) filters [5] applied to signals described in Examples 1–4.

Figure 12. Examples of selected signals to be estimated from observed data considered in Example 5-9.

Figure 12. Examples of selected signals to be estimated from observed data considered in Example 5-9.

Figure 13. Examples of the observed signal and the estimates obtained by different filters.

Figure 13. Examples of the observed signal and the estimates obtained by different filters.

Figure 14. Illustration of errors associated with the piecewise interpolation filter $F^{(p-1)}$ of order p and the generic optimal linear (GOL) filters [5] applied to signals described in Examples 5–8.

Figure 14. Illustration of errors associated with the piecewise interpolation filter $F^{(p-1)}$ of order p and the generic optimal linear (GOL) filters [5] applied to signals described in Examples 5–8.

Figure 15. Illustration of errors associated with the averaging polynomial filters [10,12] in Examples 9 and 10.

Figure 15. Illustration of errors associated with the averaging polynomial filters [10,12] in Examples 9 and 10.

5.6. Summary of Simulations

The above simulations confirm the theoretical results obtained in Theorems 1–3. In particular, Figs. Figure 11 and Figure 14 demonstrate that the error associated with the piecewise interpolation filter $F^{(p-1)}$ decreases when the number of sub-filters $F_1,\dots ,F_p$, p, increases.

A comparison between the proposed filter $F^{(p-1)}$ and the known related filters [5,10,12,22,23] has been done. The filter $F^{(p-1)}$ estimates the reference signals with the accuracies that are much better than those of the generic optimal linear (GOL) filters [5] and the averaging polynomial filter [10,12]. Further, the filters proposed in [22,23] fail in processing the signals under consideration. This is because the observed signals in (40), (49), (50) and (51) are grossly corrupted and, therefore, the inverse matrices used in the filter structures in [23] do not exist. The technique in [22] requires the use of the reference signal in the proposed filter which is supposed to be unknown in the simulations above.

The filters have been applied to the different signal sets (presented in Section 5.2 and Section 5.3), using different forms of noise (given in (40), (49), (50) and (51)).

computational work associated with the proposed filter $F^{(p-1)}$ is substantially less than that associated with the known filters discussed in Section 1 (in particular, with the filters in [11]–23]). This is because, for the processing of a data set containing N signals, filter $F^{(p-1)}$ requires computation of p covariance matrices with $p\ll N$ while the known filters require computation of N matrices (in the above Examples, $p=5,7,15,28$, respectively, and $N=141$).

6. Conclusions

The technique of the constructive approximation of a nonlinear operator $K_Y\to K_X$ has been provided. Here, $ℱK_X=\{\mathbf{x}(t,·)\in L^2(\Omega ,{\mathbb{R}}^m)|t\in T\}$ and $K_Y=\{\mathbf{y}(t,·)\in L^2(\Omega ,{\mathbb{R}}^n)|t\in T\}$ where $T:=[a,b]\subseteq \mathbb{R}$ and $\Omega =\left\{\omega \right\}$ is the set of outcomes of a probability space. The device behind the proposed method is based on a special extension of the piecewise linear interpolation technique to the case of stochastic sets $K_X$ and $K_Y$ . The proposed methodology is motivated by the problems arising in signal processing where the nonlinear operator $ℱ$ is interpreted as a nonlinear system (or a nonlinear filter) transforming a set of stochastic signals K_Y to the set of stochastic signals K_X Therefore, the provided technique provides an effective way to transform large data sets.

Distinctive features of the approach are as follows.

(i) The proposed filter ${ℱ}^{(p-1)}:K_Y\to K_X$ is nonlinear and is presented in the form of a sum with $p-1$ terms where each term, ${ℱ}_j:K_{Y,j}\to K_{X,j}$, is interpreted as a particular sub-filter. Here, $K_{Y,j}$ and $K_{X,j}$ are ‘small’ pieces of $K_Y$ and $K_X$, respectively.

(ii) The prime idea is to exploit a prior information only onfew reference signals, p, from the set $K_X$ that contains $N\gg p$ signals (or even an infinite number of signals) and determine $ℱ$_j separately, for each pieces $K_{Y,j}$ and $K_{X,j}$, so that the associated error is minimal. In other words, the filter ${ℱ}^{(p-1)}$ is flexible to changes in the sets of observed and reference signals $K_Y$ and $K_X$, respectively.

(iii) Due to the specific way of determining $ℱ$_j, the filter ${ℱ}^{(p-1)}$ provides a smaller associated error than that for the processing of the whole set $K_Y$ by a filter which is not specifically adjusted to each particular piece $K_{Y,j}$. Moreover, the error associated with our filter decreases when the number of its terms, ${ℱ}_1,\dots ,{ℱ}_{(p-1)}$, increases.

(iv) While the proposed filter ${ℱ}^{(p-1)}$ processes arbitrarily large (and even infinite) signal sets, the filter is nevertheless fixed for all signals in the sets.

(v) The filter ${ℱ}^{(p-1)}$ is determined in terms of pseudo-inverse matrices so that the filter always exists.

(vi) computational load associated with the filter ${ℱ}^{(p-1)}$ is less than that associated with other known filters applied to the processing of large signal sets.