Approximating Functions with Multi-Features on Sphere by Deep Convolutional Neural Networks

1. Introduction

Deep neural networks have demonstrated exceptional computational power and benefited greatly from the availability of large-scale data [1], leading to remarkable performance across a wide range of scientific and engineering applications. Among these architectures, deep convolutional neural networks (DCNNs), which consist of multiple convolutional layers, have proven particularly effective. DCNNs have been successfully applied to speech recognition, image classification, strategic decision-making in board games, and many other practical tasks [2,3,25,26,27].

In the context of approximation theory, DCNNs have shown strong potential [4,5,6,7,12,28], and their approximation capabilities are often regarded as a form of nonlinear approximation [31]. Early foundational work [13,29,30] examined shallow neural networks with sigmoidal-type activation functions for function approximation. More recently, DCNNs with rectified linear unit (ReLU) activation functions have been shown to possess universal approximation properties and continuous functions [8]. Furthermore, [9] demonstrated that DCNNs require only one-eighth of the free parameters needed by fully connected neural networks (FNNs) to approximate smooth functions.

Nevertheless, several challenges persist. For instance, the error rates reported in [10,11,19] suffer from the curse of dimensionality, with the approximation error deteriorating as the input dimension d increases. In an effort to address this, [21] considered the approximation of composite functions of the form $f(Q(x))$ in the Sobolev space $W_{\infty }^{\beta }({\mathbb{B}}^d)$ for $(0<\beta \le 1)$, where $Q(x)$ is polynomial. This approach achieved an error rate of $O({ϵ}^{{\displaystyle \frac{-1}{\beta }}})$ for $(0<ϵ<1)$. However, this method assumes a single feature, implying strong dependence among the input variables.

It has been shown that convolutional layers can effectively extract relevant features [32,33]. Neural networks, due to their mesh-free architectures, have also demonstrated the ability to overcome the curse of dimensionality when the target functions exhibit intrinsic structures [17,18,34], possess fast decaying Fourier coefficients [13,14], or belong to general mixed smooth function spaces [15,16]. To further reduce variable dependence and alleviate the curse of dimensionality, [20] proposed the approximation of functions with multiple features in Sobolev space $W_{\infty }^{\beta }({[0,1]}^d)$ for $(0<\beta \le 1)$. This setting is more realistic for complex tasks, where numerous features are typically required to describe objects. In this case, the error rate becomes $O({ϵ}^{{\displaystyle \frac{-d^{\ast }}{\beta }}})$, where $d^{\ast }$ denotes the number of features and $d^{\ast }\le d$. This indicates that the curse of dimensionality can be mitigated by extracting features; however, the error rate is still worse than in the case $d^{\ast }=1$, as the input variables exhibit reduced dependence.

Despite these advances, several challenges remain unresolved. This paper focuses on approximating functions with multiple features defined on the unit sphere, i.e., $f^{\ast }\in W_{\infty }^{\beta }({\mathbb{S}}^{d-1})$ for $\beta \in (0,1]$. The target functions are of the form:

$$f^{\ast }(x)=f(F_1(x),F_2(x),\dots ,F_{d^{\ast }}(x)),x\in {\mathbb{S}}^{d-1},$$

(1)

where each ${\left\{F_{\tau }(x)\right\}}_{\tau =1}^{d^{\ast }}$ is a smooth function in $W_{\infty }^r({\mathbb{S}}^{d-1})$ with $(r>0)$. In addition, for further alleviating the curse of dimensionality in function approximation, we approximate functions with spherical polynomial features and functions with symmetric polynomial features in Scetion 3.4 and 3.5 respectively. The DCNN architecture proposed in Section 3.2 is capable of extracting three types of features: general smooth features, polynomial features, and symmetric polynomial features. In contrast, DCNNs constructed in [20] are limited to extracting only polynomial and symmetric polynomial features.

2. Preliminaries

We begin by establishing notation for function classes. Let ${\mathbb{S}}^{d-1}=\{x\in {\mathbb{R}}^d{:\parallel x\parallel }_2=1\}$ denote the unit sphere in ${\mathbb{R}}^d$.

2.1. Deep Convolutional Neural Networks with Down Sampling

DCNNs, constructed using convolutional layers, have demonstrated significant effectiveness in tasks, such as image classification [3]. These networks consist of a sequence of convolutional filters denoted by $\mathbf{w}={\{w^{(j)}:\mathbb{Z}\to \mathbb{R}\}}_{j=1}^J$, where each filter $w^{(j)}$ has infinite support over the index set $\{0,\dots ,s^{(j)}\}$ for some $s^{(j)}\in \mathbb{N}$, referred to as filter length. In this work, we assume a uniform filter length across layers, that is, $s^{(j)}\equiv s\in \mathbb{N}$, which yields the dimensionality sequence ${\{d_j=d+js\}}_{j=1}^J$. The activation function employed is the ReLU.

$$\sigma (u)=\left\{\begin{array}{cc}u\hfill & u\ge 0,\hfill \\ 0\hfill & u<0.\hfill \end{array}\right.$$

(2)

Given a filter w with support on the set $\{0,\dots ,s\}$, its convolution with a sequence $v=(v_1,\dots ,v_{d_{j-1}})$ produces a new sequence denoted by $w\ast v$ defined as follows:

$${(w\ast v)}_i=\sum _{k\in \mathbb{Z}}{w}_{i-k}{v}_k=\sum _{k=1}^{d_{j-1}}{w}_{i-k}{v}_k,i\in \mathbb{Z},$$

(3)

which has support on the set $\{1,\dots ,d_{j-1}+s\}$. A DCNN comprising J hidden layers, with neuron mappings ${\{h^{(j)}:\mathbb{R}\to {\mathbb{R}}^{d_j}\}}_{j=1}^J$ is defined iteratively starting from the input vector $h^{(0)}=x$, $x=(x_1,\dots ,x_d)$ and

$$h^{(j)}(x)=\sigma (T^{(j)}{h}^{(j-1)}(x)-b^{(j)}).$$

(4)

where $T^{(j)}$ represents the Toeplitz type convolutional matrix.

$$T^{(j)}:={(w_{i-k}^{(j)})}_{i=1,\dots ,d_j,k=1,\dots ,d_{j-1}},$$

(5)

associated with a filter $w^{(j)}$ of filter length s and $d_j\in N$ given explicitly by

$$T^{(j)}=\left[\begin{array}{ccccccc}{w}_0^{(j)}& 0& 0& \dots & \dots & & 0\\ w_1^{(j)}& w_0^{(j)}& 0& 0& \dots & & 0\\ ⋮& \ddots & \ddots & \ddots & \ddots & & ⋮\\ w_s^{(j)}& w_{s-1}^{(j)}& \dots & w_0^{(j)}& 0& \dots & 0\\ 0& w_s^{(j)}& \dots & w_1^{(j)}& w_0^{(j)}& \dots & 0\\ ⋮& \ddots & \ddots \ddots & \ddots \ddots & \ddots & & ⋮\\ \dots & \dots & 0& w_s^{(j)}& \dots & & w_0^{(j)}\\ \dots & \dots & \dots & 0& w_s^{(j)}& \dots & w_1^{(j)}\\ ⋮& \ddots & \ddots & \ddots & \ddots & ⋮\\ 0& \dots & \dots & \dots & 0& w_s^{(j)}& w_{s-1}^{(j)}\\ 0& \dots & \dots & \dots & \dots & 0& w_s^{(j)}\end{array}\right]\in {\mathbb{R}}^{d_j\times d_{j-1}}.$$

(6)

A constraint is imposed on the bias vectors ${\left\{b^{(j)}\right\}}_{j=1}^J$ associated with the convolutional layers,

$$b_{s+1}^{(j)}=\dots =b_{d_j-s}^{(j)},j=1,\dots ,J.$$

(7)

Definition 1 

(Deep Convolutional Neural Networks with Downsampling). Let $x=(x_1,x_2,\dots ,x_d)\in {\mathbb{S}}^{d-1}$ represent the input data vector, and let $s\in \mathbb{N}$ denote the filter length. A downsampled DCNNs with mappings ${\{h^{(j)}:{\mathbb{R}}^d\to {\mathbb{R}}^{d_j}\}}_{j=1}^J$, incorporating downsampling operations at selected layers, is defined recursively. The layer widths ${\left\{d_j\right\}}_{j=1}^J$ are determined as follows: $d_0=d$, $d_{J_1}=\lfloor {\displaystyle \frac{d+J_{1}s}{D_1}}$⌋, $d_{J_2}=\lfloor {\displaystyle \frac{d_{J_1}+(J_2-J_1)s}{D_2}}\rfloor$, $d_{J_k}=\lfloor {\displaystyle \frac{d_{J_{k-1}}+(J_k-J_{k-1})s}{D_k}}\rfloor$ and more generally,

$$d_j=d_{j-1}+s,j\in \{1,2,\dots ,J\}\setminus \{J_1,J_2,\dots ,J_k\}$$

is a sequence of vectors of functions defined iteratively by $h^{(0)}=x$. The downsampling operator ${\mathfrak{D}}_D:{\mathbb{R}}^{\widehat{D}}\to {\mathbb{R}}^{\lfloor \widehat{D}/D\rfloor }$, parameterized by scaling factor $D\le \widehat{D}$, is defined by

$${\mathfrak{D}}_D(v)={(v_{iD})}_{i=1}^{\lfloor \widehat{D}/D\rfloor },v={(v_i)}_{i=1}^{\widehat{D}}\in {\mathbb{R}}^{\widehat{D}},$$

where $\lfloor u\rfloor$ denotes the integer part of $u>0$. Denote the activated affine mapping ${\mathcal{C}}_{T,b}$ induced by the connection matrix T and bias vector b as

$${\mathcal{C}}_{T,b}(x)=\sigma (Tx-b),$$

and

$$h^{(j)}(x)=\left\{\begin{array}{cc}{\mathcal{C}}_{T^{(j)},b^{(j)}}(h^{(j-1)}(x))\hfill & \mathrm{if}{J}_{k-1}<j<J_k,\hfill \\ {\mathfrak{D}}_{D_k}\circ {\mathcal{C}}_{T^{(j)},b^{(j)}}(h^{(j-1)}(x))\hfill & \mathrm{if}j=J_k,\hfill \end{array}\right.$$

where $J_0=0$, and let $\{D_1,D_2,\dots ,D_k\}$ denote the downsampling scaling factors corresponding to the layers $\{J_1,J_2,\dots ,J_k\}$. The bias vectors satisfy

$${(b^{(j)})}_{s+1}={(b^{(j)})}_{s+2}=\dots ={(b^{(j)})}_{d_{j-1}},j\ne \{J_1,J_2,\dots ,J_k\}.$$

In this study, we refer to $J_1$ as the first group of convolutional layers, while $J_2-J_1$ constitutes the second group, and in general, $J_k-J_{k-1}$ represents the k-th group of convolutional layers.

2.2. Spherical Harmonics and Sobolev Space on the Sphere

Let ${\mathcal{H}}_k^d$ represent the set of all spherical harmonics of degree k on ${\mathbb{S}}^{d-1}$. This space can be characterized as the eigenfunction corresponding to the Laplace-Beltrami operator ${\mathsf{\Delta }}_0$ on ${\mathbb{S}}^{d-1}$ (see [22], Chapter 1.4):

$${\mathcal{H}}_k^d=\{F\in C^2({\mathbb{S}}^{d-1}):{\mathsf{\Delta }}_{0}F=-{\lambda }_{k}F\},$$

(8)

where ${\lambda }_k=k(k+d-2)$. The dimension of the linear space ${\mathcal{H}}_k^d$ is

$$N(k,d)=\left({\displaystyle \genfrac{}{}{0pt}{}{k+d-1}{k}}\right)-\left({\displaystyle \genfrac{}{}{0pt}{}{k+d-3}{k-2}}\right)=O(k^{d-2}).$$

(9)

Let ${\mathrm{Proj}}_k:L_2({\mathbb{S}}^{d-1})\to {\mathcal{H}}_k^d$ represent the orthogonal projection from $L_2({\mathbb{S}}^{d-1})$ to ${\mathcal{H}}_k^d$. The spaces ${\mathcal{H}}_k^d$, where $k\in {\mathbb{Z}}_{+}$, are mutually orthogonal with respect to the inner product of $L_2({\mathbb{S}}^{d-1})$ and possess a spherical harmonic expansion:

$$F=\sum _{k=0}^{\infty }{\mathrm{Proj}}_{k}F=\sum _{k=0}^{\infty }\sum _{l=1}^{N(k,d)}{\widehat{F}}_{k,l}{Y}_{k,l},$$

(10)

where ${\left\{Y_{k,l}\right\}}_{l=1}^{N(k,d)}$ forms an orthonormal basis of ${\mathcal{H}}_k^d$, and ${\widehat{F}}_{k,l}$ represents the Fourier coefficients of F, which are given by the expression:

$${\widehat{F}}_{k,l}={\langle F,Y_{k.l}\rangle }_{L_2({\mathbb{S}}^{d-1})}={\int }_{{\mathbb{S}}^{d-1}}F(x)Y_{k,l}(x)d\mu (x).$$

(11)

The normalized spherical measure $\mu$ on ${\mathbb{S}}^{d-1}$ is defined by the surface area ${\omega }_d={\displaystyle \frac{2{\pi }^{\frac{d}{2}}}{\Gamma (\frac{d}{2})}}$, such that $\mu ({\mathbb{S}}^{d-1})=1$).

In addition, ${\mathrm{Proj}}_{k}F(x)$ has an integral representation given by:

$${\mathrm{Proj}}_{k}F(x)={\int }_{{\mathbb{S}}^{d-1}}F(y)Z_k(x,y)d\mu (y),x,y\in {\mathbb{S}}^{d-1},$$

(12)

where

$$Z_k(x,y)=\sum _{l=1}^{N(k,d)}{Y}_{k,l}(x)Y_{k,l}(y),x,y\in {\mathbb{S}}^{d-1}.$$

(13)

It can be easily demonstrated that $Z_k(x,y)$ is the reproducing kernel of ${\mathcal{H}}_k^d$ independent of the choice of ${\left\{Y_{k,l}\right\}}_{l=1}^{N(k,d)}$. Furthermore, with $\lambda ={\displaystyle \frac{d-2}{2}}$,

$$Z_k(x,y)={\displaystyle \frac{k+\lambda }{\lambda }}{C}_k^{\lambda }(\langle x,y\rangle ),x,y\in {\mathbb{S}}^{d-1},$$

(14)

where $C_k^{\lambda }(t)$ is the Gegenbauer polynomial of degree k with parameter $\lambda >{\displaystyle \frac{-1}{2}}$, as discussed, for instance, in [22]. Consequently, for any $x,y\in {\mathbb{S}}^{d-1}$, ${\displaystyle \frac{k+\lambda }{\lambda }}{C}_k^{\lambda }(\langle x,y\rangle )$ serves as a reproducing kernel of ${\mathcal{H}}_k^d$ in the sense that

$${\int }_{{\mathbb{S}}^{d-1}}p(y){\displaystyle \frac{k+\lambda }{\lambda }}{C}_k^{\lambda }(\langle x,y\rangle )d\mu (y)=p(x),\forall p\in {\mathcal{H}}_k^d.$$

(15)

Definition 2 

(Sobolev Space on Sphere). Let $3\le d\in \mathbb{N}\setminus \left\{0\right\}$, and for $r>0$, the Sobolev space $W_{\infty }^r({\mathbb{S}}^{d-1})$ is defined as

$$W_{\infty }^r({\mathbb{S}}^{d-1}):=\left\{F\in L_{\infty }({\mathbb{S}}^{d-1}):{\parallel {(-{\mathsf{\Delta }}_0+I)}^{r/2}F\parallel }_{L_{\infty }({\mathbb{S}}^{d-1})}<\infty \right\},$$

(16)

with the norm given by

$${\parallel F\parallel }_{W_{\infty }^r({\mathbb{S}}^{d-1})}:={\parallel {(-{\mathsf{\Delta }}_0+I)}^{r/2}F\parallel }_{L_{\infty }({\mathbb{S}}^{d-1})}=\parallel \sum _{k=0}^{\infty }{(1+{\lambda }_k)}^{{\displaystyle \frac{r}{2}}}\sum _{l=1}^{N(k,d)}{\widehat{F}}_{k,l}{Y}_{k,l}{\parallel }_{L_{\infty }({\mathbb{S}}^{d-1})}.$$

(17)

2.3. Near-Optimal Approximation and Cubature Formula

The optimal approximation of a function using polynomial spaces of varying degrees is generally nonlinear. In spherical harmonic analysis, a valuable method for approximation is the application of a linear operator, denoted as $L_n$.

Let $\eta \in C^{\infty }([0,\infty ])$ with $\eta (t)=1$ be a smooth function, such that $t\in [0,1]$, $0\le \eta (t)\le 1$ for $t\in [1,2]$, and $\eta (t)=0$ for $t\ge 2$. Define

$$k_n(t)=\sum _{k=0}^{2n}{\left(\eta ({\displaystyle \frac{k}{n}})\right)}^2{\displaystyle \frac{\lambda +k}{\lambda }}{C}_k^{\lambda }(t),t\in [-1,1],$$

and introduce a sequence of linear operators $K_n$, for $n\in {\mathbb{Z}}_{+}$, on the space $L_{\infty }({\mathbb{S}}^{d-1})$ as follows:

$$K_n(F)(x)={\int }_{{\mathbb{S}}^{d-1}}F(y)k_n(\langle x,y\rangle )d\mu (y),x\in {\mathbb{S}}^{d-1}.$$

This integral operator is bounded and yields effective approximations for functions belonging to Sobolev spaces.

Proposition 1 

([19]). For $n\in \mathbb{N}$, $r>0$ and $F\in W_{\infty }^r({\mathbb{S}}^{d-1})$, there holds

$$\parallel F-K_n(F){\parallel }_{L_{\infty }({\mathbb{S}}^{d-1})}\le C_{\eta }{2}^{d-1}{n}^{-r}{\parallel F\parallel }_{W_{\infty }^r({\mathbb{S}}^{d-1})},$$

where $C_{\eta }$ represents a constant depending only on the function η. Furthermore, with a constant $C_{\eta ,d}>0$ depending on d and η,

$$\parallel K_n{(F)\parallel }_{L_{\infty }({\mathbb{S}}^{d-1})}\le C_{\eta ,d}{\parallel F\parallel }_{W_{\infty }^r({\mathbb{S}}^{d-1})}.$$

To obtain a discrete representation of $K_n(F)$, a cubature formula is employed for integrating polynomials of degree up to $4n$ on the unit sphere ${\mathbb{S}}^{d-1}$, with $d\ge 3$ ([23], Theorem 3.1).

(Cubature Formula) There exist a constant $C_d>0$ that depends only on d, such that for any $m\ge C_d{n}^{d-1}$, there exist positive constants ${\gamma }_l$ and points $y_l\in {\mathbb{S}}^{d-1}$, for $l=1,\dots ,m$, satisfying the following relation:

$${\int }_{{\mathbb{S}}^{d-1}}F(x)d\mu (x)=\sum _{l=1}^m{\gamma }_{l}F(y_l),\forall F\in \prod _{4n}({\mathbb{S}}^{d-1}),$$

(18)

where ${\prod }_{4n}({\mathbb{S}}^{d-1})$ denotes to the space of polynomials of degree up to $4n$ on ${\mathbb{S}}^{d-1}$. Furthermore, for $F\in {\prod }_{4n}({\mathbb{S}}^{d-1})$, the following inequality holds:

$${\parallel F\parallel }_{L_{\infty }({\mathbb{S}}^{d-1})}\asymp \underset{l=1,\dots ,m}{max}{n}^{d-1}{\gamma }_l\left|F(y_l)\right|,$$

(19)

where $A\asymp B$ indicates that existence of $C_1>0$ and $C_2>0$ that are independent of n or m, such that $C_{1}A\le B\le C_{2}A$. In particular, we say that the family ${\left\{({\gamma }_l,y_l)\right\}}_{l=1}^m$ follows a cubature rule of degree $4n$.

3. Approximating Functions with Multiple Features on Sphere

In practical deep learning applications, the input dimension d of the target function is often quite large. As d increases, the approximation rates generally worsen exponentially, leading to what is known as the curse of dimensionality. This poses significant challenges when working with high-dimensional functions.

However, when the input variables are interdependent, meaning there are inherent structures within the target functions, DCNNs can help alleviate the curse of dimensionality by making use of these underlying relationships.

Our first target is to approximate composite function with general smooth features

$$f\in W_{\infty }^{\beta }({[-B_F,B_F]}^{d^{\ast }}),\beta \in (0,1]$$

with $B_F={max}_{\tau }\{\parallel F_{\tau }{{\parallel }_{\infty }\}}_{\tau =1}^{d^{\ast }}$:

$$f^{\ast }(x)=f(F_1(x),F_2(x),\dots ,F_{d^{\ast }}(x)),x\in {\mathbb{S}}^{d-1},$$

(20)

where $d^{\ast }\le d$, and ${\left\{F_{\tau }(x)\right\}}_{\tau =1}^{d^{\ast }}\in W_{\infty }^r({\mathbb{S}}^{d-1})$ ($r>0$). Since the outer function f is Lipschitz-$\beta$ continuous, the norm of f is given by

$${\parallel f\parallel }_{W_{\infty }^{\beta }({[-B_F,B_F]}^{d^{\ast }})}:=\underset{x\in {[-B_F,B_F]}^{d^{\ast }}}{sup}\left|f(x)\right|+\underset{x\ne y\in {[-B_F,B_F]}^{d^{\ast }}}{sup}{\displaystyle \frac{|f(x)-f(y)|}{{|x-y|}^{\beta }}}.$$

(21)

Theorem 1. 

Let $2\le s\le d$, $d\ge 3$, $d^{\ast }\le d$, $N\in \mathbb{N}$, and let ${\left\{F_{\tau }(x)\right\}}_{\tau =1}^{d^{\ast }}$ be smooth functions. Define $B_F=max\{\parallel F_{\tau }{{\parallel }_{\infty }\}}_{\tau =1}^{d^{\ast }}$, and let $f\in W_{\infty }^{\beta }({[-B_F,B_F]}^{d^{\ast }})$ with $\beta \in (0,1]$. For approximating functions of the form (20), there exists a downsampled DCNNs as defined in Definition 1 (explicitly constructed as in Section 3.1), with downsampling scaling parameters $\{d,1\}$ at layers $\{J_1,J_2\}$, respectively, and filters ${\left\{w^{(j)}\right\}}_{j=1}^{J_2}$ of uniform length s, bias vectors ${\left\{b^{(j)}\right\}}_{j=1}^{J_2+1}$, connection matrix $F^{(J_2+1)}$, and coefficient vector $c^{(J_2+1)}\in {\mathbb{R}}^N$, such that

$$\parallel c^{(J_2+1)}·h^{(J_2+1)}-f^{\ast }{\parallel }_{L_{\infty }({\mathbb{S}}^{d-1})}\le c_1{\parallel f\parallel }_{W_{\infty }^{\beta }}{N}^{{\displaystyle \frac{-\beta }{d^{\ast }}}}$$

where $c_1=c_1(d^{\ast },d,\beta ,F_{\tau },\eta )$, and $\eta \in C^{\infty }([0,\infty ])$ defined in Section 2.3.

3.1. Proof of Theorem 1

Before the proof, we firstly introduce the two Lemmas for approximating inner and outer functions.

Define a kernel:

$$l_n(t)=\sum _{k=0}^{2n}\eta ({\displaystyle \frac{k}{n}}){\displaystyle \frac{\lambda +k}{\lambda }}{C}_k^{\lambda }(t),t\in [-1,1],$$

(22)

which is a polynomial of degree $2n$. According to [19], combined with Proposition 1 and the cubature formula, we have:

$$\begin{array}{cc}\hfill \parallel F_{\tau }-K_n(F_{\tau }){\parallel }_{L_{\infty }({\mathbb{S}}^{d-1})}& =\parallel F_{\tau }-\sum _{l=1}^m{\gamma }_l{L}_n(F_{\tau })(y_l)l_n(\langle y_l,·\rangle ){\parallel }_{L_{\infty }({\mathbb{S}}^{d-1})}\hfill \\ & \le C_{\eta }{2}^{d-1}{n}^{-r}{\parallel F_{\tau }\parallel }_{W_{\infty }^r({\mathbb{S}}^{d-1})},\hfill \end{array}$$

(23)

where

$$L_n(F_{\tau })(y_l)={\int }_{{\mathbb{S}}^{d-1}}{F}_{\tau }(z)l_n(\langle y_l,z\rangle )d\mu (z),$$

and ${\left\{({\gamma }_l,y_l)\right\}}_{l=1}^m$ follows a cubature rule of degree $4n$.

For $N_1\in \mathbb{N}$, let $\mathbf{t}={\left\{t_i\right\}}_{i=1}^{2N_1+3}$ be the uniform mesh on $[-1-{\displaystyle \frac{1}{N_1}},1+{\displaystyle \frac{1}{N_1}}]$ with $t_i=-1+{\displaystyle \frac{i-2}{N_1}}$. Construct a linear operator $I_{\mathbf{t}}$ on $C[-1,1]$ by

$$I_{\mathbf{t}}(l_n)(u)=\sum _{i=2}^{2N_1+2}{l}_n(t_i){\delta }_i(u),u\in [-1,1],l_n\in C[-1,1],$$

(24)

where ${\delta }_i\in C(\mathbb{R})$, $i=2,\dots ,2N_1+2$, is given by

$${\delta }_i(u)=N_1(\sigma (u-t_{i-1})-2\sigma (u-t_i)+\sigma (u-t_{i+1})).$$

(25)

To facilitate the analysis of the number of free parameters, we define a linear operator ${\mathcal{L}}_{N_1}:{\mathbb{R}}^{2N_1+1}\to {\mathbb{R}}^{2N_1+3}$, which acts on $\zeta ={({\zeta }_i)}_{i=1}^{2N_1+1}\in {\mathbb{R}}^{2N_1+1}$ as follows:

$${({\mathcal{L}}_{N_1}(\zeta ))}_i=\left\{\begin{array}{cc}{\zeta }_2,\hfill & \mathrm{for}i=1,\hfill \\ {\zeta }_3-2{\zeta }_2,\hfill & \mathrm{for}i=2,\hfill \\ {\zeta }_{i-1}-2{\zeta }_i+{\zeta }_{i+1},\hfill & \mathrm{for}3\le i\le 2N_1+1,\hfill \\ {\zeta }_{2N_1+1}-2{\zeta }_{2N_1+2},\hfill & \mathrm{for}i=2N_1+2,\hfill \\ {\zeta }_{2N_1+2},\hfill & \mathrm{for}i=2N_1+3.\hfill \end{array}\right.$$

This operator ${\mathcal{L}}_{N_1}$ facilitates the representation of the approximation operator $I_{\mathbf{t}}$ on $C([-1,1])$ in terms of ${\left\{\sigma (·-t_i)\right\}}_{i=1}^{2N_1+3}$ as

$$I_{\mathbf{t}}(l_n)=N_1\sum _{i=1}^{2N_1+3}{\left({\mathcal{L}}_{N_1}\left({\left\{l_n(t_k)\right\}}_{k=2}^{2N_1+2}\right)\right)}_i\sigma (·-t_i).$$

(26)

By [19,24], we have

$$\begin{array}{cc}\hfill & \parallel \sum _{l=1}^m{\gamma }_l{L}_n(F_{\tau })(y_l)l_n(\langle y_l,·\rangle )-\sum _{l=1}^m{\gamma }_l{L}_n(F_{\tau })(y_l)I_{\mathbf{t}}(l_n)(\langle y_l,·\rangle ){\parallel }_{\infty }\hfill \\ \hfill \le & {\displaystyle \frac{C{3}^{d+2}{n}^{d+3}{C}_1{C}_{\eta ,d}}{N_1^2}}{\parallel F_{\tau }\parallel }_{W_{\infty }^r},\hfill \end{array}$$

(27)

where $C>0$ is a constant. Combining this with (23) and (27), we obtain the following Lemma:

Lemma 1. 

Let $d\ge 3$, $r>0$, $n\in \mathbb{N}$, $N_1\in \mathbb{N}$. There exist ${\gamma }_l\in \mathbb{R}$ and $y_l\in {\mathbb{S}}^{d-1}$, $l=1,2,\dots ,m$, such that for any $F_{\tau }\in W_{\infty }^r({\mathbb{S}}^{d-1})$ ($\tau =\{1,2,\dots ,d^{\ast }\}$),

$$\parallel F_{\tau }-\sum _{l=1}^m{\gamma }_l{L}_n(F_{\tau })(y_l)I_{\mathbf{t}}(l_n)(\langle y_l,·\rangle ){\parallel }_{\infty }\le \tilde{c}{\parallel F_{\tau }\parallel }_{W_{\infty }^r({\mathbb{S}}^{d-1})}max\left\{n^{-r},{\displaystyle \frac{n^{d+3}}{N_1^2}}\right\},$$

(28)

where $\tilde{c}=C_{\eta }{2}^{d-1}+C·3^{d+2}{C}_1{C}_{\eta ,d}$.

For approximating the outer function f of the form (20), we use the result of shallow sigmoidal neural networks approximation with scaling [35].

Lemma 2 

([35]). Let $d^{\ast },N\in \mathbb{N}$, $\beta >0$. For any $f\in W_{\infty }^{\beta }({[-B_F,B_F]}^{d^{\ast }})$, there exists ${\left\{{\widehat{\alpha }}_k\right\}}_{k=1}^N\in {\mathbb{R}}^{d^{\ast }}$, ${\left\{{\widehat{b}}_k\right\}}_{k=1}^N\in \mathbb{R}$, ${\left\{{\widehat{c}}_k\right\}}_{k=1}^N\in \mathbb{R}$, and a constant $C_{d^{\ast },\beta }$ depending on $d^{\ast }$ and β such that

$$\underset{z\in {[-B_F,B_F]}^{d^{\ast }}}{sup}|f(z)-\sum _{k=1}^N{\widehat{c}}_k\widehat{\sigma }({\widehat{\alpha }}_k·z-{\widehat{b}}_k)|\le B_F^{\beta }{C}_{d^{\ast },\beta }{\parallel f\parallel }_{W_{\infty }^{\beta }}{N}^{{\displaystyle \frac{-\beta }{d^{\ast }}}},$$

(29)

where $\widehat{\sigma }$ is the activation function of the sigmoidal type.

Proof 

(Proof of Theorem 1).

For the approximate target functions of the form (20), we construct DCNNs based on a series of convolutional layers and a final fully connected layer. The convolutional transformations are represented by matrices ${\{T^{(j)}:=T^{w^{(j)}}\in {\mathbb{R}}^{(d_{j-1}+s)\times d_{j-1}}\}}_{j=1}^{J_2}$, each induced by corresponding filter $\mathbf{w}:={\left\{w^{(j)}\right\}}_{j=1}^{J_2}$ each supported in $\{0,1,\dots ,s\}$. Associated with these layers are bias vectors ${\{b^{(j)}\in {\mathbb{R}}^{d_{j-1}+s}\}}_{j=1}^{J_2}$. The network concludes with a fully connected layer $h^{(J_2+1)}:{\mathbb{R}}^{d_{J_2}}\to {\mathbb{R}}^N$ defined by a connection matrix $F^{(J_2+1)}\in {\mathbb{R}}^{N\times d_{J_2}}$ and a bias vector $b^{(J_2+1)}\in {\mathbb{R}}^N$, expressed as

$$h^{(J_2+1)}(x)=\widehat{\sigma }(F^{(J_2+1)}{h}^{(J_2)}(x)-b^{(J_2+1)}),$$

where $\widehat{\sigma }$ is the activation function of the sigmoidal type. The activation function $\widehat{\sigma }$ that satisfies the assumptions:

$${\widehat{\sigma }}^{(i)}(u)\ne 0,u\in \mathbb{R},i\in {\mathbb{Z}}_{+},$$

and for some integer $q\ne 1$, we have ${lim}_{u\to -\infty }{\displaystyle \frac{\widehat{\sigma }(u)}{{\left|u\right|}^q}}=0$ and ${lim}_{u\to +\infty }{\displaystyle \frac{\widehat{\sigma }(u)}{u^q}}=1$.

The associated hypothesis space $\mathfrak{H}$, used for learning and function approximation, encompasses all output functions defined by the filter sequence $\mathbf{w}={\left\{w^{(j)}\right\}}_{j=1}^{J_2}$, the connection matrix $F^{(J_2+1)}$, bias sequence $\mathbf{b}={\left\{b^{(j)}\right\}}_{j=1}^{J_2+1}$, and a coefficient vector $c\in {\mathbb{R}}^N$, is given by:

$$\mathfrak{H}:=\left\{c^{(J_2+1)}·h^{(J_2+1)}(x):\mathbf{w},\mathbf{b},F^{(J_2+1)},c^{(J_2+1)}\right\}.$$

We design the initial convolutional layer to extract linear features by employing a convolutional factorization approach [8,10], which facilitates the construction of ridge functions for approximating the smooth components ${\left\{F_{\tau }(x)\right\}}_{\tau =1}^{d^{\ast }}$. Let $s\ge 2$ and consider a sequence $U={(U_k)}_{k=-\infty }^{\infty }$ supported on the index set $\{0,\dots ,\mathcal{M}\}$ with $\mathcal{M}\le 0$. Then, there exists a finite sequence of filters ${\left\{w^{(j)}\right\}}_{j=1}^p$, each supported in $\{0,\dots ,s\}$, where $p\le \lceil {\displaystyle \frac{\mathcal{M}}{s-1}}\rceil$, such that the following convolutional factorization is satisfied:

$$U=w^{(p)}\ast w^{(p-1)}\ast \dots \ast w^{(2)}\ast w^{(1)}.$$

(30)

For $m\in \mathbb{N}$ and $\mathbf{y}=\{y_1,\dots ,y_m\}\subset {\mathbb{S}}^{d-1}$, we define U as a sequence supported on $\{0,\dots ,md-1\}$, given by: $U_{(l-1)d+(d-k)-{(y_l)}_k}$ for $l\in \{1,\dots ,m\}$ and $k\in \{1,\dots ,d\}$. Let $\mathcal{M}=md-1$, $p\le ⌈{\displaystyle \frac{\mathcal{M}}{s-1}}⌉$. There exists a sequence of filters $\mathbf{w}={\left\{w^{(j)}\right\}}_{j=1}^{J_1}$ supported on $\{0,\dots ,s\}$ with $J_1\ge ⌈{\displaystyle \frac{\mathcal{M}}{s-1}}⌉$ that satisfies the convolutional factorization:

$$U=w^{(J_1)}\ast w^{(J_1-1)}\ast \dots \ast w^{(2)}\ast w^{(1)}.$$

Here, for $j=p+1,\dots ,J_1$, we take $w^{(j)}$ to be a delta sequence, i.e., $\{1,0,\dots ,0\}$:

$$T^{(J_1)}\dots T^{(2)}{T}^{(1)}=T^{(J_1,1)}={(U_{i-k})}_{i=1,\dots ,d+Js,k=1,\dots ,d}\in {\mathbb{R}}^{(d+J_{1}s)\times d},$$

(31)

where $T^{(j)}$ represents the Toeplitz matrix in (6).

The next step involves constructing the bias vectors in the neural network. We define this as:

$${\parallel w\parallel }_1=\sum _{k=-\infty }^{\infty }\left|w_k\right|,$$

(32)

For the first layer, the bias is set as:

$$b^{(1)}=-{\parallel w\parallel }_1{\mathbf{I}}_{d_0}$$

and

$$b^{(j)}=\left({\mathsf{\Pi }}_{p=1}^{j-1}{\parallel w^{(p)}\parallel }_1\right)T^{(j)}{\mathbf{I}}_{d_{j-1}}-\left({\mathsf{\Pi }}_{p=1}^j{\parallel w^{(p)}\parallel }_1\right){\mathbf{I}}_{d_j}$$

(33)

and for layers $j=2,\dots ,J_1$, the bias vectors are:

$$b_{s+1}^{(j)}=\dots =b_{d_j-s}^{(j)}.$$

(34)

Note that for ${\parallel x\parallel }_{\infty }\le 1$ for $x\in {\mathbb{S}}^{d-1}$. Let ${\parallel h\parallel }_{\infty }=max\{\parallel h_j{\parallel }_{\infty }:j=1,\dots ,q\}$ for a vector of functions $h:{\mathbb{S}}^{d-1}\to {\mathbb{R}}^q$. It is known that for $h:{\mathbb{S}}^{d-1}\to {\mathbb{R}}^{d_{j-1}}$, we have the following inequality:

$$\parallel T^{(j)}{h\parallel }_{\infty }\le \parallel w^{(j)}{\parallel }_1{\parallel h\parallel }_{\infty },$$

(35)

Therefore, the components of $h^{(J_1)}(x)$ satisfy the following:

$${\left(h^{(J_1)}(x)\right)}_{ld}=\langle y_l,x\rangle +B^{(J_1)},l=1,\dots ,m,$$

(36)

where $B^{(J_1)}={\mathsf{\Pi }}_{p=1}^{J_1}{\parallel w^{(p)}\parallel }_1$. We have

$${\mathfrak{D}}_d\left(h^{(J_1)}(x)\right)=\left[\begin{array}{c}\langle y_1,x\rangle \\ ⋮\\ \langle y_m,x\rangle \\ 0\\ ⋮\\ 0\end{array}\right]+B^{(J_1)}{\mathbf{I}}_{\lfloor (d+J_{1}s)/d\rfloor }.$$

(37)

Since $J_1\ge ⌈{\displaystyle \frac{md-1}{s-1}}⌉$, we obtain the following:

$${\displaystyle \frac{d+J_{1}s}{d}}\ge 1+{\displaystyle \frac{md-1}{d}}{\displaystyle \frac{s}{s-1}}>1+{\displaystyle \frac{md-1}{d}}\ge m,$$

(38)

Therefore, we have $d_{J_1}=\lfloor (d+J_{1}s)/d\rfloor \ge m$. To satisfy the constraint $J_1\ge ⌈{\displaystyle \frac{md}{s-1}}⌉$ for the DCNNs, and $m=\lfloor C_d+1\rfloor n^{d-1}$ for the cubature formula, we need to select $n=⌊{\left({\displaystyle \frac{(s-1)J_1}{\lfloor C_d+1\rfloor d}}\right)}^{{\displaystyle \frac{1}{d-1}}}⌋$.

Considering that $|y_l·x|\le 1$, we define re exist two groups of convolutional layers with filters ${\left\{w^{(j)}\right\}}_{j=1}^{J_2}$ of uniform length s and biases ${\left\{b^{(j)}\right\}}_{j=1}^{J_2}$ that satisfy the restriction (7) for $j\ne J_2$. These layers are downsampled in the $J_1$-th layer by a scale factor d, ensuring that the output of the $J_2$-th layer, denoted $h^{(J_2)}(x)\in {\mathbb{R}}^{d_{J_2}}$ is given by:

$${\left(h^{(J_2)}(x)\right)}_{(i-1)d_{J_1}+l}=\left\{\begin{array}{cc}\sigma (y_l·x-t_i)\hfill & \mathrm{if}1\le l\le m,1\le i\le 2N_1+3,\hfill \\ 0\hfill & \mathrm{otherwise},\hfill \end{array}\right.$$

(39)

where $J_1\ge ⌈{\displaystyle \frac{md-1}{s-1}}⌉$, $J_2-J_1\ge ⌈{\displaystyle \frac{(2N_1+3)d_{J_1}}{s-1}}⌉$. So $d_{J_1}=\lfloor (d+J_{1}s)/d\rfloor \ge m$ and $d_{J_2}=d_{J_1}+(J_2-J_1)s>(2N_1+3)d_{J_1}\ge (2N_1+2)d_{J_1}+m$. Therefore, we choose the bias vector $b^{(J_2)}$ as:

$$\begin{array}{cc}\hfill {\left(b^{(J_2)}\right)}_k& =B^{(J_1)}\left({\mathsf{\Pi }}_{p=J_1+1}^{J_2-1}{\parallel w^{(p)}\parallel }_1\right){\left(T^{(J_2)}{\mathbf{I}}_{d_{J_2}-1}\right)}_k\hfill \\ & +\left\{\begin{array}{cc}{t}_i,\hfill & \mathrm{if}(i-1)d_{J_1}+1\le k\le (i-1)d_{J_1}+m,1\le i\le 2N_1+3,\hfill \\ B^{(J_1)},\hfill & \mathrm{otherwise}.\hfill \end{array}\right.\hfill \end{array}$$

(40)

Finally, we can express $h^{(J_2)}(x)$ as:

$$h^{(J_2)}(x)={\left[\begin{array}{c}{H}_1^T{H}_2^T\dots H_{2N_1+3}^{T}0\dots 0\end{array}\right]}^T\in {\mathbb{R}}^{d_{J_2}},$$

(41)

where

$$H_i^T=\left[\begin{array}{c}\sigma (y_1·x-t_i)\sigma (y_2·x-t_i)\dots \sigma (y_m·x-t_i)0\dots 0\end{array}\right]\in {\mathbb{R}}^{d_{J_1}},i=1,2,\dots ,2N_1+3.$$

(42)

We define $v_i={\left({\mathcal{L}}_{N_1}\left({\left\{l_n(t_k)\right\}}_{k=2}^{2N_1+2}\right)\right)}_i$ for $i=1,2,\dots ,2N_1+3$. Additionally, we define the zero matrices: the $m\times (d_{J_1}-m)$ matrix as $\mathbf{O}$, and the $m\times (d_{J_2}-(2N_1+2)d_{J_1}-m)$ matrix as $\widehat{\mathbf{O}}$. Then, we have:

$$F^{(N_1)}=N_1\left[\begin{array}{c}{v}_1{\mathbf{I}}_m\mathbf{O}{v}_2{\mathbf{I}}_m\mathbf{O}\dots v_{2N_1+3}{I}_m\widehat{\mathbf{O}}\end{array}\right]\in {\mathbb{R}}^{m\times d_{J_2}}.$$

(43)

Next, to obtain ${\sum }_{l=1}^m{\gamma }_l{L}_n(F_{\tau })(y_l)I_{\mathbf{t}}(l_n)(\langle y_l,·\rangle )$ ($\tau =1,2,\dots ,d^{\ast }$), by (27), we obtain as follows:

$$F^{(\gamma )}=\left[\begin{array}{c}{\gamma }_1{L}_n(F_1)(y_1){\gamma }_2{L}_n(F_1)(y_2)\dots {\gamma }_m{L}_n(F_1)(y_m)\\ {\gamma }_1{L}_n(F_2)(y_1){\gamma }_2{L}_n(F_2)(y_2)\dots {\gamma }_m{L}_n(F_2)(y_m)\\ ⋮\\ {\gamma }_1{L}_n(F_{d^{\ast }})(y_1){\gamma }_2{L}_n(F_{d^{\ast }})(y_2)\dots {\gamma }_m{L}_n(F_{d^{\ast }})(y_m)\end{array}\right]\in {\mathbb{R}}^{d^{\ast }\times m}.$$

(44)

Define the connection matrix $F^{(\widehat{\alpha })}\in {\mathbb{R}}^{N_2\times d^{\ast }}$ for the fully connected layer as the following:

$$F^{(J_2+1)}=F^{(\widehat{\alpha })}{F}^{(\gamma )}{F}^{(N_1)}=\left[\begin{array}{c}{\widehat{\alpha }}_1\\ ⋮\\ {\widehat{\alpha }}_{N_2}\end{array}\right]F^{(\gamma )}{F}^{(N_1)},$$

(45)

where ${\widehat{\alpha }}_k\in {\mathbb{R}}^{d^{\ast }}$ ($k=1,2,\dots ,N_2$). Next, select the bias vector $b^{(J_2+1)}\in {\mathbb{R}}^{N_2}$ as $b^{(J_2+1)}={[{\widehat{b}}_1,{\widehat{b}}_2,\dots ,{\widehat{b}}_{N_2}]}^T$. Additionally, define the coefficient vector $c^{(J_2+1)}\in {\mathbb{R}}^{N_2}$ as $c^{(J_2+1)}={[{\widehat{c}}_1,{\widehat{c}}_2,\dots ,{\widehat{c}}_{N_2}]}^T$.

We refer to N and s as structural parameters because the architecture of the DCNNs is entirely defined by these two parameters. The remaining parameters are termed training parameters, as they can be trained or selected once s and N are specified.

The error rate for smooth features was established in Lemma 1. By setting $n=N_1^{{\displaystyle \frac{2}{d+3+r}}}$, we obtain the following expression

$$\parallel F_{\tau }-\sum _{l=1}^m{\gamma }_l{L}_n(F_{\tau })(y_l)I_{\mathbf{t}}(l_n)(\langle y_l,·\rangle ){\parallel }_{\infty }=\parallel F_{\tau }-{\widehat{F}}_{\tau }{\parallel }_{\infty }\le \tilde{c}{\parallel F_{\tau }\parallel }_{W_{\infty }^r({\mathbb{S}}^{d-1})}{N}_1^{{\displaystyle \frac{-2r}{d+3+r}}}.$$

(46)

Denote $S(x)=(F_1(x),F_2(x),\dots ,F_{d^{\ast }}(x))$ and $\widehat{S}(x)=({\widehat{F}}_1(x),{\widehat{F}}_2(x),\dots ,{\widehat{F}}_{d^{\ast }}(x))$. We have

$$|f(S(x))-f(\widehat{S}(x))|\le {\parallel f\parallel }_{W_{\infty }^{\beta }}{\left(\tilde{c}\sqrt{d^{\ast }}max{\left\{\parallel F_{\tau }{\parallel }_{W_{\infty }^r({\mathbb{S}}^{d-1})}\right\}}_{\tau =1}^{d^{\ast }}\right)}^{\beta }{N}_1^{{\displaystyle \frac{-2r\beta }{d+3+r}}},$$

(47)

where $\tilde{c}=C_{\eta }{2}^{d-1}+C·3^{d+2}{C}_1{C}_{\eta ,d}$.

Using (46) and Lemma 2, the resulting error rates can be derived as follows:

Taking $N_2=N$ and $N_1=⌈N_2^{{\displaystyle \frac{d+3+r}{2r{d}^{\ast }}}}⌉$, we have

$$\begin{array}{cc}\hfill & |c^{(J_2+1)}·h^{(J_2+1)}(x)-f(S(x))|\hfill \\ \hfill \le & |c^{(J_2+1)}·h^{(J_2+1)}(x)-f(\widehat{S}(x))|+|f(\widehat{S}(x))-f(S(x))|\hfill \\ \hfill \le & B_F^{\beta }{C}_{d^{\ast },\beta }{\parallel f\parallel }_{W_{\infty }^{\beta }}{N}^{{\displaystyle \frac{-\beta }{d^{\ast }}}}+{\parallel f\parallel }_{W_{\infty }^{\beta }}{\left(\tilde{c}\sqrt{d^{\ast }}max{\left\{\parallel F_{\tau }{\parallel }_{W_{\infty }^r({\mathbb{S}}^{d-1})}\right\}}_{\tau =1}^{d^{\ast }}\right)}^{\beta }{N}_1^{{\displaystyle \frac{-2r\beta }{d+3+r}}}\hfill \\ \hfill \le & c_1{\parallel f\parallel }_{W_{\infty }^{\beta }}{N}^{{\displaystyle \frac{-\beta }{d^{\ast }}}},\hfill \end{array}$$

where $c_1=C_{d^{\ast },\beta }{B}_F^{\beta }+{\left(\tilde{c}\sqrt{d^{\ast }}max{\left\{\parallel F_{\tau }{\parallel }_{W_{\infty }^r({\mathbb{S}}^{d-1})}\right\}}_{\tau =1}^{d^{\ast }}\right)}^{\beta }$. □

Remark 1. 

Since $N_1=⌈N^{{\displaystyle \frac{d+3+r}{2r{d}^{\ast }}}}⌉$, exponent part depend on input dimension d. For approximating functions in $W_{\infty }^{\beta }({\mathbb{S}}^{d-1})$, the index of inner function r and the number of features $d^{\ast }$ should satisfy the following condition:

$${\displaystyle \frac{d+3+r}{2r}}\le d^{\ast }<d,$$

such that the curse of dimensionality can be reduced.

We give a comparison with [10] in Table 1. Assume that the index of the outer functions $\beta$ are equal. Our error rate is much faster when the index of the inner function r and the number of features $d^{\ast }$ satisfy ${\displaystyle \frac{d+3+r}{2r}}\le d^{\ast }<d$ ($d\ge 3$).

Theorem 1 concerns the approximation of composite functions in Sobolev spaces on the unit sphere $f^{\ast }\in W_{\infty }^{\beta }({\mathbb{S}}^{d-1})$$(\beta \in (0,1])$. In contrast, related previous studies—such as the approximation of composite functions in $W_{\infty }^{\beta }({[0,1]}^d)$$(\beta \in (0,1])$— have been discussed in [20]. When $d^{\ast }=d$ and $\beta =1$, meaning that there are d general smmoth features on sphere, corresponding to that of approximating functions in d-dimension cube [11], the following corollary is obtained. It illustrates that the DCNNs we constructed can also approximate or represent functions in a cube.

Corollary 1. 

Let $2\le s\le d$, $d\ge 3$, $N\in \mathbb{N}$. For approximating functions $f\in W_{\infty }^1({[0,1]}^d)$, there exists a downsampled DCNNs as defined in Definition 1 (explicitly constructed as in Section 3.1), with downsampling scaling parameters $\{d,1\}$ at layers $\{J_1,J_2\}$, respectively, and filters ${\left\{w^{(j)}\right\}}_{j=1}^{J_2}$ of uniform length s, bias vectors ${\left\{b^{(j)}\right\}}_{j=1}^{J_2+1}$, connection matrix $F^{(J_2+1)}$, and coefficient vector $c^{(J_2+1)}\in {\mathbb{R}}^N$, such that

$$\parallel c^{(J_2+1)}·h^{(J_2+1)}-f{\parallel }_{L_{\infty }({[0,1]}^d)}\le c_1^{\prime }{\parallel f\parallel }_{W_{\infty }^{\beta }}{N}^{{\displaystyle \frac{-1}{d}}}$$

where $c_1^{\prime }=c_1^{\prime }(d,\eta )$, and $\eta \in C^{\infty }([0,\infty ])$ defined in Section 2.3.

3.2. Approximating Functions with Polynomial Features

The approximation of functions using polynomial features is also discussed in [20], where the domain is considered as ${[0,1]}^d$. In Theorem 1, we derived the error bounds for functions with smooth features defined on the unit sphere and introduced the reproducing kernel ${\displaystyle \frac{k+\lambda }{\lambda }}{C}_k^{\lambda }(\langle x,y\rangle )$ associated with the space ${\mathcal{H}}_k^d$ in Section 2.2.

To reduce the curse of dimensionality, we leverage the reproducing kernel to approximate functions with polynomial features on the unit sphere. Specifically, we consider the function:

$$f^{\ast }(x)=f(P_{1,q_1}(x),P_{2,q_2}(x),\dots ,P_{d^{\ast },q_{d^{\ast }}}(x))=f(P(x)),x\in {\mathbb{S}}^{d-1}$$

(48)

where ${\left\{P_{\tau ,q_{\tau }}(x)\right\}}_{\tau =1}^{d^{\ast }}$ denotes spherical polynomials of degree $q_{\tau }$.

Define $B_P=max\{\parallel P_{\tau ,q_{\tau }}{{\parallel }_{\infty }\}}_{\tau =1}^{d^{\ast }}$, assume further that $f\in W_{\infty }^{\beta }({[-B_P,B_P]}^{d^{\ast }})$ ($\beta \in (0,1]$). Recall the polynomial $k_n(t)$ introduced in Section 2.3., suppose $max{\left\{q_{\tau }\right\}}_{\tau =1}^{d^{\ast }}\le 2n$, we have the following Theorem:

Theorem 2. 

Let $2\le s\le d$, $d\ge 3$, $d^{\ast }\le d$, $N\in \mathbb{N}$, $n\in \mathbb{N}$. Let ${\left\{P_{\tau ,q_{\tau }}(x)\right\}}_{\tau =1}^{d^{\ast }}$ be spherical polynomials of degree $q_{\tau }$, satisfying $max{\left\{q_{\tau }\right\}}_{\tau =1}^{d^{\ast }}\le 2n$, $B_P=max\{\parallel P_{\tau ,q_{\tau }}{{\parallel }_{\infty }\}}_{\tau =1}^{d^{\ast }}$ and suppose $f\in W_{\infty }^{\beta }({[-B_P,B_P]}^{d^{\ast }})$ with $\beta \in (0,1]$. For approximating functions of the form (47), there exists a downsampled DCNNs, as defined in Definition 1 (explicitly constructed in Section 3.1), with downsampling scaling parameters $\{d,1\}$ applied at layers $\{J_1,J_2\}$, respectively. The network uses filters ${\left\{w^{(j)}\right\}}_{j=1}^{J_2}$ of uniform length s, bias vectors ${\left\{b^{(j)}\right\}}_{j=1}^{J_2+1}$, a connection matrix $F^{(J_2+1)}$, and a coefficient vector $c^{(J_2+1)}\in {\mathbb{R}}^N$, such that

$$\parallel c^{(J_2+1)}·h^{(J_2+1)}-f^{\ast }{\parallel }_{L_{\infty }({\mathbb{S}}^{d-1})}\le c_2{\parallel f\parallel }_{W_{\infty }^{\beta }}·n^{(d+3)\beta }·N^{{\displaystyle \frac{-\beta }{d^{\ast }}}}$$

where $c_2=c_2(d^{\ast },d,\beta ,B_P,n,\eta )$, and $\eta \in C^{\infty }([0,\infty ])$ defined in Section 2.3.

Proof. 

Let $P_{\tau ,q_{\tau }}(x)$ be a spherical polynomial of degree $q_{\tau }$, where $q_{\tau }\le 2n$. According to (15), we have:

$$P_{\tau ,q_{\tau }}(x)={\int }_{{\mathbb{S}}^{d-1}}{P}_{\tau ,q_{\tau }}(y)k_n(\langle x,y\rangle )d\mu (y).$$

Applying (18) and (19), and setting $m=\lfloor C_d+1\rfloor n^{d-1}$. Since $P_{\tau ,q_{\tau }}(y)k_n(\langle x,y\rangle )$ is a spherical polynomial in y of degree at most $4n$, there exist nodes $y_1,y_2,\dots ,y_m\in {\mathbb{S}}^{d-1}$ and positive weights ${\gamma }_1,{\gamma }_2,\dots ,{\gamma }_m>0$ such that

$$P_{\tau ,q_{\tau }}(x)=\sum _{l=1}^m{\lambda }_l{P}_{\tau ,q_{\tau }}(y_l)k_n(\langle x,y_l\rangle ),x\in {\mathbb{S}}^{d-1}.$$

Similar to the proof of Theorem 1, we can derive the following error rates by selecting $N_1=N_2=N$:

$$\begin{array}{cc}\hfill & |c^{(J_2+1)}·h^{(J_2+1)}(x)-f(P(x))|\hfill \\ \hfill \le & |c^{(J_2+1)}·h^{(J_2+1)}(x)-f(\widehat{P}(x))|+|f(\widehat{P}(x))-f(P(x))|\hfill \\ \hfill \le & B_P^{\beta }{C}_{d^{\ast },\beta }{\parallel f\parallel }_{W_{\infty }^{\beta }}{N}_2^{{\displaystyle \frac{-\beta }{d^{\ast }}}}+{\parallel f\parallel }_{W_{\infty }^{\beta }}{\left(c^{\prime }{B}_P\sqrt{d^{\ast }}\right)}^{\beta }{n}^{(d+3)\beta }{N}_1^{-2\beta }\hfill \\ \hfill \le & c_2{\parallel f\parallel }_{W_{\infty }^{\beta }}{n}^{(d+3)\beta }{N}^{{\displaystyle \frac{-\beta }{d^{\ast }}}},\hfill \end{array}$$

where $c_2=B_P^{\beta }{C}_{d^{\ast },\beta }+{\left(c^{\prime }{B}_P\sqrt{d^{\ast }}\right)}^{\beta }$, and $c^{\prime }=C·3^{d+2}{C}_1{C}_{\eta ,d}$.

□

Remark 2. 

Extrcting sphrical polynomial features can also derive the rates of approximating functions in $W_{\infty }^1({[0,1]}^d)$ when taking $d^{\ast }=d$ and $\beta =1$.

Remark 3. 

Compared with [20], our spherical polynomial features are not restricted to the same degree, require only less or equal to $2n$.

We provide a comparison with [10] in Table 2. Clearly, for same index $\beta$, our results provide faster error rates since $d^{\ast }$ can be much smaller than d$(d\ge 3)$. Unlike in the case of extracting general smooth features (Theorem 1), the index of the inner function r and the number of features $d^{\ast }$ do not need to be limited to a specific range.

3.3. Approximating Functions with Symmetric Polynomial Features

Even though $d^{\ast }$ much smaller than d, in practical applications of machine learning $d^{\ast }$ is large. In order to further improve the rates of approximation, we consider the polynomial features ${\left\{P_{\tau ,q_{\tau }}(x)\right\}}_{\tau =1}^{d^{\ast }}$ in (47), which exhibits symmetric structures with degree n. We denote these features by ${\left\{Q_{\tau ,n}(x)\right\}}_{\tau =1}^{d^{\ast }}$, where $Q_{\tau ,n}(x)=Q_{\tau ,n}(x_1,x_2,\dots ,x_d)$ and $x\in {\mathbb{S}}^{d-1}$. For any permutation in the symmetric group $Q_{\tau ,n}$:

$$Q_{\tau ,n}(x_1,\dots ,x_i,\dots ,x_j,\dots ,x_d)=Q_{\tau ,n}(x_1,\dots ,x_j,\dots ,x_i,\dots ,x_d),$$

(49)

Therefore, the target function is of the form

$$f^{\ast }(x)=f(Q_{1,n}(x),Q_{2,n}(x),\dots ,Q_{d^{\ast },n}(x)),x\in {\mathbb{S}}^{d-1},$$

(50)

and $f\in W_{\infty }^{\beta }({[-B_Q,B_Q]}^{d^{\ast }})$ ($\beta \in (0,1]$) with $B_Q=max\{\parallel Q_{\tau ,n}{{\parallel }_{\infty }\}}_{\tau =1}^{d^{\ast }}$.

Theorem 3. 

Let $2\le s\le d$, $d\ge 3$, $n<d^{\ast }\le d$, $N\in \mathbb{N}$, $n\in \mathbb{N}$, ${\left\{Q_{\tau ,n}(x)\right\}}_{\tau =1}^{d^{\ast }}$ be a symmetric spherical polynomial of degree n, and $f\in W_{\infty }^{\beta }({[-B_Q,B_Q]}^{d^{\ast }})$ with $B_Q=max\{\parallel Q_{\tau ,n}{{\parallel }_{\infty }\}}_{\tau =1}^{d^{\ast }}$ and $\beta \in (0,1]$. To approximate functions of the form (49), one can construct a downsampled DCNNs as described in Definition 1 (explicitly defined in Section 3.1), employing downsampling scaling parameters $\{d,1\}$ at layers $\{J_1,J_2\}$, respectively. The network uses filters ${\left\{w^{(j)}\right\}}_{j=1}^{J_2}$ of uniform length s, bias vectors ${\left\{b^{(j)}\right\}}_{j=1}^{J_2+1}$, a connection matrix $F^{(J_2+1)}$, a coefficient vector $c^{(J_2+1)}\in {\mathbb{R}}^N$, such that

$$\parallel c^{(J_2+1)}·h^{(J_2+1)}-f^{\ast }{\parallel }_{L_{\infty }({\mathbb{S}}^{d-1})}\le c_3{n}^{(d+3)\beta }{N}^{{\displaystyle \frac{-\beta }{n}}}$$

where $c_3=c_3(d^{\ast },d,\beta ,f,Q_{\tau ,n},n,\eta )$, and $\eta \in C^{\infty }([0,\infty ])$ defined in Section 2.3.

Proof. 

Let the elementary symmetric polynomials on $x\in {\mathbb{S}}^{d-1}$ be denoted by

$$q_k(x)=\sum _{1\le i_1<i_2<\dots <i_k\le d}{x}_{i_1}{x}_{i_2}\dots x_{i_k},k=1,2,\dots ,d.$$

(51)

According to the fundamental theorem of symmetric polynomials, any symmetric polynomial can be uniquely expressed as a polynomial in the variables $(q_1(x),q_2(x),\dots ,q_d(x))$. Given that the symmetric polynomials ${\left\{Q_{\tau ,n}(x)\right\}}_{\tau =1}^{d^{\ast }}$ are of degree n, and $n\le d$, they have a unique expression as a polynomial in $(q_1(x),q_2(x),\dots ,q_n(x))$. Consequently, for $Q_{\tau ,n}$ with $n\le d^{\ast }\le d$, there exists a unique polynomial ${\tilde{Q}}_{\tau ,n}$ such that

$$Q_{\tau ,n}(x)={\tilde{Q}}_{\tau ,n}(q_1(x),q_2(x),\dots ,q_n(x)),\tau =1,2,\dots d^{\ast }.$$

(52)

Therefore, the target function $f^{\ast }(x)$ can expressed as

$$\begin{array}{cc}\hfill f^{\ast }(x)=& f(Q_{1,n}(x),Q_{2,n}(x),\dots ,Q_{d^{\ast },n}(x))\hfill \\ \hfill =& f({\tilde{Q}}_{1,n}(q_1(x),q_2(x),\dots ,q_n(x)),\dots ,{\tilde{Q}}_{d^{\ast },n}(q_1(x),q_2(x),\dots ,q_n(x)))\hfill \\ \hfill =& \tilde{f}(q_1(x),q_2(x),\dots ,q_n(x)),\hfill \end{array}$$

(53)

for some function $\tilde{f}$ defined on ${\mathbb{R}}^n$ as

$$\tilde{f}(z)=f({\tilde{Q}}_{1,n}(z),\dots ,{\tilde{Q}}_{d^{\ast },n}(z)),z\in {\mathbb{R}}^n.$$

(54)

Since the polynomial ${\tilde{Q}}_{\tau ,n}\in W_{\infty }^1({[-B_q,B_q]}^n)$ with semi-norm $|{\tilde{Q}}_{\tau ,n}{|}_{W_{\infty }^1}$, and where $B_q={max}_k\{\parallel q_k{{\parallel }_{\infty }\}}_{k=1}^n$, it follows that for any $z_1\ne z_2\in {[-B_q,B_q]}^n$, we have:

$$\begin{array}{cc}\hfill \tilde{f}(z_1)-\tilde{f}(z_2)=& f({\tilde{Q}}_{1,n}(z_1),\dots ,{\tilde{Q}}_{d^{\ast },n}(z_1))-f({\tilde{Q}}_{1,n}(z_2),\dots ,{\tilde{Q}}_{d^{\ast },n}(z_2))\hfill \\ \hfill \le & {\left(\sqrt{d^{\ast }}\underset{\tau }{max}\left\{\right|{\tilde{Q}}_{\tau ,n}(z_1)-{\tilde{Q}}_{\tau ,n}(z_2){\left|\right\}}_{\tau =1}^{d^{\ast }}\right)}^{\beta }\hfill \\ \hfill \le & {\left(\sqrt{d^{\ast }}\underset{\tau }{max}\left\{\right|{\tilde{Q}}_{\tau }{|}_{W_{\infty }^1}{\}}_{\tau =1}^{d^{\ast }}|z_1-z_2|\right)}^{\beta }.\hfill \end{array}$$

(55)

Since ${\left\{q_k(x)\right\}}_{k=1}^n$ are spherical polynomials of degree up to n, let us denote

$$q(x)=(q_1(x),q_2(x),\dots ,q_n(x))and\widehat{q}(x)=({\widehat{q}}_1(x),{\widehat{q}}_2(x),\dots ,{\widehat{q}}_n(x)).$$

Similar to the proof of Theorem 2, by selecting $N_1=N_2=N$, we obtain:

$$\begin{array}{cc}\hfill & |c^{(J_2+1)}·h^{(J_2+1)}(x)-\tilde{f}(q(x))|\hfill \\ \hfill \le & |c^{(J_2+1)}·h^{(J_2+1)}(x)-\tilde{f}(\widehat{q}(x))|+|\tilde{f}(\widehat{q}(x))-\tilde{f}(q(x))|\hfill \\ \hfill \le & B_q^{\beta }{C}_{d^{\ast },\beta }{\parallel \tilde{f}\parallel }_{W_{\infty }^{\beta }}{N}_2^{{\displaystyle \frac{-\beta }{n}}}+{\left(B_q\underset{\tau }{max}\left\{\right|{\tilde{Q}}_{\tau }{{|}_{W_{\infty }^1}\}}_{\tau =1}^{d^{\ast }}\sqrt{d^{\ast }}\right)}^{\beta }{(c^{\prime }{n}^{d+3})}^{\beta }{N}_1^{-2\beta }\hfill \\ \hfill \le & B_q^{\beta }{C}_{d^{\ast },\beta }\left({\left(B_q\underset{\tau }{max}\left\{\right|{\tilde{Q}}_{\tau }{{|}_{W_{\infty }^1}\}}_{\tau =1}^{d^{\ast }}\sqrt{d^{\ast }}\right)}^{\beta }+{\parallel f\parallel }_{W_{\infty }^{\beta }}\right)N_2^{{\displaystyle \frac{-\beta }{n}}}\hfill \\ \hfill +& {\left(B_q\underset{\tau }{max}\left\{\right|{\tilde{Q}}_{\tau }{{|}_{W_{\infty }^1}\}}_{\tau =1}^{d^{\ast }}\sqrt{d^{\ast }}\right)}^{\beta }{(c^{\prime }{n}^{d+3})}^{\beta }{N}_1^{-2\beta },\hfill \\ \hfill \le & c_3{n}^{(d+3)\beta }{N}^{{\displaystyle \frac{-\beta }{n}}},\hfill \end{array}$$

(56)

where $c_3={\left(B_q{max}_{\tau }\left\{\right|{\tilde{Q}}_{\tau }{{|}_{W_{\infty }^1}\}}_{\tau =1}^{d^{\ast }}\sqrt{d^{\ast }}\right)}^{\beta }(c^{\prime \beta }+B_q^{\beta }{C}_{d^{\ast },\beta })+B_q^{\beta }{C}_{d^{\ast },\beta }{\parallel f\parallel }_{W_{\infty }^{\beta }}$, and $c^{\prime }=C·3^{d+2}{C}_1{C}_{\eta ,d}$.

□

4. Conclusions

This paper proposes the construction of DCNNs with multiple downsampled layers for approximating functions with multi-features—namely, general smooth features, polynomial features and symmetric polynomial features on the unit sphere. The result of Theorem 1 implies that if the index of the inner function r and the number of features $d^{\ast }$ satisfy ${\displaystyle \frac{d+3+r}{2r}}\le d^{\ast }<d$, the curse of dimensionality can be reduced. For further reducing the curse of dimensionality, we also derive error rates for cases where the features are either polynomials or symmetric polynomials on the unit sphere, as shown in Theorem 2 and 3. This implies that the DCNNs developed in Section 3.2 demonstrate the capability to extract three types of features: general smooth features, polynomial features, and symmetric polynomial features. In comparison, the DCNNs in [20] are limited to extracting only polynomial and symmetric polynomial features. In addition, compare to [20], when extracting polynomial features on sphere, our spherical polynomial features are not restricted to the same degree. Although our input vector x is spherical, we show that the DCNNs designed in Section 3.2 can also approximate or represent functions in $W_{\infty }^1({[0,1]}^d)$ by extrcating general smooth or spherical polynomial features, as shown in Corollary 1. These results underscore the superior learning capacity of the proposed DCNNs framework in capturing complex and high-dimensional feature representations on the unit sphere.