Optical Fringe Projection Driving 3D Metrology Uncomplicated

1. Introduction

Optical fringe projection is an essential technology driving important applications in science, engineering, medicine, entertainment, and many other fields [1,2,3]. This technology, also known as fringe projection profilometry, is a contactless metrological tool suitable for accurate and high-resolution 3D object reconstruction [4,5]. Moreover, modern digital cameras, projectors, and computers enable fringe projection systems to operate at high rates [6,7,8], allowing the sensing of fast shape changes in dynamic phenomena [9,10]. The development of portable profilometers [11] has expanded the usefulness of fringe projection systems for in-situ studies in archaeological [12] and forensic applications [13].

Despite progress in hardware and theoretical models, fringe projection profilometry is an active research area within the scientific community worldwide [14,15]. Researchers are working to overcome new challenges in fringe projection technology [16]. Current investigations include miniaturization of electronic devices [17], multimodal imaging [18,19], self-calibration [20,21], setup optimization for full-view object reconstruction [22,23,24], dynamic range improvement for reconstructing reflective objects [25,26], and enabling automatic and adaptive operation [27,28,29]. For this reason, the specialization of students and professionals in this research area is crucial.

Inexperienced readers of fringe projection profilometry may feel overwhelmed by the abundance of specialized literature available today [30,31]. This technology encompasses various topics, from phase demodulation to triangulation and calibration, with several books and scientific articles dedicated to each subject [32,33]. Frustrating months of literature review could pass without acquiring enough background to construct and operate a profilometer. This paper presents essential concepts and helpful insights into the working principles of optical fringe projection. Elementary models for setting up and operating a 3D profilometer are provided. Illustrative applications where this technology has been implemented are described. This study offers a concise guide through the fundamentals of fringe projection profilometry, making this technology an uncomplicated tool for driving 3D metrology toward new frontiers.

2. Theoretical Preliminaries

2.1. Camera Imaging

A digital camera is a sophisticated device consisting mainly of a compound lens and a photosensitive sensor [34]. Cameras are designed to collect light rays traveling from the 3D space and record their intensities. The resultant intensity map is the camera output, known as the image. Although the physical imaging process is a complicated phenomenon, it can be modeled with a good approximation using the pinhole model [35,36]. Let $\mathit{p}$ be the vector of a point in the 3D space, and let $\mathit{\mu }$ be the pixel where $\mathit{p}$ was registered by the camera, as shown in Figure 1(a). Considering the pinhole model, the vectors $\mathit{p}$ and $\mathit{\mu }$ are related as

$$\mathit{\mu }={\mathcal{H}}^{-1}\left[C\mathcal{H}\left[\mathit{p}\right]\right],$$

(1)

where $\mathcal{H}$ is the homogeneous coordinate operator [37], ${[·]}^{-1}$ denotes the inverse, and

$$C=K[R^T,-R^T\mathit{t}]$$

(2)

is a $3\times 4$ matrix, known as the camera matrix, K is the intrinsic parameter matrix (non-singular upper triangular), R is a rotation matrix defining the camera orientation, ${[·]}^T$ denotes the transpose, and $\mathit{t}$ is a translation vector specifying the camera position.

2.2. Cameras as Direction Sensors

Although digital cameras are well-known for their ability to capture photographs, optical fringe projection systems employ cameras for an additional purpose. Note that a digital image is an array of pixels, and each pixel detects a light ray coming from the 3D space in a specific direction, as shown in Figure 1(b). Thus, every pixel is associated with a particular light ray that has a unique direction. In this context, a helpful insight is that cameras are direction sensors [38]. A more formal analysis is performed by reversing the imaging process to return a space point $\mathit{p}$ when the pixel $\mathit{\mu }$ is given. For this, Eqs. (1) and (2) can be rewritten as

$$\lambda \mathcal{H}\left[\mathit{\mu }\right]=K[R^T,-R^T\mathit{t}]\left[\begin{array}{c}\mathit{p}\\ 1\end{array}\right],$$

(3)

where $\lambda \ne 0$ is an arbitrary real-valued variable. After a few algebraic manipulations of Eq. (3), the following equation is reached:

$$\mathit{p}=\mathit{t}+\lambda \mathit{d},$$

(4)

which describes a line in the 3D space passing through the camera position $\mathit{t}$, with direction determined by the pixel $\mathit{\mu }$ as

$$\mathit{d}=R{K}^{-1}\mathcal{H}\left[\mathit{\mu }\right].$$

(5)

2.3. Stereo Camera Systems and Triangulation

How can direction sensors be employed for 3D metrology? Consider two cameras forming a stereo system capturing an object from two different viewpoints, as shown in Figure 2(a). Let us assume the parameters of both cameras are known, say $K_1$, $R_1$, and ${\mathit{t}}_1$ for the first camera and $K_2$, $R_2$, and ${\mathit{t}}_2$ for the second camera. Let $\mathit{p}$ be a point on the object surface captured by the two cameras at the pixels ${\mathit{\mu }}_1$ and ${\mathit{\mu }}_2$, respectively. The lines representing the captured light rays can be reproduced using the available camera parameters and the given pixel points, as shown in Figure 2(b), namely

$$\begin{array}{cc}\hfill \mathit{p}& ={\mathit{t}}_1+{\lambda }_1{\mathit{d}}_1,(\mathrm{Line}-1),\hfill \end{array}$$

(6)

$$\begin{array}{cc}\hfill \mathit{p}& ={\mathit{t}}_2+{\lambda }_2{\mathit{d}}_2,(\mathrm{Line}-2).\hfill \end{array}$$

(7)

This reasoning leads to computing the captured point $\mathit{p}$ as the intersection of line-1 and line-2. This computation is generically known as triangulation, although more general constructions for determining points are comprised, such as the intersection of multiple lines, planes, and other geometrical objects, even in combinations [39].

Figure 2. (a) Two cameras capturing a point $\mathit{p}$ from different viewpoints. (b) The captured point is determined as the intersection of the lines defined by ${\mathit{\mu }}_1$ and ${\mathit{\mu }}_2$. (c) Experimental noise may cause skewed lines; therefore, $\mathit{p}$ is determined as the mean point between the solution points ${\mathit{p}}_1$ and ${\mathit{p}}_2$.

Figure 2. (a) Two cameras capturing a point $\mathit{p}$ from different viewpoints. (b) The captured point is determined as the intersection of the lines defined by ${\mathit{\mu }}_1$ and ${\mathit{\mu }}_2$. (c) Experimental noise may cause skewed lines; therefore, $\mathit{p}$ is determined as the mean point between the solution points ${\mathit{p}}_1$ and ${\mathit{p}}_2$.

In particular, triangulation in a stereo system can be performed as follows. Since $\mathit{p}$ represents a common point of the intersecting lines, then Eqs. (6) and (7) can be equated as ${\mathit{t}}_1+{\lambda }_1{\mathit{d}}_1={\mathit{t}}_2+{\lambda }_2{\mathit{d}}_2$, which can be solved for the unknowns ${\lambda }_1$ and ${\lambda }_2$ by the least squares method as

$$\left[\begin{array}{c}{\lambda }_1\\ {\lambda }_2\end{array}\right]={\left(D^{T}D\right)}^{-1}{D}^T({\mathit{t}}_2-{\mathit{t}}_1),$$

(8)

where $D=[{\mathit{d}}_1,-{\mathit{d}}_2]$ is the regression matrix. The computed values of ${\lambda }_1$ and ${\lambda }_2$ allow determining the vector of the captured point as ${\mathit{p}}_1$ using Eq. (6) or ${\mathit{p}}_2$ using Eq. (7). Ideally, ${\mathit{p}}_1$ and ${\mathit{p}}_2$ are equal due to the intersection assumption. Nevertheless, slight deviations caused by experimental errors may result in skewed lines, as shown in Figure 2(c). For this reason, the vector $\mathit{p}$ is defined as the average $({\mathit{p}}_1+{\mathit{p}}_2)/2$; i.e.,

$$\mathit{p}={\displaystyle \frac{1}{2}}({\mathit{t}}_1+{\mathit{t}}_2)+{\displaystyle \frac{1}{2}}\left[\begin{array}{cc}{\mathit{d}}_1,& {\mathit{d}}_2\end{array}\right]\left[\begin{array}{c}{\lambda }_1\\ {\lambda }_2\end{array}\right].$$

(9)

2.4. The Corresponding Point Problem

It is noteworthy that a 3D reconstruction requires the pair of pixels ${\mathit{\mu }}_1$ and ${\mathit{\mu }}_2$ where the captured point $\mathit{p}$ was imaged, see Figure 2. The pixels ${\mathit{\mu }}_1$ and ${\mathit{\mu }}_2$, related by a common point $\mathit{p}$, are known as a corresponding point, denoted by

$${\mathit{\mu }}_1\leftrightarrow {\mathit{\mu }}_2.$$

(10)

Obtaining corresponding points is a challenging task because it depends on the object’s texture [40,41]. For example, no correspondence points can be established from a white object on a white background under homogeneous illumination because of the lack of feature points [42]. Even for objects with abundant texture, such as the human face in biometric applications, the resolution is low because more than one pixel is required to detect a feature, and not all image regions contain reliable features. As a result, stereo camera systems tend to produce low-resolution 3D reconstructions, and their accuracy depends on the texture of the object under study.

2.5. Equivalence Between Cameras and Projectors

Physically, the difference between a camera and a projector is that the light rays propagate in opposite directions, as shown in Figure 3. While a camera captures light rays traveling from space to its image pixels, a projector emits light rays from its slide pixels to space. However, if the sign of the direction vector of the light rays is omitted, cameras and projectors are identical. For this reason, cameras and projectors are mathematically equivalent, and both can be described by the pinhole model given in Eq. (1). Therefore, in addition to cameras being direction sensors, another valuable insight is that projectors are direction-controlled ray generators.

2.6. Camera-Projector Systems

The equivalence between cameras and projectors can be exploited to modify a stereo camera system by replacing one of the cameras with a projector. The resultant camera-projector system has the advantage of not having the corresponding point problem. This advantage is inferred from the following simplified description of the operation of a camera-projector system.

A computer-generated slide controls the brightness of every pixel of the projector. A black slide will turn off all the projector pixels. Suppose this slide is modified by setting the pixel ${\mathit{\mu }}_2$ to white; then, a light ray will be emitted to the 3D space, illuminating the object at point $\mathit{p}$, as shown in Figure 4. In the absence of additional light sources, the camera will capture a dark image except for a bright point at the image pixel ${\mathit{\mu }}_1$. In this way, the corresponding point ${\mathit{\mu }}_1\leftrightarrow {\mathit{\mu }}_2$ is known by simply reading the coordinates of the bright image pixel ${\mathit{\mu }}_1$ and taking the coordinates of the white slide pixel ${\mathit{\mu }}_2$. Therefore, sophisticated algorithms to search corresponding points are unnecessary. For this reason, camera-projector systems can reconstruct even objects without texture, achieving high accuracy and resolution.

Although illustrative, the described working principle is impractical because one image is required to obtain only one corresponding point. Since modern digital projectors have millions of pixels, millions of images will be required for a single 3D reconstruction. Fortunately, efficient illumination techniques have been proposed to significantly reduce the number of required images.

2.7. Structured Illumination

The primary advantage of camera-projector systems is the absence of the corresponding point problem. Instead, slides are designed to “mark” the projector pixels such that they are recognized by the camera and paired with the image pixels, producing corresponding points. The different techniques for marking and recognizing projector pixels are known generically as structured illumination [43].

Typical structured illumination techniques include projecting dots, stripes, grids, codewords, rainbows, and fringes. These techniques can even be combined to produce hybrid structured illumination techniques. Each technique has different advantages and disadvantages regarding accuracy, resolution, number of images, noise robustness, object color sensitivity, and other criteria. Depending on the application, one technique will be more appropriate. In particular, the fringe projection technique is recommended for applications requiring higher accuracy and resolution.

2.8. Fringe Projection

The “mark-based” approach helps to understand the different structured illumination techniques intuitively. Alternatively, a powerful insight is gained by considering a camera-projector setup as a telecommunication system. Remember that the projector slide coordinates $(u,v)$ must be registered on the camera image plane $(r,s)$ to produce corresponding points. In this context, the projector would be considered a transmitter emitting the signals u and v, while the camera is a receiver detecting u and v to produce corresponding points.

The projector slide coordinates can be transmitted using phase modulation. For example, let us encode the values of u as the phase of a 2D cosine signal, known as grating, of the form

$$G_k(u,v)={\displaystyle \frac{1}{2}}+{\displaystyle \frac{1}{2}}cos(\omega u+{\delta }_k),\mathrm{for}k=1,2,\cdots ,n,$$

(11)

where $\omega$ is the grating spatial frequency, ${\delta }_k$ is a reference phase known as phase shift or grating displacement, and n is the number of images required to perform phase demodulation. Figure 5 shows two gratings, with low and high frequencies, respectively, and four phase shifts. When the grating $G_k(u,v)$ is used as a slide, the object under study is illuminated with fringes, as shown in Figure 6. Consequently, this particular structured illumination technique is known as fringe projection. The image captured by the camera is known as fringe pattern, modeled as

$$I_k(r,s)=a(r,s)+b(r,s)cos(\varphi (r,s)+{\delta }_k),$$

(12)

where $a(r,s)$ is the background light, $b(r,s)$ is the fringe amplitude, and $\varphi (r,s)$ is the phase containing the encoded signal u. Indeed, comparing the arguments of the cosine functions in the Eqs. (11) and (12), the slide projector coordinate u is read in the camera image plane from the demodulated phase as

$$u(r,s)={\displaystyle \frac{1}{\omega }}\varphi (r,s).$$

(13)

The fringe projection process can be repeated for the projector axis v. Namely, assuming the gratings were created using the angular frequency w and the recovered phase was $\phi (r,s)$, then the slide projector coordinate v is available in the camera image plane as

$$v(r,s)={\displaystyle \frac{1}{w}}\phi (r,s).$$

(14)

As a result, for every camera image pixel $(r,s)$, the corresponding projector slide coordinates $\left(u\right(r,s),v(r,s\left)\right)$ are determined.

Figure 5. (a)-(d) A grating $G_k(u,v)$ with the frequency $\omega =\pi$ (one fringe) and four phase shifts. (e)-(h) A grating with a higher frequency, $\omega =3.7\pi$, and four phase shifts.

Figure 5. (a)-(d) A grating $G_k(u,v)$ with the frequency $\omega =\pi$ (one fringe) and four phase shifts. (e)-(h) A grating with a higher frequency, $\omega =3.7\pi$, and four phase shifts.

Figure 6. Fringe patterns $I_k(r,s)$ obtained by displaying the gratings shown in Figure 5 on a 3D object.

Figure 6. Fringe patterns $I_k(r,s)$ obtained by displaying the gratings shown in Figure 5 on a 3D object.

The phase retrieval algorithms used in fringe projection systems are inherited from optical metrology. Some adaptations are included considering the convenience of using digital camera-projector systems. For instance, the frequency of the gratings and the phase shifts are controlled precisely by the computer. Subsection 3 presents a phase retrieval algorithm suitable for fringe projection 3D metrology systems.

2.9. Phase and object profile misconception

It is worth remarking on the frequent confusion occurring when the phase extracted from a fringe projection experiment is plotted, as shown in Figure 7. Since the phase looks like the object profile, inexperienced practitioners may conclude that elementary transformations on the phase, such as rotation and scaling, are sufficient to achieve metric 3D object reconstruction. Unfortunately, this misconception leads to the formulation of a transformation that, in addition to being excessively complicated, is unnecessary. It is important to remember that the phase simply provides the projector slide coordinates required to establish corresponding points. Subsequently, the corresponding points are used to perform object reconstruction by triangulation.

3. Phase Demodulation Fringe Pattern Processing

The optical metrology community has developed a wide variety of fringe pattern processing methods for different applications and requirements [44,45]. For instance, Fourier fringe analysis allows phase recovery from a single fringe pattern [46], but the intrinsic spectrum filtering limits the spatial resolution. On the other hand, the phase-shifting method achieves the highest (pixel-wise) spatial resolution [47], but multiple fringe patterns with prefixed phase shifts are required.

Nowadays, digital computer-controlled cameras and projectors allow the capture of multiple fringe patterns at high speed and precise control of the grating frequency and phase shift [48]. For this reason, phase demodulation by phase-shifting and multi-frequency phase unwrapping is the preferred choice for fringe projection profilometry [49].

3.1. Phase-Shifting Wrapped Phase Extraction

The design of phase-shifting algorithms depends mainly on the distribution of the phase shifts and whether they are known or unknown. Exploiting the fact that phase shifts can be controlled with high precision, they are required to be

$${\delta }_k={\displaystyle \frac{2\pi }{n}}(k-1).$$

(15)

For this particular case, the set of n fringe patterns given by Eq. (12) can be processed to estimate the background light, the fringe amplitude, and the encoded phase using the Bruning method [47,50], as

$$\begin{array}{cc}\hfill a(r,s)& ={\displaystyle \frac{1}{n}}\sum _{k=1}^n{I}_k(r,s),\hfill \end{array}$$

(16)

$$\begin{array}{cc}\hfill b(r,s)& ={\displaystyle \frac{2}{n}}\sqrt{J_1^2(r,s)+J_2^2(r,s)},\hfill \end{array}$$

(17)

$$\begin{array}{cc}\hfill \psi (r,s)& =-{tan}^{-1}\left(J_1(r,s)/J_2(r,s)\right),\hfill \end{array}$$

(18)

where ${tan}^{-1}$ represents the four-quadrant arctangent function, with the auxiliary functions defined as

$$\begin{array}{cc}\hfill J_1(r,s)& =\sum _{k=1}^n{I}_k(r,s)sin{\delta }_k,\mathrm{and}\hfill \end{array}$$

(19)

$$\begin{array}{cc}\hfill J_2(r,s)& =\sum _{k=1}^n{I}_k(r,s)cos{\delta }_k.\hfill \end{array}$$

(20)

As an illustration, the fringe patterns shown in Figure 6(e)-(h) were processed by Eqs. (16)-(18), obtaining the results presented in Figure 8. It is worth noting that the retrieved phase is $\psi (r,s)$, shown in Figure 8(c), while the required phase is $\varphi (r,s)$, shown in Figure 8(d). These phases are displayed using 3D plots in Figure 8(e) and Figure 8(f) for better visualization. The function $\psi (r,s)$ is known as the wrapped phase due to its distinctive sawtooth-like shape. The phases $\varphi (r,s)$ and $\psi (r,s)$ are equivalent since

$$cos\varphi (r,s)=cos\psi (r,s).$$

(21)

Unfortunately, $\varphi \in (-\infty ,\infty )$ can take any real value while $\psi \in (-\pi ,\pi ]$ is always constrained to the left-open interval known as principal values. Adding any multiple of $2\pi$ to the wrapped phase is also equivalent to $\varphi (r,s)$; i.e.,

$$cos\varphi (r,s)=cos\left[\psi (r,s)+2\pi h(r,s)\right],$$

(22)

which leads to the general relationship between $\varphi (r,s)$ and $\psi (r,s)$ as

$$\varphi (r,s)=\psi (r,s)+2\pi h(r,s),$$

(23)

where $h(r,s)$ is an integer-valued function, known as fringe order. Obtaining the actual phase $\varphi (r,s)$ from the available wrapped phase $\psi (r,s)$ is a process known as phase unwrapping.

Figure 8. Results of processing the fringe patterns shown in Figure 6(e)-(h). (a) Background light. (b) Fringe amplitude. (c) Extracted wrapped phase $\psi (r,s)$. (d) Required phase $\varphi (r,s)$. (e) and (f) 3D plots of the phases shown in (c) and (d), respectively.

Figure 8. Results of processing the fringe patterns shown in Figure 6(e)-(h). (a) Background light. (b) Fringe amplitude. (c) Extracted wrapped phase $\psi (r,s)$. (d) Required phase $\varphi (r,s)$. (e) and (f) 3D plots of the phases shown in (c) and (d), respectively.

3.2. Hierarchical Multi-Frequency Phase Unwrapping

Since the wrapping phenomenon appears when the encoded phase exceeds the principal values, the straightforward way to recover the required phase is by preventing it exceeds the principal values. For this, the frequency of the grating should be chosen appropriately. For instance, assuming the projector slide axis u is normalized in the interval $(-1,1]$, the angular frequency

$${\omega }_1=\pi$$

(24)

will limit the fringe pattern phase ${\varphi }_1={\omega }_{1}u$ within the principal values. Therefore, the required phase will coincide with that retrieved by Eq. (18), i.e.,

$${\varphi }_1(r,s)={\omega }_{1}u(r,s)={\psi }_1(r,s).$$

(25)

Note that the frequency ${\omega }_1$ is so low that only one fringe is displayed, as shown in Figure 6(a)-(d). However, low-frequency gratings cannot underline the fine details of the object. On the other hand, high-frequency gratings highlight shape details, as shown in Figure 6(e)-(h), but the wrapping phenomenon appears.

The hierarchical multi-frequency phase unwrapping employs both low- and high-frequency gratings. The phase retrieved from low-frequency gratings assists in solving the wrapping problem, while the phase from high-frequency gratings permits achieving high fidelity. For example, let us consider a second grating with a frequency ${\omega }_2$ higher than ${\omega }_1$,

$${\omega }_2>{\omega }_1.$$

(26)

Since the encoded phase exceeds the principal values, the required phase $\varphi$ and the retrieved phase $\psi$ are related as in Eq. (23), namely

$${\varphi }_2(r,s)={\omega }_{2}u(r,s)={\psi }_2(r,s)+2\pi h_2(r,s).$$

(27)

The unknown function $h_2(r,s)$ can be determined using the previous phase ${\varphi }_1(r,s)$ given by Eq. (25). Specifically, substituting $u={\varphi }_1/{\omega }_1$ in Eq. (27), we obtain

$${\displaystyle \frac{{\omega }_2}{{\omega }_1}}{\varphi }_1(r,s)={\psi }_2(r,s)+2\pi h_2(r,s),$$

(28)

which leads to

$$h_2(r,s)=⌊{\displaystyle \frac{({\omega }_2/{\omega }_1){\varphi }_1(r,s)-{\psi }_2(r,s)}{2\pi }}⌉,$$

(29)

where $\lfloor ·\rceil$ is the round operator ensuring $h_2$ takes integer values. This process can be repeated for a third frequency ${\omega }_3>{\omega }_2$, and so on, until the desired resolution is reached. In general, the k-th retrieved phase ${\psi }_k(r,s)$ can be unwrapped using a previous phase ${\varphi }_{k-1}(r,s)$ as

$$\begin{array}{cc}\hfill {\varphi }_k(r,s)& ={\psi }_k(r,s)+2\pi h_k(r,s),\hfill \end{array}$$

(30)

$$\begin{array}{cc}\hfill h_k(r,s)& =⌊{\displaystyle \frac{{\alpha }_k{\varphi }_{k-1}(r,s)-{\psi }_k(r,s)}{2\pi }}⌉,\hfill \end{array}$$

(31)

where ${\alpha }_k$ is the amplification between the adjacent phases ${\varphi }_{k-1}$ and ${\varphi }_k$, defined as

$${\alpha }_k={\omega }_k/{\omega }_{k-1}.$$

(32)

This phase unwrapping method is a recursive process that works hierarchically. It starts with the lowest (single-fringe) grating frequency and ends with the highest grating frequency supported by the projector.

3.3. Choosing Grating Frequencies

The operation of the hierarchical phase unwrapping method depends on how many frequencies can be used. Ideally, two frequencies, the lowest and highest, should be sufficient to attain a high-fidelity phase. However, in practice, using two frequencies often fails due to the excessive difference between the phases, as illustrated in Figure 9(a). For this reason, intermediate frequencies are employed to produce phases with less drastic changes, as shown in Figure 9(b)-(i).

Although any set of increasing frequencies would be chosen, using the same amplification between adjacent phases is recommended [38]; i.e.,

$${\alpha }_1={\alpha }_2=\cdots ={\alpha }_m=\alpha ,$$

(33)

where m is the number of frequencies to be used. The grating frequencies fulfilling the constant amplification requirement are

$$\begin{array}{cc}\hfill {\omega }_1& =\pi ,\hfill \\ \hfill {\omega }_2& =\pi \alpha ,\hfill \\ \hfill {\omega }_3& =\pi {\alpha }^2,\hfill \\ \hfill & ⋮\hfill \\ \hfill {\omega }_m& =\pi {\alpha }^{m-1}.\hfill \end{array}$$

(34)

The amplification coefficient $\alpha$ is determined based on the projector resolution, which limits the supported maximum grating frequency. For instance, assuming the projector has N pixels along the u-axis, the amplification coefficient is given as

$$\alpha ={\left({\displaystyle \frac{N}{\xi }}\right)}^{1/(m-1)},$$

(35)

where $\xi$ is the number of pixels per fringe at maximum frequency (usually between 10 and 20 pixels). The frequencies for the gratings encoding the projector v-axis are determined analogously.

4. System Calibration

In the theoretical preliminaries presented in Section 2, the camera and projector parameters were assumed to be known. In practice, these parameters need to be estimated through the process known as system calibration. Earlier methods to calibrate a camera and a projector together employed cumbersome procedures. Nowadays, more practical alternatives to calibrate a camera-projector pair are available. In this section, a simple and flexible camera-projector calibration method is presented. First, the calibration of a single camera is explained to gain background. Then, the methodology is extended to projectors by exploiting the equivalence between cameras and projectors. Finally, the procedure for simultaneous camera and projector calibration is described.

4.1. Camera Calibration

Consider the pinhole model given by Eq. (1), rewritten here as

$$\mathit{\mu }={\mathcal{H}}^{-1}\left[K\left[\begin{array}{cccc}{\overline{\mathit{r}}}_1,& {\overline{\mathit{r}}}_2,& {\overline{\mathit{r}}}_3,& -R^T\mathit{t}\end{array}\right]\left[\begin{array}{c}x\\ y\\ z\\ 1\end{array}\right]\right],$$

(36)

where the rotation matrix was row-partitioned as $R^T=[{\overline{\mathit{r}}}_1,{\overline{\mathit{r}}}_2,{\overline{\mathit{r}}}_3]$. The camera can be seen as a black box that receives a point $\mathit{p}={[x,y,z]}^T$ as input and returns an image pixel point $\mathit{\mu }$ as output. Therefore, a 3D target can be used to obtain a set of input-output pairs, as shown in Figure 10(a), and estimate the camera matrix C by minimizing the output error [35], as outlined in Figure 10(c). The parameters, K, R and $\mathit{t}$, can be extracted from C by a triangular-orthogonal matrix decomposition. Nonetheless, despite the single-shot calibration feature, this approach is impractical because high-precision multi-point 3D targets are bulky and expensive.

Alternatively, instead of capturing a single image of a 3D target, the camera can be calibrated using multiple images of a 2D target [51], as shown in Figure 10(b). We refer to the plane where the calibration target is located as the reference plane. Without loss of generality, the reference plane is assumed to be the $xy$-plane. Therefore, since z is always zero, the third entry of $\mathit{p}$ and the vector ${\overline{\mathit{r}}}_3$ in Eq. (36) can be removed, simplifying the imaging process to

$$\mathit{\mu }={\mathcal{H}}^{-1}\left[G\mathcal{H}\left[\mathit{\rho }\right]\right].$$

(37)

where $\mathit{\rho }={[x,y]}^T$ represents points on the calibration target, and G is a $3\times 3$ non-singular matrix known as homography, defined as

$$G=K\left[\begin{array}{ccc}{\overline{\mathit{r}}}_1,& {\overline{\mathit{r}}}_2,& -R^T\mathit{t}\end{array}\right].$$

(38)

An image of the 2D calibration target establishes a set of input-output pairs, allowing a homography to be estimated by minimizing the output error, as outlined in Figure 10(d). Although a single homography is insufficient for camera calibration, multiple homographies provide enough information to recover the required intrinsic and extrinsic parameters. For this, the upper-triangular shape of K and the orthogonality property of rotation matrices are exploited.

4.2. Projector Calibration

The data required for homography-based calibration are input-output pairs consisting of points from the reference plane matched with points from the device plane. This requirement is independent of whether the device to be calibrated is a camera or a projector due to their mathematical equivalence. Accordingly, the method for calibrating a camera or a projector is the same [52]; they only differ in how the required input-output pairs are acquired.

For camera calibration, input-output pairs are acquired by placing the calibration target on the reference plane and capturing photographs, as shown in Figure 10(b). The target provides known points on the reference plane, while the images provide the corresponding pixel points using automatic feature point detection [53]. Note that this input-output acquisition strategy is unsuitable for projectors because they cannot take photographs of the reference plane.

For projector calibration, input-output pairs are acquired by placing the calibration target on the device plane as a slide and illuminating the reference plane, as shown in Figure 11(a). The target provides known points on the projector plane, and the corresponding points are measured on the reference plane. Unlike camera calibration, where the input-output acquisition is automatic using dedicated image processing routines, projector calibration requires a manual measurement of feature point coordinates on the reference plane.

4.3. Camera-Projector Calibration

Calibrating a fringe projection system requires estimating the parameters of a camera and a projector that are working together. Although cameras and projectors can be calibrated independently, simultaneous calibration is advantageous because the camera assists the projector calibration.

Remember that a physical target on the reference plane is required for camera calibration. In addition, a virtual target on the slide plane is necessary for projector calibration. Although both targets are superposing with each other on the reference plane, they can be distinguished using targets of different colors [54]. For instance, a yellow target on the reference plane, see Figure 11(b), and a cyan target on the projector plane, see Figure 11(c). In this manner, the two superposed targets are retrieved separately from the captured image through its red and blue channels, as shown in Figure 11(d)-(f).

The image of the physical and virtual targets on the camera plane is opportune because manual measurements on the reference plane are avoided. First, the image of the physical target is used to estimate the camera homography $G_c$ from the relation

$${\mathit{\mu }}_p={\mathcal{H}}^{-1}\left[G_c\mathcal{H}\left[{\mathit{\rho }}_p\right]\right],$$

(39)

where ${\mathit{\rho }}_p$ is a feature point of the physical target (yellow), and ${\mathit{\mu }}_p$ is the corresponding point on the image (blue channel). In addition to using $G_c$ for camera calibration, this homography prevents measuring manually points on the reference plane for projector calibration. Namely, let ${\mathit{\mu }}_v$ be an image point of the captured virtual target (red channel). The corresponding point ${\mathit{\rho }}_v$ on the reference plane (cyan) can be computed using the inverse of $G_c$ as

$${\mathit{\rho }}_v={\mathcal{H}}^{-1}\left[G_c^{-1}\mathcal{H}\left[{\mathit{\mu }}_v\right]\right].$$

(40)

As a result, since the feature points ${\mathit{\mu }}_s$ on the projector slide plane are known because it contains the virtual calibration target, the projector homography $G_p$ can be estimated from the relation

$${\mathit{\mu }}_s={\mathcal{H}}^{-1}\left[G_p\mathcal{H}\left[{\mathit{\rho }}_v\right]\right].$$

(41)

This procedure is repeated for three or more poses of the devices or the reference plane moved freely. Finally, the camera and projector parameters are estimated from the estimated homographies. This methodology, known as camera-projector calibration, remains valid even when more advanced imaging models are employed [36].

For illustration purposes, the camera-projector calibration method was employed to calibrate the experimental system shown in Figure 12(a) using the software available in Ref. [52]. Then, the $50\times 100\times 50$ millimeter pyramid on the reference plane was reconstructed. Figure 6 and Figure 9(i) show, respectively, some of the captured fringe patterns and the demodulated phase encoding the projector u-axis. Additional fringe patterns were captured to demodulate the phase encoding the v-axis. Equations (13) and (14) were used to obtain the projector slide coordinates $u(r,s)$ and $v(r,s)$ from the demodulated phase. Finally, Eq. (9) was used to compute an object point for each established corresponding point, resulting in the 3D reconstruction shown in Figure 12(b).

5. Optical 3D Metrology

Nowadays, fringe projection profilometry has been successfully implemented in several fields of science and engineering. Its valuable features, such as accuracy and high resolution, make it an outstanding 3D metrology tool. This section describes a few illustrative applications in which optical fringe projection has been employed.

Non-contact and high-accuracy features have motivated the implementation of fringe projection profilometry in medical applications [55]. Some representative applications include human respiration rate measurement [56], diagnosis of tympanic-membrane and middle-ear disease [57], skin assessment [58], tympanic membrane characterization [59], body measurement for guided radiotherapy [60], assistance in laparoscopic procedures [61], biomechanics studies [62], and intraoral measurement for orthodontic treatment [63]. Figure 13 shows a dental teeth model reconstruction illustrating an intraoral measurement.

Automatic and uninterrupted inspection of items on production lines is essential to ensure high-quality products [64]. For this purpose, fringe projection profilometry is a valuable tool because of its capacity for visual inspection [65]. Other tasks where fringe projection profilometry is useful are metal corrosion monitoring [66], crack recognition assistance [67], welding surface inspection [68], micro-scale component testing [64], turbine blade wear and damage characterization [69], aircraft surface defect detection [70], and electrical overload fault detection [18]. Figure 14 shows a visible-thermal profilometer for multimodal 3D reconstruction, detecting a heat source caused by a simulated fault.

Conventional biometric-based security systems employ gray-scale images to perform user authentication. However, these systems lose valuable information by omitting important human biometric features such as the color and 3D shape of faces, palms, and fingers [71]. Fringe projection profilometry has been employed in security systems for face recognition [72], 3D palmprint and hand imaging [73,74], 3D fingerprint imaging [75], and ear recognition [76].

Other recent applications using fringe projection profilometry include robot vision for object detection and navigation [77], footwear and tire impression in forensic science [13], shape and strain measurements for mechanical studies [78], plant phenotyping and leafy green evaluation in agriculture [79], high-precision panel telescope alignment [80], whole-body scanning for animation and entertainment [81], airbag inflation for car safety studies [82,83], ancient coin and sculpture imaging for heritage preservation [84,85], fast prototyping and reverse engineering [86], three-dimensional color mural capturing for cultural heritage documentation [87], and many others [14].

This section is not an exhaustive review but rather a description of the state-of-the-art of fringe projection profilometry. Readers are welcome to contribute fresh ideas to drive 3D metrology with even more efficient, accurate, practical, and affordable optical profilometers.

6. Conclusions

The theoretical and experimental fundamentals of fringe projection profilometry have been reviewed. Helpful insights explaining challenging and confusing concepts were given, making this technology an uncomplicated tool. Simple yet powerful methods for fringe pattern processing, triangulation, and calibration were presented. The studied methods provide the reader with an adequate background to understand and operate a fringe projection profilometer. Some applications were described to expand the reader’s scope and stimulate research on further developments, driving 3D metrology toward new frontiers.