PSO based Unconstrained Polygonal Fitting of 2D Shapes

1. Introduction

Polygonal shape fitting is a key problem in computer vision and computer graphics with several applications including object recognition, computational cartography, signal summarization, and compression [1,2,3,4]. When curves are modeled, straight-line segments are usually preferred due to their simplicity. The polygonal fitting process saves memory space, reduces rendering time on graphics applications, and gives a more compact representation of the original shape.

According to the classical polygonal approximation problem of a 2D shape, the goal is to compute an N-vertex polygonal curve P that approximates the boundary B of the original shape S according to a predefined error criterion. In the classical polygonal approximation, the vertices of P are an ordered sub-sequence of the boundary points B [1]. The classical polygonal shape fitting provides poor results, especially when the number of vertices is low and the shape complexity is high. In this work, we study a general version of polygonal fitting called Unconstrained Polygonal Fitting (UPF) problem to provide better solutions with the same number of vertices. According to UPF, the N-vertices of P can be placed anywhere in the 2D space. Therefore, the search space of UPF problem is a superset of the classical polygonal approximation (PA) problem, where the vertices are constrained to belong in the boundary of the given 2D shape. This means that the resulting solutions of UPF may better approximate the given curve than the PA problem solutions. This theoretically interesting computer vision problem has compact and easy-to-grasp description, but a very high algorithmic complexity due to the large search space. Even if just a triangle ($N=3$) is used, there does not exist any trivial method to compute the optimal solution of UPF problem.

Different error criteria have been proposed for shape fitting - polygonal approximation problems. In this research, the Intersection over Union (IoU) metric between S and P is maximized given a fixed number of polygonal vertices N, without any assumption or prior knowledge of the object structure. IoU metric can be easily replaced by any other segmentation metric, for example, accuracy and dice coefficient ($DICE$) without any change in the proposed methodology. Another problem instance, that we experimentally study is to define the problem under the Equal Area constraint which states that the area covered by the polygonal curve P should be equal to the area of the given shape S.

Figure 1 shows two example outputs of the classical polygonal approximation using the Douglas-Peucker algorithm [5] (left) and the unconstrained polygonal fitting using the proposed method (right) for $N=4$ vertices. In both cases, it holds that the solution of the UPF problem provides higher IoU, DICE values compared with the results of Douglas-Peucker algorithm. In the first example with an apple, the Douglas-Peucker algorithm yields $DICE=79.6\%$, $IoU=66.1\%$ (Figure 1a), while the proposed method yields $DICE=90.8\%$, $IoU=83.2\%$ (Figure 1b). In the second example with a building, the Douglas-Peucker algorithm yields $DICE=88.9\%$, $IoU=80.1\%$ (Figure 1c), while the proposed method yields $DICE=97.5\%$, $IoU=95.1\%$ (Figure 1d). There exists several examples where there exists significant differences in performance between the optimal solutions of the two problems, especially for lower values of N where the optimal solutions of UPF space may be far from the shape boundary.

In summary, the major contributions of our work are the following: To the best of our knowledge, this is the first work to define and solve the UPF problem. The proposed method solves the UPF problem via Particle Swarm Optimization (PSO) [6] that quickly explore the search space. The proposed framework suggests solutions that outperform baselines in quantitative terms, even if are not necessarily optimal due to the PSO. Especially in a low-dimensional search space, that is true for low number of vertices, the proposed method clearly outperforms baselines of classical PA, since even the optimal solutions of the classical PA problem usually fail to obtain high performance results due to the limited search space (shape boundary) of PA problem.

2. Related Work

The problem of shape fitting - polygonal approximation has been studied extensively during the last five decades [1,4,5,7]. The methods that have been developed solve the problem by approximating the original shape S by a polygonal curve P under the constraint that the P vertex sequence is an ordered sub-sequence of the vertices along shape boundary. The polygonal approximation problem can be formulated in two ways [1,7]:

• The problem of better fitting, where the approximation error is minimized or the fitting accuracy is maximized given the number of vertices N.
• The problem of minimum number of vertices, where the approximation error or the fitting accuracy is bounded and the goal is to find the minimum number of vertices that satisfy the given bound.

The polygonal approximations algorithms can be classified as (1) optimal or non-optimal and (2) supervised or unsupervised algorithms [7]. The non-optimal algorithms do not guarantee any kind of optimum as optimal algorithms but can find reasonable polygonal approximations faster than the optimal ones. The supervised algorithms take into account parameters to generate polygonal approximations, while the unsupervised algorithms generate the polygonal approximations without parameters. Yin [8] presents a polygonal approximation approach based on the discrete particle swarm optimization (PSO) algorithm. Each particle represented as a binary vector corresponds to a candidate solution to the polygonal approximation problem. A swarm of particles is initiated and fly through the solution space for targeting the optimal solution. In [9], the authors proposes a non-optimal and unsupervised algorithm for generation of polygonal approximations based on the convex hull of the 2D closed curves or contours. The significance levels of the contour points are computed using a symmetric version of the well-known Douglas-Peucker algorithm and, finally, a thresholding process is applied to obtain the vertices or dominant points of the polygonal approximation.

In the case of 2D shape fitting, a 2D binary image is given with foreground points representing the shape to be modeled. This image can be the result of any object detection or image segmentation method (e.g., [10,11,12]). Several models can be used to solve the 2D shape fitting problem. In the literature, there exists several approaches that fit a set of ellipses to the given 2D shape based on (a) the Hough Transform,(b) Genetic and Optimization Algorithms and (c) edge-following [12,13]. In [12], a framework has been proposed to represent a given 2D shape with an automatically determined number of ellipses based on expectation maximization criterion, so that the total area covered by the ellipses is equal to the area of the original shape. The method has been successfully applied to cell [14] and tree detection [15] problems. In [4], an algorithm is proposed to extract and vectorize objects in images with low-complexity polygons based on local merging and splitting of cells. Departing from a polygonal partition that oversegments an image into convex cells, the algorithm refines the geometry of the partition while labeling its cells by a semantic class. The authors applied their method to a variety of scenes, from organic shapes to man-made objects through floor maps and line-drawing sketches.

Concerning the metrics that have been used to measure the performance on shape fitting problem can be classified into :

• Boundary based metrics that compares the distance between the boundary of given shape and the approximated curve. Some of the most most metrics are the root mean square error ($RMSE$) between the two curves and the maximum error between the boundary S and their corresponding subcurves of P. These metrics have the advantage that can also be applied on given curves (non closed contours) but are sensitive to boundary noise [1,7].
• Region based metrics that compares the region of the approximated shape and the given shape. Region-based metrics are more tolerant to noise since they do not restrict themselves to the boundaries of shapes but rather take into account all shape points [12]. Common metrics of this category are the Intersection over Union (IoU) and the Dice coefficient (DICE), which is defined by twice the area of the shapes’ intersection divided by the sum of the areas of each shape. Both of them have been also used to measure the performance of image segmentation methods [16].

3. Unconstrained Polygonal Fitting

3.1. Problem Formulation

We assume a binary image I that represents a 2D shape S. A pixel p of I belongs either to the foreground S ($I\left(p\right)=1$) or to the background ($I\left(p\right)=0$). The area A of the 2D shape is given by

$$A\left(S\right)=\sum _{p\in S}I\left(p\right).$$

(1)

We also assume a closed polygonal curve P with N vertices $P_i$, $i\in \{1,...,N\}$, which belong in the 2D space (not necessary on the boundary of S). The binary image $I_P$ is also defined so that $I_P\left(p\right)=1$ at points p that are inside any of the polygonal curve P and $I_P\left(p\right)=0$, otherwise. Then, the Intersection over Union $IoU\left(P\right)$ metric between the 2D shape S and P is calculated as follows:

$$IoU\left(P\right)=\frac{{\sum }_{p\in S}{I}_P\left(p\right)}{{\sum }_{p\in I}max(I\left(p\right),I_P\left(p\right))}.$$

(2)

Essentially, $IoU\left(P\right)$ is the percentage of the 2D shape points that are under polygonal curve P divided by the number of 2D points that belong to shape S either polygonal curve P. The optimal solution of Unconstrained Polygonal Fitting problem is the closed polygonal curve P that maximizes the $IoU\left(P\right)$.

$$P^{*}=argma{x}_{P}IoU\left(P\right)$$

(3)

3.2. Equal Area Principle

The equal area principle (constraint) has been used in [12,14] restricting the area of P to be equal with the area of the given shape. This is a reasonable constraint in the sense that we except in many cases that the optimal solutions of the UPF problem have almost the same area with the given shape S. So, by adding this extra constraint, we reduce the search space of the problem without mainly affecting the performance of the method. Additionally, some applications may also include this requirement. Let $A\left(P\right)$ denote the areas of polygonal curve P, $A\left(P\right)={\sum }_{p\in I}{I}_P\left(p\right)$. According to the Equal Area constraint it holds that the area of polygonal curve P is equal with the area of shape S meaning that $A\left(P\right)=A\left(S\right)$. Therefore the optimal solution of Unconstrained Polygonal Fitting problem under equal area principle is the closed polygonal curve P that maximizes the $IoU\left(P\right)$ under equal area principle.

$$P^{*}=argma{x}_{P}IoU\left(P\right),A\left(P\right)=A\left(S\right)$$

(4)

Hereafter, we describe a simple procedure that applies the equal area principle on a polygonal curve P, yielding a new polygonal curve $\overline{P}$ so that $A\left(\overline{P}\right)=A\left(S\right)$. Let C be the centroid of S1 and ${\overline{P}}_i$ be the i vertex of polygonal curve $\overline{P}$, $i\in \{1,...,N\}$.

$$C=\frac{1}{A\left(S\right)}\sum _{p\in S}p$$

(5)

$${\overline{P}}_i=C+\sqrt{\frac{A\left(S\right)}{A\left(P\right)}}·(P_i-C)$$

(6)

3.3. UPF-PSO Algorithm

In this work, we cast the search for a single polygonal curve as a stochastic optimization problem that is solved based on Particle Swarm Optimization (PSO) [6]. PSO is a derivative-free optimization method that handles multi-modal, discontinuous objective functions with several local minima. Optimization is performed through the evolution of particles (candidate solutions) of a population (swarm). Particles lie in the parameter space of the objective function to be optimized and evolve through a limited number of generations (iterations) according to a policy which emulates “social interaction”. The main parameters of PSO are the number of particles and generations, the product of which determines its computational budget (i.e., the number of objective function evaluations). PSO achieves near-optimal solutions and has been successfully applied in several challenging optimization problems in computer vision and pattern recognition such as classification, clustering, prediction, image segmentation, video co-segmentation, object tracking [17,18].

The proposed Unconstrained Polygonal Fitting based on Particle Swarm Optimization Algorithm (UPF-PSO) optimizes IoU metric (see Eq. 3) for different polygonal curves that are directly represented by PSO particles. The input to UPF-PSO is the binary image I representing the given shape S and the number of vertices N of the polygonal closed curve that will be fitted to S. The UPF-PSO method quickly explores the search space starting from random solutions. Iteratively, PSO searches for a polygonal curve that maximizes the IoU metric.

We represent each particle by a $2·N$ vector with the 2D coordinates of the N points of the polygonal curve. In order to simplify the search space, we assume that the vertices are in clockwise order; otherwise, in the evolution process we correct the order of vertices of each particle according to this hypothesis. The fitness (objective function) of the particle is directly given by the IoU metric of the particle (see Eq. 2). The UPF-PSO Algorithm is analytically described hereafter.

Initially, we create a population of M (e.g. $M=20$) particles that are located in random positions around the centroid of the shape S under the Equal Area Principle (see Section 3.2) to be in valid positions of the search space. In the evolution process, PSO finds the current optimal solution in order to update the best global solution. Additionally, the best local solution of each particle is also updated, where the IoU of the particle reaches a better solution. The method terminates when the number of iterations of the evolution process exceeds the given number of generations, e.g. 100.

4. Experimental Evaluation

The evaluation of the proposed approach was based on two standard datasets from the literature. More specifically, we employ:

• MPEG-7 [19], which consists of 1,400 binary shapes organised in 70 categories with 20 shapes per category. This dataset has been extensively used in shape tasks [12,14,20].
• A subset of LEMS [21], that is, 1,462 shapes that come from the following categories of the original database: Buildings, Containers, Fish, Fruit and vegetables, Misc Animal, People, Robots, Toddlers, and Turtles [14].

Figure 2 shows twelve sample images form the MPEG-7 dataset and LEMS dataset.

We compared the proposed Unconstrained Polygonal Fitting based on Particle Swarm Optimization Algorithm (UPF-PSO) and UPF-PSO under equal area principle (UPF-PSO-EA) methods2 with the Douglas-Peucker algorithm (DP) [5] and Douglas-Peucker algorithm under equal area principle (DP-EA) [5]. Additionally, we have implemented a sequential baseline method of classical Polygonal Approximation that maximizes IoU. This method uses an initial polygonal curve that is given by the execution of the Douglas-Peucker algorithm (DP) with high number of vertices (e.g. $50>>N$). In each step, it sequentially removes the vertex of the polygonal curve so that the IoU of the resulting polygonal curve is maximized. The method terminates when the number of vertices are reduced to N. This baseline method is called Sequential Maximization of IoU (SM). After the execution of SM, the equal area principle can be applied yielding the SM-EA method.

In our experiments, we have evaluated the methods for different values of N. Since, our framework makes sense to be applied for low values of N, where the difference in performance between the solutions of UPF and PA usually is significant, we have fitted polygonal curves with $3\le N\le 10$ (eight cases). According to the formulation of the UPF problem, we have compared the performance of UPF-PSO, UPF-PSO-EA, DP, DP-EA, SM and SM-EA on the IoU metric. For a given dataset, we also compute $Pr(m/IoU)$, where m is a method in {UPF-PSO, UPF-PSO-EA, DP, DP-EA, SM, SM-EA}. $Pr(m/IoU)$ is quantified as the percentage of shapes of the datasets where the method m clearly outperforms the two others under the IoU. This also means that the value $100\%-{\sum }_{m}Pr(m/IoU)$ gives the percentage of images for which there is no clear winner method.

Table 1 and Table 2 present the average IoU, DICE and Pr(m/IOU) metrics computed on all images from the MPEG7 and LEMS datasets, respectively. It holds that UPF-PSO and UPF-PSO-EA clearly outperform the rest methods in any dataset. UPF-PSO slightly outperforms UPF-PSO-EA. UPF-PSO or UPF-PSO-EA outperform all the rest methods in about $80\%$ of shapes under MPEG-7 and LEMS datasets. SM or SM-EA outperform all the rest methods in about $17\%$ of shapes under MPEG-7 dataset and LEMS dataset. DP or DP-EA outperform all the rest methods in about $3\%$ of shapes under MPEG-7 and LEMS datasets.

Figure 3 depicts the average IoU of UPF-PSO, UPF-PSO-EA, DP, DP-EA, SM and SM-EA computed for different values of $N\in \{3,...,10\}$ in the two datasets used. UPF-PSO and UPF-PSO-EA clearly outperform the rest methods under any dataset and $N\le 8$. As expected, the lower N, the better outer-performance of the proposed methods compared to the classical PA methods. For high values of N ($N>8$), the problem search space increases, so it is more difficult for the PSO to find near-optimal solutions. At the same time, when N obtains high values, the solutions of classical PA significantly better approximate the given shape, reducing the gap from the UPF methods. Theoretically, it holds that as N tends to infinity, the optimal solution of both problems (classical PA and UPF) converges to the initial shape having IoU equal to one ($li{m}_{N\to \infty }IoU=1$). When equal area principle is applied it holds that the results are similar under any method. Usually, the equal area principle provides better results for low values of N where the fitting polygon is less complex. The results under DICE metric depicted in Figure 4 agree with the results of Figure 3.

Figure 5 illustrates the Pr(m/IoU), $m\in${UPF-PSO, UPF-PSO-EA, DP, DP-EA, SM, SM-EA} over the 1,400 images of MPEG-7 dataset (Figure 5(a)) the 1,462 of LEMS dataset Figure 5(b). The results agree with the results of Figure 3. If we sum the cases where UPF-PSO or UPF-PSO-EA outperforms all the rest methods, it holds that for $N\le 8$ is more than $70\%$ and $68\%$ under MPEG-7 dataset and LEMS dataset. As it was expected, when $N=3$, UPF-PSO or UPF-PSO-EA clearly outperform the rest method on more than $98\%$ of shapes and any dataset.

Finally, Figure 6, Figure 7 and Figure 8 depict several output examples of the proposed methods and baselines to further study their behaviour. The DICE-IoU metrics and the used method-N are depicted in the title and caption of each result, respectively. These figures depict results for $N=3,6,9$ vertices on a turtle, bat and horse shapes from MPEG 7 and LEMS datasets. The results of $N=3$, $N=6$ and $N=9$, corresponds to low, middle and high values of N according to the proposed framework. In most of the cases the proposed methods outperform the baselines. As it was expected, the difference in performance between the proposed methods and baselines is higher for low value of N ($N=3$), due to the limited classical polygonal approximation problem - search space that is restricted on the shape boundary. PSO based methods yield high performance results, when it is more possible to find near-optimal solutions, which is true under low values of N and less complex given shapes. On other cases (high values of N and complex shapes), the PSO based methods may fail to provide high performance solutions due to the complicated and high dimensional search space. For example, in the case of $N=9$ and horse shape (see Figure 8) that is the most complex shape, the results of UPF-PSO and UPF-PSO-EA underperform the SM method. However, it should be noticed that under this complex shape, when $N=3$ or $N=6$ it seems that the proposed methods outperform all baselines. Concerning, the other two simpler shapes, it holds that UPF-PSO or UPF-PSO-EA outperform any other method even when $N=9$, showing that the proposed methods are able to provide high performance results even for high values of N.

5. Conclusions

In this work, a novel PSO based polygonal fitting algorithm (UPF-PSO) has been proposed to solve a general version of the polygonal fitting problem called Unconstrained Polygonal Fitting (UPF). An N-vertex polygonal curve P with known number of vertices is fitted to a given 2D shape S so that the Intersection over Union is maximized. $IoU$ metric can be easily replaced by any other segmentation metric e.g. Accuracy/Dice coefficient without any change in the proposed methodology. According to the UPF-PSO algorithm, the location of the N-vertices of P that can be placed anywhere in the 2D space generally provides better solutions compared to those of the classical polygonal approximation problem, where the vertices are restricted to belong in the boundary of the given 2D shape. In our experimental results, we have also compared the proposed UPF-PSO with several baselines in two standard datasets of 2D shapes of more than 2,800 images showing the high performance of the proposed framework. As it was expected, when the number of vertices is low, the difference in performance between UPF-PSO and the rest baseline methods increases as the solutions of classical polygonal approximation problem generally fails to provide well fitting results. In ongoing and future work, our aim is to study more unconstrained fitting problems and to provide better solutions especially for high values of N by relaxing the constraint that the new shape border should be a subset of the given shape boundary. Additionally, we plan to consider extensions of UPF-PSO towards handling more complex shapes than polygons. Finally, we plan to extend the proposed framework on 3D shape and to explore real applications that the proposed proposed system may be useful.