Modelling Socioeconomic Determinants of Building Fires through Backward Elimination by Robust Final Prediction Error Criterion

1. Introduction

Building fire is a continuing concern for households, businesses and authorities throughout Australia. In a single year, the country spent over $2.5 billion on fire protection products and services [1]. Despite the enormous expenditure, building fires still took the life of 51 Australians in 2020 [2]. Additionally, it cost Australia’s economy 1.3 per cent of the country’s Gross Domestic Product (GDP) [3]. The amount was an accumulation of losses due to injury, property damages, environmental damages, destruction of heritage and various costs to impacted businesses. The state of Queensland itself recorded 1,554 fire incidents involving damages to building structures and contents in 2020 alone [4]. Every single incident is about a Queenslander who has lost a family home, a loved one or a livelihood that has fed generations of Australians. Therefore, continued effort to understand and limit the incidence of fires is warranted.

Studies linking socioeconomic data to building fires have been conducted in various jurisdictions using different quantitative and qualitative methodologies. Lizhong, et al. [5] established the relationship between GDP-per-capita and education level to fire rate and fire death rate in Jiangsu, Guangdong and Beijing, China. It adopted the partial correlation analysis to compute the correlation coefficient of every variable pairing. In Cook County, United States, geocoding and visual mapping connected poverty rates to higher 'confined fires' incident rates in one-family and two-family dwellings [6]. Logistic regression is also used to identify relevant socioeconomic variables through four implementations within a four-stage conceptual framework [7]. The study utilises the census data of New South Wales residents and the corresponding variables selected to calculate indexes within the Socioeconomic Indexes for Areas (SEIFA) project.

Other methodologies have also adopted algorithms to not only assign coefficients but also select variables that build the most fitting model. For example, Chhetri, et al. [8] utilised the classical stepwise regression method and discriminant factor analysis (DFA) to select predictive determinants from variables identified in the technical papers of the Socioeconomic Indexes for Areas (SEIFA). As a result, it managed to capture variables that possess high t-statistics. However, its use of the classical stepwise regression method, as proposed by Efroymson [9], has been known for several limitations. Critics have also discouraged its use of t-statistics or p-value elimination criteria and the forward selection procedure to build statistical models [10,11,12,13,14]. The limitations of the classical stepwise regression method can be summarised into five issues – overreliance on chance, overstated significance, lack of guarantee for global optimisation, inconsistency-causing collinearity and non-contingency of outliers [10,11,12,13,14]. In addition, the method has been shown to provide poorer accuracy than the Principal Component Analysis (PCA) [15]. Therefore, the methodology was improved in a study in West Midlands, U.K., by adding PCA to discover the most predictive variables or components [16].

Similar to the West Midlands study, this paper also attempts to improve the methodology in Chhetri, Corcoran, Stimson and Inbakaran [8]. We propose a backward elimination by the Robust Final Prediction Error (RFPE) criterion to model the socioeconomic determinants of building fires. Such modifications to the model-building algorithm and elimination/selection criterion have the potential to produce a socioeconomic model that makes superior predictive accuracies. Additionally, the resulting model may make more cautious representations of individual parameters’ influence, preventing false confidence and reflecting the real world more accurately. The contribution of this paper includes the first application of the backward elimination by RFPE criterion and the comparative analysis of RFPE to other criteria applicable to the backward elimination procedure. Over and above that, the paper aims to play a part in improving the effectiveness of future fire safety regulations and programs that better protect households with the identified socioeconomic risk profile.

In assessing the method’s suitability, this paper delivers in six sections—the subsequent Section 2 reviews related work to explore the limitations of the classical regression method. Next, Section 3 describes the proposed robust backward elimination method by the RFPE criterion. A case study applied for South East Queensland is presented in Section 4. Afterwards, Section 5 details the available alternative criteria to the backward elimination procedure and the comparative analysis of the proposed criterion. Finally, Section 6 concludes the paper by discussing the study’s findings and future research directions.

2. Related Work

Before going into the method’s ingrained limitations, the common purpose of adopting the classical regression method has to be understood. Often, researchers adopted the method to disregard ‘insignificant’ variables to achieve parsimony, i.e., ‘simpler’ equations [10,11]. Then, the parsimonious model is inferred for the explanatory variables’ individual influences on the dependent variable [13,14]. Others utilise the resulting model for prediction and forecasting purposes [10,17].

Chhetri, Corcoran, Stimson and Inbakaran [8] conducted an ingenious study to model socioeconomic determinants of building fires. It has resourcefully identified the Index of Socioeconomic Advantages and Disadvantages (IRSAD) by the Australian Bureau of Statistics (ABS) as a suitable pool for candidate explanatory variables. In addition, the study uses discriminant function analysis (DFA) to identify determinants of fires in different suburbs – the culturally diversified and economically disadvantaged suburbs, the predominantly traditional family suburbs and the high-density inner suburbs with community housing. However, one subsection uses classical stepwise regression to identify the overall socioeconomic determinants of building fires. Although there are many valid uses of the method as an exploratory tool, misguided confidence and interpretations are likely to mislead research conclusions.

The limitations of the classical stepwise regression can be summarised into five issues – overreliance on chance, overstated significance, lack of guarantee for global optimisation, inconsistency-causing collinearity and non-contingency of outliers which can be described one by one, as follows.

2.1. Limitation 1: Over-Reliance on Chance

There is a high probability of the regression failing to identify actual causal variables. One of the main reasons is that the set of variables might, by chance, affect the particular training datasets. Without a validation process, the same variables might not show the same degree of influence if based on other sample datasets, such as datasets from other periods. The chance of nuisance variables getting selected, synonymously known as type I error, has been quantified by multiple studies such as the one by Smith [11]. Apart from referencing experiments that show poor performance in small datasets [18,19], Smith [11] conducts a series of Monte Carlo simulations to show that stepwise regression can include nuisance variables 33.5% of the time while choosing from 50 candidate variables. The rate almost tripled when the method was utilised for 1000 candidate variables. The simulations also found that at least one valid variable is not selected 50.5% of the time in choosing from 100 candidate variables [11]. The main reason for the limitation is that the statistical tests used in stepwise regression are designed for priori, i.e. the tests are made to quantify a model that has been previously built or established, for example, through expert knowledge and causation studies [10]. It is never intended for model-building purposes. In turn, the method produced results that often presented with overstated significance.

2.2. Limitation 2: Overstated Significance

McIntyre, Montgomery, Srinivasan and Weitz [13] determine that statistical significance tests have been too liberal for any stepwise regression model since it has been ‘best-fitted’ to the dataset, biasing the data towards significance. Additionally, Smith [11] stressed that stepwise regression tends to underestimate the standard error of coefficient estimate, leading to a narrow confidence interval, overstated t-statistic and understated p-values. The phenomenon also signifies the overfitting of the model to the training dataset. In practical terms, the stepwise algorithm does not pick the set of variables that determine the response variable in the population, but it picks the set of variables that ‘best’ fit the training sample dataset.

2.3. Limitation 3: Collinearity Causes Inconsistency

Stepwise regression assumes that explanatory variables are independent of each other. Therefore, there is no provision for collinearity in the stepwise regression procedure. As a result, collinearity in stepwise regression produces high variances and inaccurate coefficient estimates [20]. These effects are again attributed to the objective of finding a model that ‘best-fit’ the training data. Models that contain different variables may have a similar fit in the presence of collinearity; therefore, the procedure will result in inconsistent results, i.e. the procedure becomes arbitrary [10,21]. Collinearity’s effect on the order of inclusion or elimination is one of the reasons for the varying outcome [22,23]. With that said, these effects are more pertinent if the purpose of adopting stepwise regression is mainly inferential [24,25,26]. As a predictive model, the inconsistency is less relevant as variables compensate for each other as their coefficients are too high or too low [26]. Therefore, the resulting function may still satisfactorily predict the dependent variable but not as reliable in estimating individual influence [26].

2.4. Limitation 4: No Guarantee of Global Optimisation

Based on the limitations discussed, it is fair to question the optimality of stepwise regression’s outcome. Thompson [27], Freckleton [21], and Smith [11] discussed whether global optimisation is achieved in stepwise regression, especially the forward selection algorithm. Since the algorithm selects variables one by one, the choice of the n-th variable depends on the (n-1)th variable. Therefore, it is reasonable to conclude that the method cannot even guarantee that the n-variable model achieved is the best-fitting n-variable equation. In other words, the local optimisation reached by conducting the stepwise regression does not guarantee that it is the global optima. In addition, the issue may also be exacerbated by erratic variable selection in multicollinear datasets. Even a small degree of multicollinearity has been shown to bias stepwise regression towards achieving local optimisation, away from global optimisation [28].

2.5. Limitation 5: Caused by Outliers

Outliers are a persistent issue in statistical analysis. They introduced bias to the most basic statistical measure, e.g. the mean value of sample data, affecting the accuracy of more advanced statistical techniques [29]. One single outlier can bias classical statistical techniques that should be optimal under normality or linearity assumptions. Firstly, population data inherently contains outliers; as the sample data gets more prominent, there is a greater likelihood of encountering outlying data points [30]. Secondly, large behavioural and social datasets are more susceptible to outliers [30,31]. Thirdly, it has been established that outliers in survey statistics of such scale are almost unpreventable, partly due to significant errors in survey responses or data entry [29,32]. Additionally, unlike the effect of collinearity, there is evidence that outliers affect inferential accuracy and a model's predictive accuracy [33].

After acknowledging the limitation of the classical stepwise regression, a natural progression should lead to exploring an alternative to the method. Although the criterion modification will not wholly replace causation studies or eliminate the same weaknesses, it will produce significantly more reliable and cautious inferences and predictions.

3. Proposed Robust Backward Elimination Method

A multivariate regression equation was sought to represent the rates of building fires based on an area’s socioceconomic composition. The resulting equation is expected to take the form of Equation (1).

$$b_i=a_0+a_1{g}_{i1}+\cdots +a_d{g}_{id},$$

(1)

where variable $b_i$ represents the rate of emergency services at demand area i, and $g_{id}$ represents d socioeconomic variables indexed as 0, 1, …, d respectively for demand area i. $a_d$ is the regression coefficient allocated to each of the d socioeconomic variables.

A backward elimination by Robust Final Predictor Error (RFPE) criterion to detect and eliminate insignificant socioeconomic variables. The use of RFPE criterion, developed by Maronna, Martin, Yohai and Salibin-Barrera [31] has the benefit of minimising the effect of outliers. The robust technique is an improvement to Akaike’s FPE criterion that can be significantly biased by outliers in the dataset [34]. The procedure is then adapted to the data sourcing and processing methodology in Chhetri, Corcoran, Stimson and Inbakaran [8] study on building fires in South East Queensland. The approach has been proposed and discussed by Untadi, et al. [35]. The proposed RFPE equation is presented in Equation (2).

$${RFPE}_C=E\rho \left(\frac{y_0-x_{0C}^{\prime }\widehat{{\beta }_C}}{\sigma }\right)$$

(2)

$$\mathrm{where}\hspace{1em}\hspace{1em}\hspace{1em}\hspace{1em}\hat{\beta }=\mathrm{a}\mathrm{r}\mathrm{g}\underset{\beta \in {\mathbb{R}}^q}{\min }{\sum }_{i=1}^n\rho \left(\frac{y_i-{\mathit{x}}_{iC}^{\prime }\mathit{\beta }}{\hat{\sigma }}\right)$$

(3)

$$y_i={\sum }_{j=1}^p{x}_{ij}{\beta }_j+u_i={\mathit{x}}_i^{\prime }\mathit{\beta }+u_i$$

(4)

$$\rho \left(r\right)=r^2$$

(5)

$$x_{iC}=\{x_{i1},\dots ,x_{ip}\}$$

(6)

$$C\subset \{1,2,\cdots ,p\}$$

(7)

$$i=\{1,2,\cdots ,n\}$$

(8)

where $(x_{ij},y_i)$ is the dataset consisting of the relevant explanatory variables x_ij and response variable y_i. $(x_0,y_0)$ represents the supplementary data point to measure the sensitivity of the dataset to outliers. An iteratively reweighted least square algorithm derived an estimator for the error scale that is notated as σ, while $\hat{\beta }$ stands for the robust M-estimator of an unknown parameter β [36]. The estimator for the RFPE equation is developed as in Equation (2) [31].

$$\widehat{RFPE}=\frac{1}{n}{\sum }_{i=1}^n\rho \left(\frac{r_{iC}}{\hat{\sigma }}\right)+\frac{q}{n}\frac{\hat{A}}{\hat{B}}$$

(9)

$$\mathrm{where}\hspace{1em}\hspace{1em}\hspace{1em}\hspace{1em}\hat{A}=\frac{1}{n}{\sum }_{i=1}^n\psi {\left(\frac{r_{iC}}{\hat{\sigma }}\right)}^2,\hat{B}=\frac{1}{n}{\sum }_{i=1}^n{\psi }^{\prime }\left(\frac{r_{iC}}{\hat{\sigma }}\right)$$

(10)

$$r_{iC}=y_i-{\mathit{x}}_{iC}^{\prime }\widehat{{\mathit{\beta }}_C}$$

(11)

$$q=\left|C\right|$$

(12)

$$\psi \left(r\right)=2\left(r\right)$$

(13)

Equation (9) is then embedded in the backward elimination procedure as in Figure 1.

Firstly, RFPE for a model that consists of all d explanatory variables and is set as M_d , is calculated. Then, every variable is eliminated and returned one by one to determine which elimination improves the RFPE of model M_k-1 the most. The algorithm removes the single variable that improves the RFPE the most. The elimination iterates until the algorithm reaches an RFPE of M_k-1 that is higher or equal to the RFPE of M_k. The termination means the algorithm assumes the subsequent iteration will not improve the model fit.

4. Case Study: South East Queensland, Australia

South East Queensland refers to a region that accounts for two-thirds of Queensland’s economy and where seventy per cent of the state’s population resides [37]. The paper defines SEQ to include the Australian Bureau of Statistics (ABS)’s twelve Statistical Area 4 (SA4) regions – Eastern Brisbane, Northern Brisbane, Southern Brisbane, WesternBrisbane, Brisbane Inner City, Gold Coast, Ipswich, Logan to Beaudesert, Northern Moreton Bay, Southern Moreton Bay, Sunshine Coast and Toowoomba. The study’s datasets are analysed at the Statistical Area 2 (SA2) level as the unit of analysis. In the 2016 Census, there were 332 SA2 areas in 12 SA4 regions in South East Queensland.

4.1. Datasets

Inspired by the methodology developed by Chhetri, Corcoran, Stimson and Inbakaran [8], the study revolves around the Australian Bureau of Statistics (ABS) technical paper for Socioeconomic Indexes for Areas (SEIFA). One of the indexes within SEIFA is the Index of Relative Socioeconomic Advantage and Disadvantage (IRSAD). In this study, the variables used to calculate IRSAD as the initial variables in the backward elimination algorithm. South East Queensland’s IRSAD is visualised in Figure 2.

The data are extracted from a 2016 Census database - '2016 Census - Counting Persons, Place of Enumeration’. It consists of tables containing aggregated values at the selected Statistical Areas, for example, the HIED dataset in Table 1. The data was accessed through the TableBuilder platform. Each category was labelled with numerical values or symbols such as &&, @@ and V.V., referring to ‘Not Stated’, ‘Not Applicable’ and ‘Overseas Visitor’, respectively. The raw data was then used to calculate the data of the corresponding variables. Every variable was a proportion of the population calculated using criteria defined for its numerator and denominator, summarised in Table 2.

On the other hand, the rate of building fires in South East Queensland was set as the response variable of the study. It was calculated based on the Queensland Fire and Emergency Services (QFES) incident data points, labelled as incident types 111, 112, 113 and 119, from 2015 to 2017 [39]. The total number of incidents throughout the three years is then cumulated, multiplied by 1,000 and divided by the number of persons counted at each SA2 area in Census 2016, resulting in the triannual rate of building fires for every 1,000 people. The data is accessible through the Queensland Government Open Data Portal.

However, inconsistency exists between QFES and ABS geographical units of data labelling. This led to QFES tagging incident locations by their state suburb (SSC), while ABS collected the relevant socioeconomic data based on its definition of SA2. The main issue brought about by the difference is that a few suburbs are located in 2-4 SA2 areas. Specifically, there are 221 suburbs out of 3,263 located in more than one SA2 area. Therefore, the study has adopted a ‘Winner Takes All’ approach by assuming overlapping suburbs as part of the SA2, where most of the suburb residents were located (50 per cent plus one). A matrix of suburb and SA2, represented as rows and columns respectively, was generated through the ABS TableBuilder platform and named ‘SSCSA2’. The ‘Winner Takes All’ approach is conducted through the code in Figure 3. It identifies the maximum value at every row and assigns the rows with a SA2 – the name of the column at which the maximum value is located.

It also has to be noted that the QFES incident data points are labelled with suburb names that contain some misspellings. For example, some identified errors include ‘Cressbrookst’ and ‘Creastmead’. Additionally, the dataset does not distinguish names used for multiple different suburbs. Therefore, the study has identified these suburb names and added parentheses, distinguishing the suburbs by following the ABS State Suburbs (SSC) naming convention and cross-referencing the postcodes of the suburbs at issue. One example is Clontarf (Moreton Bay – Qld) and Clontarf (Toowoomba – Qld).

4.2. Parameters

The results were obtained using the R software at its 2021.09.0 version on a device equipped with AMD Ryzen 5 3450U, Radeon Vega Mobile Gfx 2.10 GHz and 5.89 GB of usable RAM. In addition, the RobStatTM package was used to execute the robust stepwise regression analysis [40]. The tuning constant for the M-scale used to compute the initial S-estimator was set to 0.5. The constant determines the breakdown point of the resulting MM-estimator. Relative convergence tolerance for the iterated weighted least square (IRWLS) iterations for the MM-estimator was set to 0.001. The tolerance level was chosen to allow convergence to occur. The desired asymptotic efficiency of the final regression M-estimator was set to 0.95. Finally, the asymptotic bias optimal family of the loss function was used in tuning the parameter for the rho function.

4.3. Results

Eleven variables have been eliminated, leaving thirteen variables in the final model. Table 3 shows the variables eliminated at every step, complete with the RFPE calculation to demonstrate improvements in subsequent models. A detailed model specification, which includes a coefficient a_d for every retained variable g_d, (see Eqn. 1) was contained in Table 4.

Overall, the model’s R-squared was calculated to be 0.4259, translating to 42.59 per cent of variations explainable by the variables retained. The adjusted R-squared of 0.4024 has indicated the model R-squared upon fitting to another dataset in the population. The figure sufficiently satisfies the threshold set by Falk and Miller [41] for endogenous constructs such as the one obtained. A robust residual standard error (RSE) of 0.3779 meant the observed building fire rates were off from the actual regression line by approximately 0.3779 units on average.

In identifying the individual parameters' influence, an abundance of caution has to be exercised as a model-building algorithm is known to overstate significance [11]. Further assessment, for example, through Monte Carlo simulations, is recommended. Four socioeconomic variables, CHILDJOBLESS, NOCAR, OCC_MANAGERS and UNEMPLOYED, were significant at the 0.001 level. Based on their t-statistics and F-statistics, the corresponding p-values (0.000106, 6.77e-07, 5.68e-07 and 0.000141 respectively) preliminarily indicated the variables’ inclusions are not due to chance. UNEMPLOYED highest positive coefficient increases building fire rates to the highest degree, while OCC_SERVICE_L lowest negative coefficient decreases building fire rates the most significantly.

The model's forecasting ability is further scrutinised through the following plots in Figure 4 and Figure 5.

The plot of response value against its fitted values in Figure 4 compares the model’s supposed rates of building fires and the actual rates. Most suburbs are evidently scattered around the identity line where the ratio of the two sets of values is one-to-one, showing a visually representative equation of the data points’ behaviour. Additionally, it shows data points labelled 105, 259 and 276 – representing Eagle Farm to Pinkenba, Rocklea to Acacia Ridge and Southern Southport as outliers. This is because the rates of building fires in these regions are exceptionally higher.

In Figure 4, the three outlying points are again highlighted because they are significant leverage points. It means that the outliers may have a significant influence on biasing the goodness-of-fit of the model. Most other data points can be classed as regular – not leverage points nor outliers. Additionally, there are also nine points with high leverage, i.e. they have the potential to influence the model dramatically.

Cross-validation is then conducted to investigate if the criteria and procedure have produced a model that is applicable to the dataset that is not used to train the model. The Root of the Mean of the Square of Errors (RMSE) in Equation (14) and the Mean of Absolute value of Errors (MAE) in Equation (15) are used as the basis for comparison.

$$\mathrm{R}\mathrm{M}\mathrm{S}\mathrm{E}=\sqrt{\frac{{\sum }_{i=1}^n{(y_i-y_p)}^2}{n}}$$

(15)

$$MAE=\frac{{\sum }_{i=1}^n\left|y_i-y_p\right|}{n}$$

(16)

where y_i is the actual rate of building fires, y_p is the projected rate of building fires and n is the number of observations/suburbs.

The cross-validation procedure is showcased in Figure 6, along with the resulting goodness-of-fit measures in Table 5.

Table 5 shows negligible differences between the root-mean-square error (RMSE) and mean absolute error (MAE). An exception was observed in the iteration with the third fold as the testing dataset. A substantial difference was detected; however, the average difference across the five iterations is still negligibly low. This indicates the model’s equal performance on a dataset not involved in training the model obtained through the proposed method.

5. Comparative Study

5.1. Alternative Backward Elimination Criteria

There are sizeable alternative criteria to equip the backward elimination in assessing the goodness-of-fit of a model. Therefore, this paper adopts four criteria as the comparative basis for the RFPE criterion.

5.1.1. Adjusted R-Squared

R-squared is biased against a model with more parameters or explanatory variables, therefore, influencing the algorithm to favour parsimonious models. Ezekiel [42] sought to counteract this effect by adjusting the R-squared equation. Adjusted R-squared can be interpreted as a less biased estimator of the population R-squared, whereas the observed sample R-squared is a positively biased estimate of the population value [43]. The difference to the classical R-squared includes a penalty for the inclusion of redundant variables by multiplying the equation, (n — 1)/(n — p) [44]. The adjusted R-squared is defined and simplified as follows.

$${\overline{R}}^2=1-\left(1-R^2\right)\frac{n-1}{n-p}$$

(17)

where p is the total number of explanatory variables in the model, and n is the sample size.

5.1.2. Akaike Information Criterion (AIC)

Akaike [45] proposed an indicator for a model’s quality by measuring its goodness-of-fit by estimating Kullback-Leibler divergence using the maximum likelihood principle. The Akaike’s Information Criterion (AIC) is proposed as follows [46].

$$AIC=2k-2\ln \left(L\left(\hat{\theta }|y\right)\right)$$

(18)

where k is the number of parameters, L represents the maximum likelihood function of the parameter estimate $\hat{\theta }$, given the data y. The criterion can be embedded within a backward elimination algorithm as follows [47].

$$AIC=n\log \left(\frac{RSS}{n}\right)+2k+constant$$

(19)

where k is the number of parameters, n is the sample size, and RSS is the residual sum of squares of the model. Stepwise AIC algorithm has been implemented in financial, medical and epidemiological applications [48,49,50].

The AIC is the most commonly used information theoretic approach to measuring how much information is lost between a selected model and the true model. It has been used widely as an effective model selection method in many scientific fields, including ecology and phylogenetics [51,52]. Compared with the use of adjusted R-squared to evaluate the model solely on fit, AIC also considers model complexity [51].

5.1.3. Bayesian Information Criterion (BIC)

Bayesian Information Criterion, also known as Schwarz Information Criterion, was proposed by Gideon Schwarz [53]. Modifying Akaike’s Information Criterion by introducing Bayes estimators to estimate the maximum likelihood of the model’s parameters. The BIC is formulated as follows.

$$BIC=k\mathrm{l}\mathrm{n}\left(n\right)-2\ln \left(L\left(\hat{\theta }|y\right)\right)$$

(20)

where k is the number of parameters, L represents the maximum likelihood function of the parameter estimate $\hat{\theta }$, given the data y. Similarly to AIC, the BIC are applicable to the backward elimination algorithm as follows.

$$BIC=n\log \left(\frac{RSS}{n}\right)+k\mathrm{l}\mathrm{n}\left(n\right)+constant$$

(21)

where k is the number of parameters, n is the sample size, and RSS is the residual sum of squares of the model.

The strength of BIC includes its ability to find the true model if it exists within the candidates. However, it comes with a significant caveat, as the existence of a true model that reflects reality is debatable. Although BIC penalises overfitting on larger models, preferring a more parsimonious or lower-dimensional model. However, for its predictive ability, AIC is better because it minimises the mean squared error of prediction/estimation [54].

5.1.4. Predicted Residual Error Sum of Squares (PRESS)

Allen [55] developed an indicator of a model’s fit through the predicted residual error sum of squares (PRESS) statistic. The differentiation to the statistic at that time is its ability to measure fit based on samples that are not used to form a model [55,56]. The statistic is a cross-validation attempt by a leave-one-out method that subtracts ${\hat{y}}_{\left(i\right)}$ and leaves the i-th observation out, reducing the sample size to n – 1 [57]. Repeating the subtraction and omission to every single data point will lead to the sum of squares of discrepancies [58,59]. PRESS is formulated as follows.

$$PRESS=\sum _{i=1}^n{\left(y_i-{\hat{y}}_{\left(i\right)}\right)}^2$$

(22)

5.2. Comparison to Akaike’s Information Criterion (AIC) Criterion

Twelve variables have been eliminated, leaving sixteen variables in the final model. Table 6 shows the variables eliminated at every step, complete with the RFPE calculation to demonstrate improvements in subsequent models. A detailed model specification, which includes a coefficient a_d for every retained variable g_d, (see Eqn. 1) was contained in Table 7.

Overall, the model’s R-squared was calculated to be 0.4730, translating to 47.30 per cent of variations explainable by the variables retained. The adjusted R-squared of 0.4532 has indicated the model R-squared upon fitting to another dataset in the population. The figure sufficiently satisfies the threshold set by Falk and Miller [41] for endogenous constructs such as the one obtained. A robust residual standard error (RSE) of 0.4544 meant the observed rate of the building fires was off from the actual regression line by approximately 0.4544 units on average. The following section provides a comparative benchmark to assess the quality of the RSE.

The same cautions should be applicable for a reason provided in Section 4.1. CERTIFICATE, CHILDJOBLESS, HIGHBED, INC_HIGH, NOCAR and OCC_MANAGERS were identified as having the most significant relationships to the response variable based on their t-statistics and F-statistics. They were shown to have p-values (7.45e-06, 0.000384, 2.31e-06, 0.000106 and 2.26e-09 respectively) significant at the 0.001 level. Therefore, these variables fluctuate building fire rates to the highest degree.

The model’s forecasting ability is compared to the one produced by using the RFPE criteria through the following plots in Figure 7.

Both models’ Q-Q plots are similarly right-skewed or positively skewed, as showcased in Figure 7a,b. Regardless, the data points remain generally on the linear line, crowding towards the zero quantiles. Therefore, both are normally distributed with a heavy right tail. Subsequently, the plot of the residual-fitted value shows that deviations to the model produced using the RFPE criterion are slightly more well-behaved than that of the AIC criterion. The relatively straighter line in Figure 7c is predominantly biased by one point. The rest of the points scatter randomly around the zero line; therefore, Gauss-Markov’s assumption of linearity is more convincing than that of Figure 7d. Furthermore, both models are also compared on their goodness-of-fit in Table 8.

The two models are comparable on RMSE and MAE. The model produced through the RFPE criterion resulted in a lower MAE, but a higher RMSE than the model produced through the AIC criterion. To investigate further, the same cross-validation procedure in Figure 6 is then applied to the model produced using the AIC criterion. The results are contained in Table 9.

Based on the RMSE and MAE measure, the RFPE criterion fares slightly better than the AIC. This is consistent with the robust premise of the RFPE criterion that sought to reduce the effect of outliers on the resulting model. Since RMSE punishes the model with large error against the outliers, it conforms that model produced using RFPE criterion has higher training RMSE, but a lower testing RMSE.

5.3. Comparison to Other Criteria

For comparative purposes, the study has also adopted p-value, adjusted R-squared, BIC and PRESS as criteria for the backward elimination procedure. The methods are carried out using the ‘SignifReg’ package in the R environment. The criteria are consistent with their respective equation in Section 5.1. Using the entire dataset for training, the resulting model from each criterion is assessed for their goodness-of-fit, as in Table 10.

The models are shown to be comparable on the three measures, although the RFPE criterion is again slightly superior on the basis of MAE. The cross-validation procedure in Figure 6 is then carried out on the four different models. The resulting goodness-of-fit measures are detailed in Table 11.

Comparing the averaged goodness-of-fit measures across the different models, the RFPE criterion still holds minuscule superiority over the averaged MAE measured on the test and train datasets. The lower MAE demonstrates the robust criterion provided by the RFPE as the MAE is not as sensitive to outliers as RMSE, indicating a model more accommodating to extreme cases [60]. Also, the robust nature of the RFPE criterion is again apparent in the third iteration, where we can observe a test dataset with a higher incidence of outliers. The RFPE criterion has produced a model with a noticeable advantage where the lower RMSE and MAE measures indicate a more resilient model that provides a better fit even to a significantly outlying dataset.

6. Conclusions

This paper has identified limitations in the prevailing socioeconomic modelling of building fires and proposed the backward elimination by Robust Final Predictor Error (RFPE) criterion to model the socioeconomic determinants of building fires. The proposed approach has been evaluated using datasets available in the region of South Eastern Queensland in Australia. A comparative study demonstrated that the proposed method is a more accurate predictive model regarding goodness-of-fit measures obtained through the cross-validation procedure. The model has relatively lower error measures, albeit the difference has not been dramatic.

Future research may involve comparing the backward elimination by the RFPE criterion to alternative approaches such as Least Absolute Shrinkage and Selection Operator (LASSO), Least Absolute Residuals (LAR) and Principal Component Analysis (PCA) [16,61,62]. Monte Carlo simulations can also be employed to assess further the resulting model's dependability for identifying individual parameters, including its measure of influence through the relevant coefficient. All in all, the paper has found sufficient justification to adopt the backward elimination with the RFPE criterion to model the socioeconomic determinants of building fires for predictive purposes.