Assessing the Likelihood of Failure Due to Stem Decay Using Different Assessment Techniques

1. Introduction

Urban forests and greenspaces are increasingly considered an important priority for improving the sustainability, resilience, and livability of the urban landscape [1]. Trees in the urban forest provide many benefits such as air pollution reduction [2], storm water runoff attenuation [3], carbon sequestration [4], and building energy conservation [5]. Benefits generally increase as the size of trees increase [6], but as trees mature they are more likely to develop decay that increases their likelihood of failure [7]. In built environments, tree failures can result in fatalities [8], power outages [9], and catastrophic fires [10]; and damage from failures is associated with higher costs [11] and legal liability [12].

Arborists have assessed tree risk for many years. Recent revisions have brought the process into better alignment with risk assessment practices used in other disciplines. The current U.S. standard considers 1) the likelihood of a tree failure, 2) the likelihood of impact of a tree or tree part on a target and 3) the severity of the consequence if impact were to occur. Arborists assign one of four ratings to likelihood of failure (improbable, possible, probable, imminent) that are defined as follows: [13]:

• Improbable: The tree or tree part is not likely to fail during normal weather conditions and may not fail in extreme weather conditions within the specified time frame.
• Possible: Failure may be expected in extreme weather conditions, but it is unlikely during normal weather conditions within the specified time frame.
• Probable: Failure may be expected under normal weather conditions within the specified time frame.
• Imminent: Failure has started or is most likely to occur in the near future, even if there is no significant wind or increased load. This is a rare occurrence for a risk assessor to encounter and may require immediate action to protect people from harm. The imminent category overrides the stated time frame.

Decay is a common defect that is often associated with tree failure [7,14,15]. Decay reduces load-bearing capacity by reducing wood strength and, if wood components are completely digested, by creating voids that reduce cross-sectional area. Many tools and techniques to detect and assess the extent of decay have been developed. Some are simple (e.g., sounding the stem with a mallet); others are sophisticated (e.g., resistance drills and tomography) [16]. Many studies have investigated how well decay detection tools and techniques work [17,18,19,20,21,22,23,24].

Despite advancements in decay detection tools and techniques, many aspects of risk assessment remain uncertain because of the lack of knowledge about how trees grow and fail. Uncertainty may also be exacerbated by assessor bias, including an assessor’s personal risk tolerance [25]. Cognitive studies on human risk perception attribute an individual’s attitude towards risk to personal experiences [26,27], personal fears [28], and biases shared by communities [29]. An assessor’s training also influences ratings: trained professionals tended to return lower likelihood of failure ratings (LoFRs) than those without training [25,30].

Our objectives for this study were as follows:

• to determine whether more detailed information about the extent of trunk decay influenced experienced assessors’ LoFRs and, if so,
• to identify factors related to assessors and trees that explained the influence.

2. Materials and Methods

The study took place on the campus of the University of Massachusetts in Amherst, Mass., USA (USDA Hardiness Zone 5b). In July 2021, 18 experienced arborists who held the International Society of Arboriculture’s (ISA) Tree Risk Assessment Qualification (TRAQ) (among other credentials) assessed the likelihood of stem failure due to decay of 30 trees using 5 (basic and advanced) assessment techniques.

We selected trees for the field assessment based on practical considerations. The first was the availability of sonic and electrical resistance (ER) tomograms taken of the trunk taken within 2 m of the ground. Tomograms had been previously obtained using a PiCUS Sonic Tomograph 3, a TreeTronic 3 for ERT, and the Caliper 3 Geometry Measurement System (Argus Electronic GMBH, Rostock, Germany) following the methods of [23]. A second consideration was variation in compartmentalization response: weak (Pinus) and strong (Quercus). Finally, only (i) larger individuals (> 50 cm stem diameter measured 1.4 m above ground—“DBH”) and (ii) individuals that were close enough to one another that they could be grouped by location were selected. In the latter case, we selected individuals in six discrete clusters around the campus. We selected clusters of individuals for two reasons: (i) it included a variety of landscape settings (open space or near infrastructure such as roads, buildings, and parking lots); and (ii) it limited travel time to maximize the number of individuals that could be assessed in the two days when assessors visited campus. Prior to conducting the study, we pre-tested the methods and determined an efficient route to assess as many trees as possible in two days.

We recruited assessors from our professional networks, inviting only experienced assessors who (i) held the TRAQ credential, (ii) regularly performed risk assessments as part of their professional practice, and (iii) were familiar with advanced decay detection techniques such as resistance drilling and tomography. We offered continuing education units to assessors, but neither financial compensation nor reimbursement of travel expenses.

Before assessors arrived on campus in July 2020 to participate in the study, we used a Resistograph® F500-S (IML North America, Moultonborough, N.H., USA) to determine the thickness of sound wood (t) at between three and six locations spaced at approximately even intervals around the stem circumference and at the same height as the tomogram. For each location, we computed the t/R ratio, where R is trunk radius [31]. We attached flagging to the stem to indicate the locations of the tomography and Resistograph measurements (Figure 1).

We provided each assessor a binder that included a sheet for each tree. The sheet contained the following information: genus and species, DBH, height, the Resistograph output (Figure 2), and the sonic and ERT tomograms (Figure 3). Output from the Resistograph included a scaled diagram of the cross-section of the stem and lines indicating where drillings were made, the height and stem diameter where the drillings were made, the mean t value, and a table of all t/R values. The tomograms included the percentages of cross-sectional area that were sound or decayed. The decayed proportion of the cross-section was computed automatically from the combined areas of blue and purple in the sonic tomogram. Since we used the default settings (SoT1 calculation option and minimum velocity established at 50%), the resulting tomogram depicts the greatest possible area of decay in comparison to those generated using SoT2 and an expanded color space to view the minimum percent velocities. But the computed proportion of decayed wood indicated at the top of the tomogram assessors viewed during the study (e.g., Figure 3) did not include areas of intermediate velocities. We explained this to assessors prior to the field study. After the field study, we computed the loss in section modulus due to decay (Z_LOSS) from each sonic tomogram following the method of [32].

We instructed assessors to assign a rating of the likelihood of stem failure due to decay (“LoFR”) within 2 m of the ground and reminded them not to assess likelihood of failure of other parts of the tree. We used LoFRs from [13] and provided assessors with the definitions (listed in the Introduction). We instructed assessors to assign their LoFR based on a timeframe of three years.

Assessors performed five consecutive assessments of the LoFR. In order, assessments were as follows:

• visual assessment of the tree and its surroundings;
• sounding the trunk with a plastic mallet;
• viewing the Resistograph output (Figure 2);
• viewing the tomograms (Figure 3);
• consulting with a randomly assigned assessor.

Assessment techniques (a) and (b) are part of a Level 2 (“basic”) risk assessment [13]. Assessment techniques (c) and (d) are more sophisticated techniques to assess the amount and location (i.e., the “extent”) of decay and part of a Level 3 (“advanced”) risk assessment [13]. For odd-numbered trees, assessors viewed the resistance drilling output (c) before the viewing the tomogram (d); for even-numbered trees, assessors viewed the tomogram first. Consulting with a peer is not explicitly recommended in common professional guidelines [13,33]. Within each cluster of trees, assessors were randomly paired and inspected individual trees at their own pace.

After each of the five assessments [(a) – (e)] on a tree, assessors completed a survey to indicate their LoFR and describe the factor(s) (e.g., species, decay severity, tree size, exposure, lean, crown, etc.) that most influenced their LoFR, and if the additional information gained in the assessment technique changed their LoFR.

Assessors also self-reported the following information on the survey: years of experience performing tree risk assessments; number of trees assessed annually; relevant credentials in addition to TRAQ; and how frequently they used assessment techniques (b), (c), and (d) as part of their professional practice.

During the field study, not every assessor completed all five assessments of every tree. As a result, approximately 15% of the expected dataset was missing values. We used multivariate imputation by chained equations [34,35] to impute the most likely value for each missing value to obtain a full dataset prior to OLR analyses.

The university campus is well maintained, and no assessor assigned a LoFR of four (“imminent”) to any tree. Consequently, we coded LoFR ordinally as one (“improbable”), two (“possible”), or three (“probable”) and built ordinal logistic regression (OLR) models to investigate the effect of assessment technique on LoFR. All analyses were performed using the statistical language R, v4.1.2 [36]. In the OLR models, we included covariates describing trees [genus; DBH; percent of cross-sectional area with decay (from tomograms); average sound wood thickness (t) from the Resistograph output; t/R, where R is stem radius; Z_LOSS] and participants (years of experience; frequency of using a mallet, resistance drilling, and tomography when conducting risk assessments). We also included tree and assessor identification as random effects in each OLR model. We built models with the “clmm” function from the “Ordinal” package by iteratively adding covariates as single effects or interactions with the main effect of assessment technique [37]. Since the order of assessments differed between even- (viewed tomogram before Resistograph output) and odd-numbered (viewed Resistograph output before tomogram) trees, the variable “assessment technique” contained ten levels that represented an interaction between the five assessment techniques and even- or odd-numbered trees. We then selected the best model using lowest AICc scores.

In addition to the OLR analyses, we created a contingency table with four rows (one for each of the assessment techniques that followed the initial visual assessment) and two columns (to indicate whether the additional information gained for the assessment technique changed (“Yes”) or did not change (“No”) assessors’ LoFRs). We used a ${\chi }^2$ test to determine whether the proportion of affirmative and negative responses varied among assessment techniques.

Lastly, we investigated the influence of the random variables in the OLR model (assessor and tree) on LoFR. To investigate the influence of assessors, we evaluated if the consistency in assessor LoFRs changed among the five assessment techniques or four frequency-of-use categories of the tomogram or Resistograph. We quantified LoFR consistency with the “betadisper” function in the “vegan” package, which performed a multivariate test of homogeneity of variances on a Bray-Curtis (rank-based) dissimilarity matrix of the proportional distribution of LoFRs [38]. A multivariate approach was needed to evaluate inconsistencies of LoFRs with a single test.

To investigate the influence of trees between the initial visual assessment and each subsequent assessment technique, we computed the ratio of the weighted mean change in LoFR to the proportion of unchanged LoFRs for each tree. The ratio illustrated the frequency, magnitude, and direction of changes in LoFR from the initial visual assessment. We computed the ratio ($\mathcal{R}$) as follows:

• Compute the difference in LoFR from the initial visual LoFR:

  $${{∆LoFR}_{ijk}=LoFR}_{ijk}-{LoFR}_{vjk}$$

  where $i$, $j$, and $k$, are indices for the 4 assessment techniques following the initial visual assessment (indicated by the subscript $v$), 30 trees, and 18 assessors, respectively.
• Compute the proportion of unchanged LoFRs (i.e., $∆LoFR=0$) for each tree and assessment technique:

  $${\kappa }_{ij}=\frac{\left|{∆LoFR}_{ij}=0\right|}{\left|{∆LoFR}_{ij}\right|}$$

  (2)
• Compute the weighted mean change in LoFR:

  $$\overline{{∆LoFR}_{ij}}=\frac{\sum _{ij}\left({∆LoFR}_{ij}*{\omega }_{ij}\right)}{\left|{∆LoFR}_{ij}\right|}$$

  (3)

  where $\omega$ is a weighting factor of 1 (for LoFRs that changed one level from the initial visual assessment, e.g., from probable to possible or improbable to possible) or 2 (for LoFRs that changed two levels, e.g., from probable to improbable).
• For each tree and assessment technique,

  $${\mathcal{R}}_{ij}=\frac{\overline{{∆LoFR}_{ij}}}{{\kappa }_{ij}}$$

  (4)

We thus computed 30 values of $\mathcal{R}$ for each of the 4 assessment techniques that followed the initial visual assessment. From the resulting distribution of 120 values of $\mathcal{R}$, we considered only values in the upper and lower quartiles as having increased and decreased LoFR, respectively. We considered values of $\mathcal{R}$ within the interquartile range (IQR) as having the same LoFR as the initial visual assessment. In the rest of the paper, we refer to “increased”, “decreased”, or “unchanged” LoFRs rather than values of $\mathcal{R}$ in upper quartile, lower quartile, and IQR, respectively.

We described the basic assessment techniques as “consistent” if the LoFR assigned in the mallet assessment was unchanged from the initial visual assessment, and “inconsistent” if the LoFR assigned in the mallet assessment was greater or less than the initial visual assessment. We described the advanced assessment techniques as consistent if the change in LoFR from the initial visual assessment was the same for both advanced assessment techniques. We described the advanced assessment techniques as inconsistent if the change in LoFR from the initial visual assessment was not the same for both advanced assessment techniques. With respect to changes in LoFR from the initial visual assessment, we described the effect of the consultation assessment as “confirming” (or not) the basic and advanced assessments. If the LoFR assigned in the mallet and consultation assessments was unchanged from the initial visual assessment, the consultation assessment confirmed the basic assessment techniques. Similarly, if the LoFR was greater than or less than the initial visual assessment for both advanced assessment techniques and the consultation assessment, the consultation confirmed the advanced assessment techniques.

3. Results

3.1. Assessors

On average, assessors held the TRAQ credential for 6.1 years (3.1 years standard deviation). Some assessors additionally held the following credentials: ISA Board Certified Master Arborist (39%), American Society of Consulting Arborists (ASCA) Registered Consulting Arborist (39%), an advanced degree (M.S., Ph.D.) in arboriculture or a related field (67%). All assessors conducted tree risk assessments as part of their job; the mean years of practice was 14.3 (10.8 years standard deviation) with a mean of 425 trees assessed annually (737 trees standard deviation). Table 1 includes assessors’ responses to inquiries about their level of experience with the techniques and tools used in the study. Nearly all “often” conducted basic visual assessments using a mallet; a majority often or “occasionally” used a resistance recording drill, sonic tomography, or both.

3.2. Trees

Trees were semimature to mature, and large, with proportions typical of open-grown trees (Table 2). Table 2 also includes (i) the stem height at which tomography and Resistograph drilling were conducted, and (ii) the following covariates included in the OLR models for each tree: mean t value, minimum t/R ratio, percent decayed wood in the stem cross-section, and Z_LOSS.

Table 3 includes the fixed effects and interactions of the best OLR model to predict LoFR. Assessment technique influenced the predicted proportions of improbable, possible, and probable LoFRs (Table 3). The proportion of improbable LoFRs was least after assessors viewed tomograms; proportions of improbable, possible, and probable LoFRs were statistically similar among the other four assessment techniques (Figure 4). There were also significant interactions between assessment technique and the following covariates: percentage of decayed wood in the cross-section, mean t, and how often a participant used resistance drilling in professional practice (Table 3).

As the percentage of the cross-section with decay increased, the proportional response revealed greater LoFRs for the Resistograph, tomography, and consultation assessments (Figure 5). But the opposite was true for the visual and mallet assessments: the proportional response revealed lower LoFRs as the percentage of decay in the cross-section increased. The findings applied whether assessors viewed the Resistograph output or tomogram first. As the average thickness of sound wood increased, the proportional response revealed lower LoFRs, but the effect was proportionally greater for odd-numbered trees in each assessment technique (Figure 5). Assessors who used resistance drilling more often to assess tree risk assigned a greater proportion of lower LoFRs for all assessment techniques except the initial visual assessment (Figure 5). For the latter, assessors who used resistance drilling more often in their tree risk assessments assigned a greater proportion of higher LoFRs.

The other statistically significant influence on the distribution of LoFRs was how often assessors used tomography when conducting risk assessments (Table 3). Those who “often” used tomography assigned proportionally more improbable LoFRs than those who “never” used tomography (Figure 6). And variance was homogeneous among the four levels of assessors’ frequency of tomography use (Table 4).

3.4. Variability in Likelihood of Failure Ratings

Despite obtaining more information following each assessment technique, the variance among assessment techniques was also homogeneous (Table 4). Neither did more information substantially reduce variability among assessors (Table 5). In the initial visual assessment, assessors did not assign the same LoFRs for any tree, and most trees (77%) received two LoFRs. The proportion of trees that received a single LoFR increased for the subsequent assessments, but for the mallet and tomogram assessments, so did the proportion of trees that received three LoFRs. Even after the consultation assessment, most trees (77%) still received two LoFRs.

3.5. Changes in Likelihood of Failure Ratings

More assessors reported that they changed their LoFR following the Resistograph and tomogram assessments than the mallet and consultation assessments (${\chi }^2$ = 30.58, p < 0.0001, Table 6). Changes in LoFR from the initial visual assessment helped identify trees that were more (or less) difficult to assess (Table 7). For 4 trees, the LoFR was unchanged from the initial visual assessment for any of the 4 subsequent assessment techniques. For 16 of the remaining 26 trees, the LoFRs assigned in the basic assessments were consistent and confirmed by the consultation assessment in 9 of the 16 trees. For 12 of the remaining 26 trees, the advanced assessments consistently changed the LoFR from the initial visual assessment and the change was confirmed by the consultation assessment for 11 of the 12 trees. For 9 of the remaining 26 trees, only the LoFR assigned in the tomogram assessment was greater than the initial visual assessment.

The most commonly reported factors that assessors noted when assigning LoFRs to trees were the presence/absence of decay, the degree to which the tree was exposed to the wind, and the presence/absence of root problems (Figure 7). Together, these factors accounted for nearly half of the responses.

4. Discussion

Our results demonstrate that detailed information about the extent of trunk decay influenced experienced TRAQ-credentialed assessors’ LoFRs, but neither consistently nor in a straightforward way. The effect was most noticeable in greater LoFRs assigned following the tomogram assessment. But covariates related to trees (percent decay and t) and assessors (frequency of using resistance drilling tools for risk assessments) led to significant interactions with assessment technique, indicating the need for a more nuanced interpretation. A larger sample of assessors may have improved our understanding of their effect on LoFRs.

Because the Resistograph output and tomogram helped assessors visualize the extent of decay, we expected that the advanced assessment techniques would influence LoFRs—particularly for assessors who used advanced techniques less frequently. The influence was obvious in the changing proportions of LoFRs as percent decay changed, but only after assessors viewed the Resistograph output and tomogram. The pattern persisted following the consultation assessment, further supporting the idea that visualizing decay affected assessors’ LoFRs. But the overall trend did not apply to every tree. Our observation that the consultation assessment confirmed the basic assessment nearly as often as the advanced assessment was the result of greater LoFRs assigned following the tomogram assessment.

We speculate that the significant increase in LoFR following the tomogram assessment was due in part to the visual presentation of tomograms themselves. Our choice of the default (and more liberal) SoT1 calculation with minimum velocity set at 50% created tomograms with the largest area of decay. Assessors who often used tomography for risk assessments would more likely have understood that the tomograms may have overestimated the extent of decay using the default calculation, whereas assessors who only rarely used tomography may have been more inclined to increase the LoFR they assigned, as our findings suggest. Especially on stems of larger diameter and less regular shape, it is imperative that assessors are familiar with the uncertainty associated with interpreting tomorgrams [39].

It is also plausible that the complete and in-color view of decayed areas in tomograms may have been perceived as a more definitive depiction of decay, especially for assessors who used tomography less frequently. For instance, the number of holes drilled for the Resistograph may not have been adequate to precisely define the extent of decay, which could in turn result in assessors to experience greater uncertainty in how to interpret the Resistograph outputs. Resistograph outputs also were truncated and did not traverse the entire diameter. In contrast, tomograms presumably presented more visually compelling cross-sectional images than the black and white line drawings of the Resistograph output. For example, the extent of decay presented in Resistograph outputs and tomograms was similar for trees 22 and 27 (Figure 8), but the change in LoFR from the visual assessment was only greater after viewing tomograms. Without comparing tomograms and the outputs from the Resistograph to pictures of the cross-sections themselves, it was not possible to know which portrayal of internal decay was more accurate. Many studies have demonstrated the accuracy and limitations of each technique [18,19,21,22,23,24], which is why using both techniques to investigate the extent of decay is helpful [40].

For assessments that followed the initial visual assessment, the decreasing proportion of probable LoFRs assigned by assessors who more frequently used a resistance drilling tool in practice was intuitive. With visual assessments, however, the trend was inverted: the proportion of probable LoFRs increased with assessors who more frequently used a resistance drilling tool. It was not clear why this occurred. It may reflect assessors being accustomed to using simple and advanced tools to detect decay rather than focusing on a tree’s outward visual appearance. But previous studies have found for several species that visual assessment of a tree’s appearance often aligned with the extent of internal decay [14,15,17].

Statistically significant differences, however, do not imply that trends applied to all trees, assessors, and techniques. Trees 3 and 4 (Figure 9) highlighted both the advantage of using more than one technique to assess likelihood of failure due to stem decay and the challenges of individual assessment techniques. Both trees were Q. bicolor with nearly identical DBH; they were in the same location and presumably exposed to the same wind loads. Both trees also showed signs of past lightning strikes, with woundwood formed around the lightning damage, and superficial trunk decay. Their tomograms showed nearly identical percentages of sound wood (86% and 87%) but with areas of green indicating intermediate velocities and the possibility of decay. The Resistograph output for tree 3 (t/R ≥ 0.59, average t = 30 cm) aligned neatly with the tomogram, confirming—at least for an assessor who appreciates the nuanced interpretation of green areas using the SoT1 setting—that the extent and severity of decay were minimal. But the Resistograph output for tree 4 (minimum t/R = 0.22, average t = 18 cm) contradicted the tomogram: the extent and severity of decay presented more of a concern. The detailed description of each tree was reflected in the changes in LoFR: LoFRs assigned following the Resistograph and tomogram assessments decreased compared to the initial visual assessment of tree 3, but LoFR of tree 4 decreased compared to the initial visual assessment only following the tomogram assessment.

Individual trees also illustrated the limitations of using simple tools and techniques. Trees 14 and 26 (both P. strobus) thwarted assessors’ attempts to assess the extent of decay by sounding the trunk with a mallet, even though all but one assessor “often” sounded trunks in practice. Following the mallet assessment, the LoFR of each tree increased from the visual assessment. Assessors described the trunk as sounding hollow, but the advanced techniques revealed little decay. There were only five P. strobus in the study; that two were problematic suggests that sounding with a mallet may not be reliable for some species. Future studies should investigate this technique’s reliability.

Previous studies have shown that risk assessments are prone to bias related to an assessor’s training, experience, and perceptions of risk [25,41,42,43]. To manage subjectivity, clear definitions of categories in a risk matrix (e.g., the four LoFRs in [13]) [44] and sufficient training to calibrate assessors [45] are imperative. Yet despite assessors (i) holding the TRAQ credential (which requires continual training to obtain and maintain), and (ii) receiving more information about the extent of decay through five successive assessments of stem decay, some variation among their LoFRs persisted. For most trees and all assessment techniques, assessors assigned two or three LoFRs, and the non-significant beta dispersion test demonstrated that obtaining more information about the extent of decay did not reduce assessors’ variation, aligning with the findings of [42]. None of the covariates that described assessors’ experience adequately explained this finding. We speculate that this reflects the innate imprecision of assessing likelihood of failure. The persistent variation in LoFRs in our study and [42] may not be as problematic as one might suppose because studies have shown that assigned LoFRs were broadly consistent with measured likelihood of failure following storms [46,47].

5. Conclusions

Experienced, credentialed tree risk assessors often changed their rating of the likelihood of failure due to stem decay in response to obtaining new information about the extent of decay. The pattern was only statistically significant after viewing tomograms, but individual assessors and trees were plainly influential overall (as demonstrated by the significant random effects of assessor and tree in OLR models) and for specific assessment techniques (e.g., trees 14 and 26 with the mallet assessment). As expected, the amount of decay in the cross-section—reflected in the covariates percent decay (from the tomogram) and t (from the Resistograph output)—predicted assessors’ LoFRs, particularly in concert with their experience using each of the advanced decay assessment tools.

In the short-term, it is essential for arborists who assess tree risk to appreciate that variation among individual ratings is common but can be reduced with additional information, training, and experience. Even among the group of experienced tree risk assessors assembled for our study, responses typically focused on two of four possible LoFRs: either improbable and possible or possible and probable. Individual tree risk assessors base rating decisions on a wide variety of factors with each assessor weighing factors differently.