Development and Validation of the Food Swamp Environmental Audit (FS-EAT) Tool

1. Introduction

The Social Ecological Model and the Getting to Equity (GTE) framework emphasize the multilevel and structural nature of diet-related disparities and highlight the need for addressing the structural and environmental contributors to poor health, especially in communities affected by racialized poverty [1]. Poor diet is a major contributor to obesity and associated diseases, and neighborhood and community food retail environments create and maintain disparities in people’s food habits, body weight, and overall health. Recent research points to an association between food swamps and obesity prevalence [2,3,4,5,6,7,8]. Food swamps are best described as geographic areas where unhealthy food retail outlets (e.g., fast-food restaurants, corner stores) inundate healthier alternatives (e.g., grocery stores, supermarkets) [9,10,11]. This is important as living in a food swamp has been linked to suboptimal health behaviors including food acquisition behaviors, diet quality, higher intake of sugary drinks, and lower intake of fruits and vegetables (F&V). Previous research has also established associations between food swamps and health disparities such as increased obesity risk, obesity-related cancer mortality, heightened diabetes severity and hospitalizations, early-onset colorectal cancer mortality, metabolic-dysfunction-associated steatotic liver disease mortality, and adverse post-operative outcomes among patients undergoing colorectal surgery [12,13,14,15,16].

Food swamp severity has increased over the past two decades, especially in urban areas where healthier food outlets are sparse [12,17,18]. Studies show that adults with limited access to supermarkets are 25–46% less likely to maintain a healthy diet [19]. This issue is even more acute among older adults due to mobility limitations [20]. Racial and ethnic minorities, particularly Black and Latino populations, are disproportionately affected by food swamps due to structural disparities in food and economic systems [17]. These communities frequently report transportation, economic, and social barriers to food access. Perceptions of the local food environment also shape shopping behaviors and diet quality, suggesting a strong role for subjective and community-informed assessments [11].

Despite growing research, public health policies have largely focused on addressing food deserts, defined as areas where people have limited access to grocery stores [21]. Efforts toward addressing food deserts by opening full-service supermarkets in underserved areas have improved perceived access but have not consistently improved diet or body mass index (BMI) [9,10,11,22,23]. This has prompted a shift in attention toward food swamps, where unhealthy options are not just available but disproportionately concentrated and more accessible. Several policy strategies, such as zoning laws, healthy retail ordinances, and nutritional labeling, have been proposed to mitigate food swamps. Yet, implementation is limited by the lack of standardized, validated tools for identifying food swamps [9,10,11,22,23,24,25,26,27].

There is a need for more accurate, context-based, validated tools for food swamps. This would advance food swamp research by improving the quality of the underlying food store data. Existing food environment research frequently depends on secondary government and commercial data sources such as the US Department of Agriculture (USDA)’s Economic Research Service (ERS) Food Environment Atlas, SNAP retailer locator, Nielsen, InfoUSA, Dunn and Bradstreet, or the National Establishment Time Series (the longitudinal version of Dunn and Bradstreet) [27,28,29]. However, validation studies of secondary food outlet databases have shown that they may lack accuracy, timeliness, and context [30,31,32,33]. Liese and colleagues found that the accuracy of store-level identification improved from 55% (D&B) and 65% (InfoUSA) to 81% after combining the two commercial datasets [31]. Mui et al. subsequently leveraged their approach to combine InfoUSA and Dunn and Bradstreet (D&B) food store data to compute food swamp scores [34]. While this approach effectively improved the food store accuracy rate, it may not always be practical or cost-effective for researchers to purchase multiple commercial datasets simultaneously. In prior food swamp research using audit data, Hager et al. measured food swamp exposure using Baltimore City Food Environment Map data compiled by the Johns Hopkins Center for a Livable Future’s original Healthy Food Availability Index (HFAI) tool [35]. Notably, food swamps were identified based on proximity to corner and convenience stores, as the HFAI tool does not include any fast-food, carry-out, or sit-down restaurants.

Context-based, validated tools would also facilitate more consistent food swamp measurement via data completeness. Currently, food swamp research is undermined by the use of several food swamp measures reflecting relative balance (vs. absolute) in the food environment, such as fast-food percentages, the Retail Food Environment Index (RFEI), the modified Food Retail Environment Index (mRFEI), and the Food Swamp Index (FSI). For instance, using Nielsen food store data, Colón-Ramos et al. defined food swamps as areas where ≥71% of food outlets are fast-food retailers. [36]. As a second example, Cooksey Stowers et al. integrated fast-food, convenience-store, and grocery store data from the USDA Food Environment Atlas to calculate adapted RFEI scores, finding stronger associations between food swamps and obesity than with food deserts [9,13]. A longitudinal study of changes in food swamp exposure in 69,904 US census tracts from 2000 to 2019 computed mRFEI scores and found racial disparities in the presence of food swamps and market concentration. Notably, each relative food swamp measure requires data on different types of food outlets. FSI cannot be computed without data on ‘Intermediate’ stores such as sit-down restaurants and mixed-specialty stores. Thus, across studies, there is no consistent measurement approach, making comparisons between studies difficult [18,37].

Lastly, in line with the growing momentum for community-engaged and community-based participatory research (CBPR) in public health research, there is a need for CBPR approaches to food swamp research, including the development of measures [1].

Given the above gaps, a valid, reliable, and community-informed food swamp tool is urgently needed to assess neighborhood-level food swamp exposure. The present study uses CBPR methods to:1) develop the Food Swamp Environment Audit (FS-EAT) tool; 2) assess content and face validity; 3) test the tool’s interrater reliability; and 4) assess convergent validity by ground truthing FS-EAT data against secondary NETS data and comparing food swamps vs. non-food swamps by data source.

2. Materials and Methods

2.1. Study Design: A Community Participatory Approach to Tool Development

The Food Swamp Environment Audit Tool (FS-EAT) was developed between 2019 and 2021 using a CBPR approach, informed by literature on neighborhood food swamps as structural drivers of diet-related health inequities and adapted from existing tools [9,11,12,14,15,16,38,39,40,41]. The development was a collaboration between academic researchers and two community principal investigators (Community PIs). The Community PIs were leaders from two local advocacy organizations- Hartford Food System and the Hispanic Health Council-and shared the funding as well as scientific and administrative oversight to ensure the research aligned with local priorities. Further, a Community Advisory Board (CAB) including local leaders was convened to guide the tool’s design and ensure cultural and contextual relevance.

The CAB was intentionally structured to reflect the experiential diversity of the North Hartford Promise Zone (NHPZ), including representatives from resident groups, neighborhood associations, public health advocacy groups, a local hospital’s community engagement center, a charitable nonprofit, a not-for-profit community development financial institution, and churches. The CAB was also designed to ensure representation across racial and ethnic backgrounds, income levels, and age groups. The CAB, which had 12 members, met bimonthly, initially in person (August 2019–March 2020) and later via Zoom during the COVID-19 pandemic. The Community PI’s organizations were supported by subawards under the parent grant (Academic PI: Cooksey Stowers), and resident CAB members were compensated for their time and contributions with gift cards and meals. The tool was validated by a brief survey with 26 key stakeholders (non-profit leaders, public health officials, community organizers, residents’ advisory board) and feedback sessions with the Community Advisory Board (CAB).

This study involved two multi-phased processes, namely i) tool development and reliability testing, and ii) face, content, and convergent validity tests. The details are provided below:

2.2. Setting

The study sites were six low-income, ethnically diverse urban neighborhoods in Hartford, Connecticut: Northeast, Upper Albany, Clay Arsenal, Asylum Hill, Frog Hollow, and Barry Square. These neighborhoods were purposively selected because they are predominantly low-income and minority, yet also exhibit diversity in income, ethnicity, percent foreign-born, urbanicity, and health risk factors. Further, during community meetings, the Community Principal Investigators (PIs) and additional local leaders acknowledged the varied food retail environments. These neighborhoods also vary by ‘North Hartford Promise Zone’ (NHPZ) status. In 2015, under the Obama Administration, three neighborhoods in Hartford (i.e., Northeast, Upper Albany, Clay Arsenal) were identified as one of 22 “high-poverty, high capacity” communities in the U.S. by the Department of Housing and Urban Development Promise Zone initiative [42]. This designation indicates that they are high-poverty, distressed areas with alarming rates of unemployment, food insecurity, and crime, and that targeted strategies are needed to advance their revitalization. The poverty rate in the NHPZ is 49% (compared to 33.6% in Hartford and 10.8% in CT) [23]. Residents of North Hartford die on average 10 years earlier than residents in a suburban community less than five miles away. Adults in the NHPZ are twice as likely to be obese as residents of the three non-NHPZ sites and have higher rates of poverty, diabetes, coronary heart disease, and hypertension [23,24]. The NHPZ comprises 19 census block groups, and the comparison sites comprise 26. Table 1 presents key demographic data of the six neighborhoods.

2.3. Tool Development

Tool development followed a three-stage process:1) initial item drafting based on a literature review and community feedback; 2) assessing content validity via a key stakeholder survey; and 3) tool pretesting on 18 streets.

2.3.1. Initial Tool Drafting and Community Feedback

This audit tool builds on existing food environment audit tools, such as NEMS-R, NEMS-S, and NEMS-CS [26,44], as well as neighborhood-level tools, including the Food Environment Audit for Diverse Neighborhoods and the Healthy Food Availability Index [40,41]. In accordance with the study’s CBPR approach, the list of food outlet types and characteristics was refined based on feedback from the Community PIs and from two feedback sessions with the CAB (n=6) and residents (n=6) on the tool's subcomponents. As a result, the initial draft of the FS-EAT Tool included subcomponents for a variety of food outlets (e.g., grocery stores, restaurants, convenience stores, food pantries), social features (e.g., loitering, graffiti, cleanliness), and fast-food restaurant accessibility (e.g., drive-thru).

2.3.2. Assessing Content Validity Via Key Stakeholder Survey

A short survey was conducted with 26 key stakeholders (e.g., non-profit leaders, public health officials, community organizers, residents) to help validate food environment-relevant items from the literature for inclusion in the FS-EAT tool (e.g., food retail outlets, social features, and food promotion). Each stakeholder was asked to quantify the relevance of 19 items that could describe the food retail environment in their neighborhood by selecting a value between 1 and 4 (where 1 is “not relevant” and 4 is “extremely relevant”).

See Figure 1 for the list of questions, which include types of food retail outlets, social features, and food promotion. In addition to the feedback sessions (referenced above in Section 2.4), these numerical ratings reflect the importance of potential retail food environment features and help ensure that the food retail items included in the tool are appropriate for the study sites.

2.3.4. Pretesting Tool & Refinement

The first draft of the FS-EAT tool was pretested by two trained observers (a postdoctoral fellow and a graduate student) in three randomly selected streets in each of the six study neighborhoods (18 streets in total) between December 2019 and January 2020. As a result of the pre-testing, revisions were made to the tool, including reframing questions for clarity, adding additional instructions, reordering questions for sequential flow, and incorporating the mystery shopper protocol to differentiate small grocery stores from corner stores, bodegas, or convenience stores. Additional details regarding the mystery shopper protocol are outlined below. These revisions, along with CAB feedback, resulted in the final version of the FS-EAT tool, which was used for data collection and evaluated in subsequent phases of the project, as described below.

The scope and content of the final FS-EAT tool included 19 items across the following topics: 1) street-level counts of food outlets by type; 2) store-level food and beverage marketing exposure; 3) store-level social features; [41] 4) store-level mystery shopper protocol to identify small grocers vs. corner stores and bodegas; 5) store-level mystery shopper protocol to collect beverage pricing; 6) store-level accessibility (parking, transit lines nearby), and 7) fast-food restaurant accessibility features (drive-thru type, pre-ordering options, and on-site playgrounds). The FS-EAT was programmed in Research Electronic Data Capture (REDCap) version 10. To achieve street-level counts of food outlets by type (section 1), longitude and latitude were captured for each outlet and street segment via REDCap’s built-in feature. Furthermore, to differentiate between small grocery stores and convenience stores, a Mystery Shopper Protocol was used, in which stores were assessed for the presence of fresh produce and meat sections (typical features of small grocery stores) to support accurate classification [45,46,47].

2.4. Tool Evaluation

Tool evaluation followed a three-stage process: i) inter-rater reliability test; ii) convergent validity testing by ground truthing FS-EAT audit data against secondary NETS business listings, and iii) face validity via feedback sessions with Community PIs and CAB.

2.4.1. Inter-Rater Reliability

The final version of the FS-EAT was used in a comprehensive assessment of all food stores across the 40 streets within the 45 block groups that make up the 6 neighborhoods. This assessment was conducted by two independent teams, each consisting of two trained observers. Each team visited with a printed data collection schedule that included a list of streets’ start and end points cross-referenced on Google Maps, geocoded food stores from the purchased 2015 National Establishment Time-Series (NETS) Database to confirm stores on street boundaries, and an iPad with the FS-EAT tool on REDCap. Each team was assigned streets to visit to avoid duplicate audits, and the full audit was completed over 16 weeks between February and May 2020. During the data collection, the data points were “ground-truthed” (i.e., confirmed directly by direct observation on the ground) to identify stores that were in operation, closed, or had been replaced [29,34,48,49,50,51,52,53].

Then, the stores data were scored; these scores were compared, differences were reconciled, and kappa scores were computed to assess interrater reliability. Interrater reliability was assessed by comparing the independent ratings from the two teams using Cohen’s kappa (κ) statistic to quantify agreement for the overall FS-EAT tool and its major subcomponents, including street-level outlet counts, store-level marketing exposure, social features, fast-food accessibility characteristics, and pricing measures. Detailed reliability estimates are presented in the results section.

2.4.2. Assessing Convergent Validity

After establishing the reliability of the FS-EAT data, the tool's convergent validity was assessed by ground-truthing the food store data from FS-EAT against NETS 2020 data.

For the objective alignment assessment, food store types from both the FS-EAT tool and NETS data were categorized as healthy, intermediate, or unhealthy based on prior research [34,54,55]. Food outlets that provide access to produce or a wider selection of fresh, nutrient-rich foods were included in the “healthy” category (e.g., grocery stores/supermarkets, produce/fruit and vegetable markets, fish and seafood markets, and superstores such as wholesale clubs). Food outlets with mixed evidence on whether they should be classified as “healthy” or “unhealthy” were included in the “intermediate” category (e.g., full-service restaurants and mixed-specialty stores such as Top Taste Caribbean Bakery & Grill and Scotts Jamaican Bakery). Food outlets that sell predominantly unhealthy foods high in fat, sugar, or salt were categorized as Unhealthy (e.g., fast-food chain restaurants, limited-service/carry-out restaurants, convenience stores, corner stores, unhealthy specialty stores, dollar stores, pharmacy chain stores, and gas station chains). Table 2 shows the food store category by data source type.

Food swamp indices (Food Swamp Index, Physical Food Environment Index, and Modified Retail Food Environment Index) and food swamp maps were then generated for different food store categories in the FS-EAT and NETS datasets. These indices and maps provide proxies for food swamp exposure needed to complete the validity tests. Three food swamp indices were computed as follows:

• Food Swamp Index (FSI) = (Unhealthy + Intermediate) / Total Stores × 100
• Modified Retail Food Environment Index (mRFEI) = Healthy / Total Stores × 100
• Physical Food Environment Index (PFEI) = Unhealthy / Total Stores × 100

Indices were scaled from 0 to 100, with higher FSI and PFEI values indicating greater concentration of unhealthy food outlets, and lower mRFEI values indicating fewer healthy options.

Sensitivity analyses were conducted at two decision points: (1) varying the name-similarity and geographic-proximity thresholds used to match food outlets between the FS-EAT and NETS datasets, and (2) comparing block group food swamp classifications derived from using the mean versus the median audit FSI as the classification cutoff. These checks were important because both the store-matching criteria and the classification threshold can shape how block groups are ultimately designated as food swamps, and confirming dependability across specifications strengthens confidence that the convergent validity findings reflect real patterns in the data rather than a single analytic choice.

2.4.3. Assessing Face Validity via Feedback sessions with Community PIs and CAB

In line with the study's CBPR approach, feedback sessions with the Community PIs were held every 2 weeks and lasted 60-90 minutes, while those with the CAB were held monthly for 1 hour. To establish face validity, during these feedback sessions, the Academic PI presented preliminary results and solicited targeted feedback regarding the following topics: 1) agreement with their lived and professional experiences in the neighborhood; 2) data interpretation and clarity in how the data was presented; and 3) next steps for revisions to data analyses or subsequent waves of data collection. CAB meetings were recorded and transcribed verbatim. Academic PI (XXX) and a notetaker also took meeting minutes.

3. Data Analysis

3.1. Tool Development: Key Stakeholder Survey Ratings

Relevance scores from the stakeholder survey were analyzed descriptively. Each of the 19 FS-EAT items was rated on a scale from 1 (“not relevant”) to 4 (“extremely relevant”). Mean scores were calculated for each item. Items with a mean score of ≥2 (“somewhat relevant” or higher) were retained in the final tool used to collect data and during the tool evaluation phase.

3.2. Tool Development: Community Advisory Board (CAB) Feedback Session Notes

Community Advisory Board meeting minutes and transcripts were analyzed via an iterative content analysis approach. Between monthly CAB meetings, the meeting notes and transcripts were systematically categorized according to the previously outlined topics (i.e., alignment with lived and professional experience, data interpretation and clarity in how data was presented, and next steps for data analysis or data collection). Then, action items, including edits, were identified before the next call. Using a participatory approach, each CAB feedback session began with a debrief of the previous meeting, presenting key highlights from the notes and how their input directly shaped revisions to the FS-EAT draft. All clarifications or amendments to the researcher’s interpretations were noted in the meeting minutes and reconciled during the subsequent round of edits.

3.3. Tool Evaluation: Interrater Reliability

Reliability of the FS-EAT tool was evaluated by measuring interrater agreement using Cohen’s kappa statistic. Pairs of trained raters independently audited stores, and kappa values were computed for the full tool and its subcomponents using Python 3.8.5 [56]. A pooled kappa score of ≥0.40 was considered an acceptable level of agreement.

3.4. Measures

3.4.1. Sensitivity and Positive Predictive Value (PPV)

Sensitivity and positive predictive value (PPV) were calculated to assess the accuracy of the secondary database compared with direct ground observations. Sensitivity measures how often the NETS database identified food sources when they were present, and PPV measures how often food sources were present when the secondary database reported their existence.

• Sensitivity = TP / (TP + FN)
• PPV = TP / (TP + FP)

3.4.2. Neighborhood-Level and Block-Group Level Food Swamp Indices

The three food swamp indices (FSI, PFEI, and mRFEI) were computed from store counts at two geographic levels (neighborhood- and block-level) using formulas including:

Neighborhood-level indices (audit store counts aggregated by neighborhood): Neighborhood-level indices were calculated from audit data by aggregating store category counts across each of the six study neighborhoods (Clay Arsenal, Northeast, Upper Albany, Asylum Hill, Frog Hollow, and Barry Square) and by study site designation (North Hartford Promise Zone vs. comparison sites). These neighborhood-level scores were compared with resident-perceived food swamp exposure scores.

Block group–level indices (audit and NETS store counts via spatial join): Block group–level indices were calculated for both the audit data and the secondary NETS data. Using the geocoded locations of food outlets, store data points were spatially joined to 2020 Hartford census block groups in ArcGIS Pro using a one-to-one spatial join with the intersect match option [57]. This produced counts of healthy, unhealthy, and intermediate stores per block group for each data source. For classification purposes, block groups with FSI > 50 were designated as food swamp areas and those with FSI ≤ 50 as non-food swamp areas. Choropleth maps were generated at the block group level using equal interval classification (three classes: 0 - 33.33, 33.34 - 66.67, 66.68 - 100.00) to facilitate direct visual comparison between datasets. A bivariate choropleth map was also produced to display the combined audit-derived and NETS-derived FSI simultaneously. The GIS maps and histograms resulting from these analyses were assessed through iterative feedback sessions with CAB members.

3.5. Statistical and Geospatial (GIS) Analyses

Statistical analyses were performed using STATA version 19. The threshold of statistical significance was set at p < 0.05 for all two-sided tests.

3.5.1. Ground Truthing (Validation against Secondary Data)

Statistical analyses were performed using STATA version 19. The threshold of statistical significance was set at p < 0.05 for all two-sided tests.

3.5.1. Ground Truthing (Validation by Secondary Data)

To assess data accuracy and completeness, descriptive statistics were used to compare food outlet counts from the FS-EAT audit (primary dataset) with listings from the 2020 NETS database (secondary dataset). Comparisons included total store counts and category classifications: Healthy, Intermediate, and Unhealthy food outlets.

To determine the match between the primary and secondary datasets, a systematic matching process was applied using two criteria: business name similarity and geographic proximity. Business name similarity was assessed by comparing the spelling of store names across the two datasets, yielding a similarity score ranging from 0 (completely different) to 1 (identical match). Geographic proximity was determined by calculating the straight-line distance in meters between the geocoded coordinates of each store pair. For the sensitivity and PPV analysis, stores were considered matched (True Positives) if they met a name similarity score of ≥ 0.6 combined with a distance of ≤ 100m, or a name similarity of ≥ 0.8 regardless of distance, or a distance of ≤ 50m with a name similarity of ≥ 0.4. Stores found in the primary dataset but not matched to any secondary listing were classified as false negatives (FN), while secondary listings not matched to any audit store were classified as false positives (FP). For the extended matching analysis, additional categories were defined: confirmed match (name similarity ≥ 0.8 and distance ≤ 100m), likely match (name similarity ≥ 0.6 and distance ≤ 100m), proximity match (distance ≤ 50m and name similarity ≥ 0.4), name match distant (name similarity ≥ 0.8 and distance > 100m), and possible match (name similarity ≥ 0.6 with weaker geographic evidence).

3.5.2. Agreement Between Objective Primary FS-EAT and Secondary NETS Data Food Swamp Measures

Food swamp indices (FSI, mRFEI, and PFEI) were computed at the census block group level for both the primary audit data and the secondary NETS data, as described in Section 3.4.1. To compare food swamp classification between the two data sources, each block group was classified as either a food swamp (FSI > 50) or a non-food swamp (FSI ≤ 50), and a cross-tabulation was generated to assess alignment.

To assess convergent validity, several complementary agreement statistics were computed at the block group level (n=39). Spearman rank-order correlations were used to assess the association between audit-derived and NETS-derived food swamp indices and store category counts, given the non-normal distribution of the scores. The three food swamp indices (FSI, mREI, and PFEI) were compared. In addition, correlations were computed for store category counts (healthy, unhealthy, and intermediate).

Cohen’s kappa was computed on the binary food swamp classification (FSI > the mean (74.8) to quantify agreement beyond chance. The intraclass correlation coefficient (ICC; two-way random, single measures, absolute agreement) was calculated to assess whether the two methods yielded the same FSI values, rather than merely consistent rankings. A Wilcoxon signed-rank test was used to assess whether the paired audit-derived and NETS-derived FSI scores differed systematically across block groups.

Additional GIS analyses were conducted to compare the spatial distributions of food swamp exposure between the two data sources using side-by-side choropleth maps and a bivariate choropleth map displaying both indices simultaneously.

4. Results

4.1. Tool Evaluation Results

4.1.1. Validity Testing

The FS-EAT tool was validated with: 1) a key stakeholder survey (content validity); 2) feedback sessions with the Community PIs and CAB (face validity); and 3) Ground truthing compared to NET 2020 data (convergence validity).

4.1.2. Key Stakeholder Survey

Among the 26 stakeholders surveyed, 80% (n = 16) rated FS-EAT items as either "relevant" or "extremely relevant" (scores of 3 or 4 out of 4). All 19 items, spanning food outlet type, marketing, and social factors, were retained in the final version of the tool.

4.2. Interrater Reliability Test

Regarding interrater reliability results, the overall pooled kappa score for the tool was 0.82, and the sub-component kappa scores were 0.90 for street-level store count, 0.61 for store-level social features, 0.90 for store-level beverage marketing exposure, 0.53 for fast-food restaurant accessibility, and 0.83 for pricing.

4.3. Ground Truthing Compared to Secondary NETS Data

4.3.1. Descriptive Analysis of Food Store Audits

Store-Level Comparison

A total of 236 food retail outlets were identified in the secondary dataset (NETS, 2020) across the six study-site neighborhoods. Ground-truthing using the FS-EAT audit identified 168 food outlets, of which five stores were no longer in operation, five were nonexistent, one was located in a restricted-access building, and one had been converted to a non-food establishment. As a result, 157 stores (92.9%) were successfully audited using the FS-EAT tool. Of the 157 active stores, 38 (24.2%) were classified as healthy food outlets, 50 (31.8%) as intermediate food outlets, and 69 (43.9%) as unhealthy food outlets. By comparison, the NETS 2020 secondary data identified 88 (37.3%) healthy, 82 (34.7%) intermediate, and 66 (28.0%) unhealthy food outlets among its 236 stores. Considerable discrepancies emerged between the two data sources. The NETS 2020 data identified substantially more healthy food outlets (88 vs. 38) and more intermediate outlets (82 vs. 50), while the FS-EAT audit data recorded slightly more unhealthy outlets (69 vs. 66) despite having fewer total stores. Table 3 presents food store counts by classification: Healthy, Intermediate, and Unhealthy, based on the typology shown in Table 2.

Comparison of Audit and NETS 2020 Data

To assess the validity of the FS-EAT audit data, we compared it against the NETS 2020 business list using business name and geographic proximity. Of the 157 active audit stores, 107 (68.2%) were matched to a corresponding NETS 2020 entry (True Positives). The remaining 50 audit stores (31.8%) were not found in the NETS 2020 data (False Negatives), while 129 NETS 2020 listings had no corresponding ground-truthed store (False Positives). Table 4 presents the concordance between the FS-EAT audit and NETS 2020 data.

To further explore the extent of overlap between the two datasets, we applied additional matching criteria beyond the initial name-similarity and proximity thresholds used in the PPV analysis. Table 5 presents the results of this extended matching. Of the 157 audit stores matched against the 236 NETS 2020 stores, 5 (3.2%) were identified as likely matches with moderate name similarity (≥ 0.6) within 100m, and 12 stores (7.6%) were matched by proximity alone (within 50m with name similarity ≥ 0.4). Thirty-seven stores (23.6%) matched by name but were geographically distant, and 53 (33.8%) were classified as possible matches based on moderate name similarity with weaker geographic evidence. In total, 107 stores (68.2%) were matched between the two datasets. Fifty audit stores (31.8%) had no corresponding match in the NETS 2020 data, while 129 NETS 2020 stores (54.6%) were not found in the audit data.

Table 6 presents sensitivity and PPV results by store type from the perspective of the FS-EAT audit tool, In this analysis, sensitivity represents how often the FS-EAT audit confirmed a food outlet when the NETS 2020 database reported its existence (sensitivity = TP/[TP+FP]), and PPV represents how often an FS-EAT-identified food outlet was also listed in the NETS 2020 database (PPV = TP/[TP+FN]). Overall, the FS-EAT tool demonstrated a sensitivity of 0.453 (45.3%), indicating that fewer than half of NETS 2020 listings were confirmed as active food outlets through ground observation. Sensitivity was highest for convenience stores (74.4%), followed by grocery stores (45.2%), and lowest for limited-service restaurants (29.2%) and specialty food outlets (29.4%), suggesting that NETS substantially overrepresented these categories. The FS-EAT tool demonstrated an overall PPV of 0.682 (68.2%), meaning that more than two-thirds of stores identified through ground observation were also present in the NETS database. PPV was highest for grocery stores (86.8%) and full-service restaurants (84.6%), indicating strong overlap between the audit and secondary data for these outlet types. In contrast, limited-service restaurants had the lowest PPV (26.9%), with 19 of 26 audit-identified outlets absent from the NETS data, highlighting the FS-EAT tool’s ability to capture food outlets that commercial databases miss.

4.3.2. Block Group–Level Comparison of Food Swamps Constructed with Secondary (NETS) vs. Primary (Audit) Data

4.3.2.1. NETS-Based Food Swamp Index

Table 7 presents the summary statistics of food swamp scores from the Audit and NET Data. The secondary NETS data FSI exhibited a mean of 61.31 and a median of 64.00, indicating generally elevated food swamp conditions across the 39 study block groups within the study area. The distribution was slightly left-skewed (skewness = −0.51), indicating that although several block groups had very high FSI values, the overall distribution remained relatively balanced. FSI values ranged from 0 to 100, with most block groups (74.4%) clustering between 40 and 90. Using the audit-derived FSI mean (74.82) as the classification threshold, the majority of block groups (25; 64.1%) were classified as non-food swamps (FSI 0.00–74.81), and 14 (35.9%) were classified as food swamps (FSI 74.82–100.00).

4.3.2.2. Audit-Food Swamp Index

The audit-derived FSI had a higher mean (74.82) and median (75.00) than the secondary NETS data-derived measure. The distribution was more strongly left-skewed (skewness = −1.23), showing a greater concentration of block groups at the upper end of the FSI range. Using the same classification threshold (FSI ≥ 74.82), a larger proportion 23(59.0%) of block groups were classified as food swamps (FSI 74.82–100.00) compared to only 35.9% under the NETS data., 23 (59.0%) of block groups were classified as food swamps (FSI 74.82–100.00) compared to only 35.9% with the NETS data.

4.3.2.3. Comparative Assessment of Food Swamp Classification by Block Group

To compare food swamp classification between the primary (audit) and secondary (NETS 2020) data sources, each block group was classified as either a food swamp or a non-food swamp using the audit-derived FSI mean (74.82) as the classification threshold. Block groups with an FSI at or above the threshold were designated as food swamps. Because the audit mean and median were nearly identical (74.82 and 75.00, respectively), both thresholds produced the same classification results. Appendix Table 1 presents the classification for each of the 45 block groups in the study area. Six of the 45 block groups were excluded because they lacked food outlet data from one or both sources, three had no food outlets identified by either data source, and three had NETS-listed stores but no audit-confirmed outlets. Of the 39 block groups with data from both sources, 18 (46.2%) had aligned classifications, and 21 (53.8%) did not. Using the audit-derived threshold, the audit classified 23 block groups (59.0%) as food swamps compared to only 14 (35.9%) in the NETS data.

A cross-tabulation shows a systematic pattern of disagreement between the two data sources (Table 8). While both sources agreed on the food swamp classification in 18 block groups (8 both food swamp and 10 both non-food swamp), the predominant direction of misalignment was that the NETS data underestimated food swamp exposure. 15 block groups (38.5%) were classified as food swamps by the FS-EAT audit data but as non-food swamps by the NETS data. Only 6 block groups (15.4%) showed the reverse pattern (i.e., non-food swamps per FS-EAT data classified as food swamps by NETS).

4.3.2.4. Visual Comparison

Figure 2 presents comparisons of Food Swamp Index (FSI) values based on FS-EAT vs. NETS data. A visual comparison of the two maps indicates considerable differences in the spatial patterning of food swamp exposure. The NETS-derived map (left panel) shows a greater mixture of FSI categories across the study area, with several block groups in the low-to-moderate range (light pink), particularly in the western and southern regions of the study area. In contrast, the audit-derived map (right panel) displays a more uniformly high FSI pattern, with the majority (59.0%) of block groups classified in the highest category (dark red, 75.0–100.00). This visual pattern is consistent with the finding that the audit-derived mean FSI (74.82) was substantially higher than the NETS-derived mean FSI (61.31), indicating that the FS-EAT identified a greater concentration of unhealthy and intermediate food outlets relative to total food outlets.

Several block groups that appeared as moderate food swamp areas in the NETS data were classified as high food swamp areas based on the audit data, particularly in the central and southern portions of the study area corresponding to the Frog Hollow and Barry Square neighborhoods. Conversely, some block groups in the northern portion of the study area (Upper Albany and Clay Arsenal) showed high FSI values in both data sources, suggesting greater concordance in areas with the most concentrated food swamp conditions. The audit-derived map also shows more block groups with no food outlets (grey areas), indicating a smaller total number of stores identified through the FS-audit observations compared to the NETS secondary data.

Figure 3 presents a bivariate choropleth map that simultaneously displays the NET-derived FSI and audit-derived FSI for each block group, providing a direct visual representation of concordance between the two data sources. In this map, the color of each block group reflects the combined levels of both indices: dark purple indicates block groups where both data sources assigned high FSI values (high concordance), pink indicates block groups where the NETS data assigned a high FSI but the audit data assigned a low FSI (NETS overestimation), and cyan/light blue indicates block groups where the audit data assigned a high FSI but the NETS data assigned a low FSI (NETS underestimation). Lavender areas represent concordant low FSI values from both sources.

The bivariate map reveals a predominantly cyan/light blue pattern across much of the study area, visually supporting the cross-tabulation finding that the largest source of disagreement was NETS underestimation of food swamp exposure (15 block groups, 38.5%). A few block groups displayed dark purple, most prominently at the northern tip of the study area (Northeast neighborhood), indicating that both data sources identified these areas as having high food swamp exposure. Fewer block groups appear in pink, indicating areas where the NETS data assigned higher FSI values than the audit, consistent with the cross-tabulation finding that only 6 block groups (15.4%) showed this reverse pattern. The lavender block groups represent areas where both sources agreed on lower food swamp conditions. The overall spatial pattern of inconsistency, with a statistically insignificant weak negative correlation between the two data sources (ρ = −0.19, p = .2452), demonstrates the importance of ground-truthing secondary food environment data, particularly in high-poverty urban neighborhoods where the food retail environment may change rapidly and is characterized by a variety of non-traditional food outlets.

4.3.3. Statistical Agreement and Correlation Between Block-Group-Level Primary and Secondary Food Swamp Measures

To quantify the level of agreement and correlation between the two data sources, we computed four complementary statistics, Spearman’s rank correlation, Cohen’s kappa, the intraclass correlation coefficient (ICC), and the Wilcoxon signed-rank test, across all food swamp indices and store category counts. Table 9 presents complete results.

Food Swamp Indices

The FSI derived from NETS 2020 data showed a weak negative correlation with the audit-derived FSI (ρ = −0.19, p = .2452), indicating poor convergence between the two measures. The mRFEI yielded identical results (ρ = −0.19, p = .2452), as the two indices are mathematically complementary (FSI = 100 − mRFEI). The PFEI showed a similar weak negative correlation (ρ = −0.18, p = .2852). None of the index correlations were statistically significant, and the negative direction indicates a slight tendency for block groups that ranked higher on food swamp exposure using one data source to rank lower using the other.

Cohen’s kappa, computed on binary classifications using audit-derived means as thresholds, confirmed this lack of agreement. For the FSI and mRFEI, κ = −0.03 (p = .5691), indicating less than chance agreement. The PFEI showed slightly better but still negligible agreement (k = 0.09, p = .2062). These results suggest that the NETS 2020 and audit-derived food swamp classifications were independent at the block-group level.

The ICC for absolute agreement was negative for all three indices: −0.18 for the FSI and mRFEI (F (38, 38) = 0.68, p = .883) and −0.11 for the PFEI (F (38, 38) = 0.76, p = .798). The negative ICC values indicate that between-method variability exceeded between-block-group variability, confirming that the two data sources produced fundamentally discordant food environment scores.

Wilcoxon signed-rank tests revealed a consistent pattern of directional disagreement. The audit-derived FSI (mean = 74.82) was higher than the NETS-derived FSI (mean = 61.31), approaching but not reaching significance (z = 1.878, p = .0603). The mRFEI showed the same pattern in reverse (z = −1.892, p = .0583). Notably, the PFEI showed a statistically significant difference (z = 2.656, p = .0070), indicating that the audit identified a significantly higher density of unhealthy food outlets than the NETS data at the block-group level.

Store Category Counts

Among the store category counts, intermediate food outlets demonstrated the strongest convergence between the two data sources across all analyses. The Spearman correlation was moderate and statistically significant (ρ = 0.39, p = .0153), and the ICC reached 0.43 (F(38, 38) = 2.81, p = .001), the only measure to achieve a statistically significant ICC. However, despite this relative agreement, the Wilcoxon test revealed that the NETS data identified significantly more intermediate outlets than the audit data (z = −2.915, p = .0029).

Healthy food outlet counts showed a marginally non-significant ICC of 0.20 (p = .052) and a negligible Spearman correlation (ρ = 0.09, p = .6014), with a highly significant Wilcoxon test (z = −3.501, p = .0003), indicating that the NETS data identified significantly more healthy stores than the audit. Unhealthy food outlet counts showed poor agreement (ICC = 0.11, p = .262) and a weak non-significant Spearman correlation (ρ = 0.16, p = .3200), but no significant difference in overall counts between the two sources (z = 0.188, p = .8384), showing that although the two data sources did not agree on which block groups had more unhealthy outlets, they produced similar overall counts at the block group level.

4.4. Feedback Sessions with Community PIs and the CAB

First, regarding tool development, the CAB validated the importance of including ‘nontraditional’ food outlets such as gas stations and dollar stores. Second, regarding tool evaluation, the CAB confirmed that the maps generated using primary FS-EAT audits were a more timely and nuanced representation of food swamp exposure in the study sites than those using secondary NETS data. Face validity was confirmed through review of FS-EAT-generated maps by Community PIs and CAB members. The CAB, local leaders and residents confirmed that these GIS maps aligned with their lived experiences in the neighborhoods.

5. Discussion

With the growing interest in community-based interventions for health promotion, a robust neighborhood-level evaluation tool is needed to measure exposure to food swamps. Leveraging CBPR, we developed and validated the FS-EAT for use in low-income urban neighborhoods. Overall, results support the FS-EAT as a feasible tool with strong interrater reliability, content validity, and face validity. Notably, our study also highlights some important nuances in measurement performance across subcomponents of the tool, such as identifying subcategories of grocery retailers and restaurants that are salient in urban food swamps. Ultimately, our results prove the value of the FS-EAT tool as a primary data collection instrument that captures a more complete and accurate picture of neighborhood food swamp exposure than NETS commercial food store data alone. Key findings from the current project will help inform future public health interventions and policy efforts that require standardized, evidence-based approaches for identifying food swamp neighborhoods and tracking related place-based health disparities.

The inter-rater reliability scores for the subcomponents and the complete tool indicated a range from moderate to excellent agreement among observers across all items. The moderate to high kappa statistic suggested that raters had little difficulty following the tool's training instructions. Additionally, content validity was established through a stakeholder survey, which confirmed that the food outlet types included in the FS-EAT were highly relevant to neighborhood-level food environment assessments. In addition, feedback sessions with the study’s CAB confirmed that the geospatial representation of food swamp exposure based on FS-EAT data aligned with their lived experiences in the neighborhoods and was more timely and accurate than those generated using NETS secondary food store data. The present study also found that food swamp exposure varied across all block groups within the 6 study neighborhoods.

Key findings from our analyses examining the convergent validity of block group-level food swamp identification suggest that the FS-EAT-based assessment captured a higher food swamp intensity and a slightly higher concentration of unhealthy food retailers (e.g., fast-food restaurants, gas stations selling food) than the secondary NETS dataset. Regarding our comparative assessment of how NETs vs. FS-EAT data classify block groups as food swamps vs. non-food swamps, the audit-based measure consistently indicated higher levels of unhealthy retail concentration and greater clustering in the high food swamp category. In particular, 15 block groups (38.5%) were classified as food swamps by the FS-EAT audit data but as non-food swamps by the NETS data. Thus, the predominant direction of misalignment was the NETS data underestimating food swamp exposure. Our ground-truthing findings suggest that this result may be driven by the overreporting of healthy food outlets and the underreporting of unhealthy food outlets, such as limited-service restaurants and corner stores.

Our results are consistent with prior store-level validation research demonstrating that secondary commercial data sources systematically undercount food retailers relative to ground-truth observations. Lucan et al. compared Infogroup business list data against direct street-level observation from the Bronx (New York City) and reported an overall sensitivity of only 39.3% and PPV of 45.5% under strict matching criteria, indicating that the commercial list both missed a substantial proportion of food outlets actually present on the ground and included listings for outlets that no longer existed [28]. Similarly, Caspi and Friebur found that commercial business listings across three rural sites in Minnesota had an average PPV of 0.57 and a sensitivity of 0.62, further showing the limited accuracy of secondary datasets [29]. While these earlier studies evaluated store-level concordance between commercial lists and field observations, the present study expands this work by comparing secondary and primary data at the block group level to identify food swamps versus non-food swamps, a compositional, area-level determination in addition to store-by-store matches. Nonetheless, the directionality of misalignment observed here, with the NETS data underestimating food swamp exposure in 38.5% of block groups, corresponds to the pattern of underdetection documented in prior validation work [32,33], reinforcing the concern that reliance on secondary commercial datasets alone may produce systematic underestimation of food swamp exposure.

This pattern suggests that primary audit data may capture dimensions of retail healthfulness not fully reflected in secondary business listing data. Finally, our statistical findings reinforced our GIS mapping results, providing converging evidence that the NETS 2020 business list and the FS-EAT audit produce fundamentally discordant assessments of the food environment at the block group level. The weak negative Spearman correlations for the composite indices are consistent with the negative ICC values and non-significant kappa statistics. The Wilcoxon tests reveal a systematic directional bias: the NETS data consistently underestimate food swamp exposure relative to the audit for the FSI and PFEI, while overestimating healthy and intermediate store counts. Among all food store classifications (i.e., unhealthy, healthy, and intermediate), intermediate food outlets (e.g., sit-down restaurants) showed the strongest agreement between the two data sources, with the highest Spearman correlation and the only statistically significant ICC. Overall, our results illustrate the value of primary audit tools such as the FS-EAT for accurate characterization of neighborhood food swamps and reveal a few data limitations of relying solely on secondary business listing data for food swamps assessments. As discussed above, this pattern of NETS underestimation is consistent with the low sensitivity and PPV values reported across prior food environment validation studies [28,29,32]. However, the present study uniquely demonstrates these discrepancies at the neighborhood level for food swamp classification rather than at the level of individual store identification. These collective findings underscore the need for researchers and policymakers to incorporate primary audit tools when characterizing food swamp exposure, particularly in low-income communities of color, where inaccuracies in secondary data may compound existing measurement gaps in documenting place-based health inequities.

By linking standardized outlet-level observations to neighborhood-level food swamp classification, FS-EAT advances existing approaches to observational food environment assessment. Unlike the NEMS instruments, which provide detailed and validated assessments of food availability, quality, and pricing within individual food store types (e.g., grocery stores, corner stores, and restaurants) [25,26,58], FS-EAT situates these outlet-level characteristics within the broader composition of the neighborhood retail environment. This makes it possible to examine not only what is available within individual outlets, but also how healthy, unhealthy, and intermediate outlets are distributed relative to one another, a defining feature of food swamp exposure [9,11]. FS-EAT also builds on the attention to racially and ethnically diverse urban settings reflected in the Food Environment Audit for Diverse Neighborhoods (FEAD-N) [41] by incorporating a CBPR process through which community stakeholders identified the importance of measuring limited/quick service restaurants that are not national chains and nontraditional food stores such as gas stations and dollar stores. Its inclusion of the full range of store and restaurant categories further extends the set of tools, such as the Healthy Food Availability Index (HFAI), which has been used in Baltimore food swamp research but focuses on healthy food availability within stores and does not capture fast-food, carry-out, or sit-down restaurants [59]. Overall, these features position FS-EAT as a complementary advancement in food environment measurement: it retains the systematic observational strengths of existing tools while integrating lived experiences in the community and geographic aggregation to characterize the relative concentration and contextual features of food outlets that shape neighborhood-level food swamp exposure.

An important strength of this study is that FS-EAT was developed with input from community stakeholders, including Community PIs and a CAB, who provided content relevant to low-income, ethnically diverse community settings. Thus, findings from this project will be highly generalizable to other low-income urban neighborhoods across the country. Second, the FS-EAT tool offers the opportunity to present time-relevant data compared to secondary datasets, which may be lagged by at least 1 to 2 years. Third, FS-EAT offers a systematic approach to compiling nuanced features (e.g., marketing, produce sections within small food retailers) of the neighborhood food environment that are tied to health-promoting behaviors but have not been captured in previous food environment audit tools [40]. Fourth, data compiled via the FS-EAT can be translated to easy-to-understand formats such as food swamp maps or street-level food swamp scores that can be aggregated to any geographic unit of interest (e.g., block group, census tract, neighborhood by name or boundary, county, state, etc.) to inform decision making, policy planning, and food environment interventions spearheaded by researchers, policymakers, or other key stakeholders. Further, the FS-EAT reflects diverse retail settings by measuring nontraditional markets and culturally distinct food outlets. For instance, the FS-EAT facilitates the identification of healthy vs. unhealthy ethnic specialty stores and restaurants, rather than classifying all ethnic stores as unhealthy. This will help strengthen the tool’s generalizability and can meaningfully inform policy and practice outcomes across diverse populations. Finally, considering accessibility, the FS-EAT makes it more feasible for smaller community-based organizations that may not have large budgets to purchase large secondary datasets (and perhaps even multiple datasets per Lierse et al) to accurately identify food swamps for research, advocacy, and program evaluation.

Still, this current study has some limitations that should be considered. The current version of the FS-EAT tool does not collect detailed information about hours of operation. This was not feasible in the first round because of concerns about the time and staff resources required for in-store data collection. Based on our Community PI’s concerns about the resources allocated to add a mystery shopper protocol to distinguish small food retailers with and without produce sections, hours of operation were not initially included to limit the time research team members would need to spend in the store. Relatedly, we wanted to limit the data collection to details that could not easily be gathered online or via telephone. Future iterations of this tool should leverage the existing mystery shopper protocol to measure hours of operation as another dimension of food access. Also, our testing only included urban neighborhoods, which could restrict its applicability to areas with different food access challenges such as suburban or rural areas. Future studies should test the FS-EAT tool in rural and suburban food retail landscapes to enhance its generalizability.

Regarding future work, psychometric testing, including a validation test comparing the FS-EAT with existing food environment assessment tools such as the Food Retail Outlet Survey Tool, may be warranted [53]. Also, future research should expand the mystery shopper protocol to collect additional store inventory or menu information as well as residents’ interactions with food swamps using approaches such as receipt data, photovoice, or focus group discussions. Future iterations of the tool could also develop a scoring system that incorporates not only outlet type but also social features, accessibility, pricing, and food and beverage marketing to provide a further enhanced measure of food swamp exposure. Furthermore, while this study demonstrates the FS-EAT tool's ability to measure food swamp exposure, it does not establish clear linkages to health outcomes or policies. Future longitudinal studies could focus on correlating the FS-EAT's findings with public health interventions such as neighborhood revitalization or zoning policy changes.

Although FS-EAT provides an objective, audit-based measure of neighborhood food swamp exposure, a complementary subjective measure is needed to capture how residents perceive and experience these environments. Perceived food swamp measures may reflect dimensions of accessibility, affordability, safety, convenience, and cultural relevance that objective outlet counts and environmental observations alone cannot fully represent [11,38]. Prior research documenting racial differences in perceived exposure to food swamps and food deserts, as well as associated disparities in self-reported dietary habits, further supports the importance of measuring this experiential dimension of the food environment [11]. Accordingly, the next phase of this work will focus on developing and validating a subjective food swamp measure that parallels FS-EAT and can be linked at various levels of the neighborhood food environment (e.g., block group, census, zip code). Used together, the two instruments could distinguish between objectively observed food swamp conditions and residents' lived experiences of those conditions, identify areas of convergence or mismatch, and provide a more comprehensive foundation for community-informed interventions.

6. Conclusions

The current study found that the FS-EAT tool is content valid and has higher interrater reliability. Importantly, the FS-EAT tool categorizes food stores based on more nuanced characteristics and more timely information about whether they are open for business. It also provides a novel means to comprehensively measure neighborhood-level food swamp exposure, which can be used to identify priority action areas. The tool is intended for use by researchers, public health practitioners, policymakers, and community-based organizations, with standardized training recommended to ensure reliable implementation. Notably, even with the required investments in training data collectors, we anticipate that the FS-EAT will be a more economical, systematic approach for resource-limited community-based organizations and public health departments to identify food swamps for policy prioritization and investment.

Thus, the FS-EAT can be applied to the development, implementation, and monitoring of land-use zoning policies targeting food swamps (e.g., zoning restrictions, stocking standards) as well as to other public health interventions or resource allocations. Without equity-centered planning, urban food environments risk deepening inequities through processes like gentrification and exclusionary zoning [12,39]. Future research could further incorporate pricing and cultural sensitivity into the tool by addressing identified limitations such as biases toward ethnic food stores. Future research will retest the tool in suburban and rural environments and further refine the technology to ease the burden of data collection. The long-term research goal for this line of research is to utilize the FS-EAT tool to better understand the effect of food swamp environments on diet, food purchasing patterns, diet-related health inequities, and the efficacy of macro-level (zoning changes, addition of supermarkets, neighborhood development) and micro-level nutrition interventions (i.e., in schools, food pantries, child care settings, fast food restaurants) and policy changes (i.e., introduction of a supermarket, zoning law changes, etc.) to alleviate them.