Automatic Estimation for Visual Quality Changes of Street Space Via Street-View Images and Multimodal Large Language Models

1. Introduction

Urban landscapes, encompassing a tapestry of streets, parks, and public squares, are more than mere channels of movement; they are the lifeblood of city vibrancy and communal interaction. Within this urban fabric, the aesthetic, functional, and social dimensions of street spaces emerge as pivotal to public well-being and community cohesion [1]. They are not just static structures but vibrant stages where the urban narrative unfolds over time. The temporal evolution of street spaces is critical to understanding and fostering urban vitality, as they are the sites where the past converses with the present and shapes the future [2]. This recognition has sparked a series of urban design initiatives [3], underscoring the global movement towards enhancing the Visual Quality of Street Space (VQoSS) [4].

The concept of VQoSS is not a contemporary invention. Its origin can be traced back to the advocacy for community-centric urban design in the mid-20th century [5]. Despite their contributions, historical limitations in data collection confined such research to the realm of small-scale studies, impeding the development of empirically grounded urban design principles that could be applied broadly. Urban planners face the complex challenge of estimating VQoSS changes as they endeavor to shape environments that align with human experiences amid the continuous progression of urban development [6].

In an era where technological advancements have revolutionized data accessibility and analytical capabilities, tools like Geographic Information Systems (GIS) for street-view images [7,8], remote sensing technologies [9,10], and deep learning based methods [11,12] present new opportunities for comprehensive urban analysis. They introduce a refined methodology for evaluating the visual quality of street spaces, both in their current state and across temporal shifts, quantifying and analyzing the temporal changes of VQoSS with unprecedented precision. By leveraging image processing and machine learning, we can dissect the physical and perceived visual qualities of urban thoroughfares, even applying our techniques to the storied alleyways of historic districts. People can examine the past, present, and potential futures of urban streets, laying the groundwork for methodologies that blend tradition with innovation in pursuit of urban spaces that are not only functional but also inviting.

Previous works [4,13,14] incorporate the ability of deep learning in computer vision, offering solutions for a variety task of VQoSS. However, current deep learning technologies face several limitations. One of the primary challenges is the inherent difficulty in capturing the subjective elements of aesthetics and social vibrancy that constitute the essence of street quality. Moreover, deep learning models require vast amounts of labeled data to learn effectively [15,16]. The labor-intensive process of gathering and annotating temporal visual data from urban environments can be prohibitive, often resulting in models that may not generalize well across different cities or cultural contexts [17]. Lastly, environmental and seasonal variations, along with the dynamic nature of urban life, introduce a level of complexity that deep learning models, with their need for stable and consistent patterns, struggle to accommodate. It seems that only using vision-related technology can not satisfy the real-world demands of VQoSS tasks [18]. This leads to a question: Can we design a method that can efficiently and accurately realize the large-scale VQoSS change estimation tasks without any manual annotation?

Recently, Large Language Models (LLMs) [19,20] have shown their power in a variety of complex language understanding and generation tasks, fundamentally altering the landscape of natural language processing. These models, trained on expansive corpora of text, excel at grasping the nuances of human language, from writing and summarizing articles to generating coherent and contextually relevant text. This ability has been further extended to Multimodal Large Language Models (MLLMs) [21], achieving outstanding accomplishments. Models like GPT-4 [22] can now not only understand and generate text but also interpret and analyze visual data. This integration of vision and language processing enables them to provide answers to questions about the content of images, effectively bridging the gap between visual perception and linguistic expression [23,24]. By processing and understanding complex visual scenes, these models can identify objects, recognize patterns, and even infer emotions or actions depicted in images [24,25,26]. We can directly use them for the purpose of VQoSS estimation. Although GPT-4 is powerful, using its API is expensive [27], and its structure is still unknown, which makes it impossible for large-scale VQoSS change estimation.

In this study, we investigate the potential for automating the estimation of VQoSS changes using street-view imagery and MLLMs. Our analysis is centered on the urban center of Nanjing, China. We collected a dataset of 10,598 data points along main thoroughfares, capturing images from four cardinal directions (0°, 90°, 180°, and 270°) at each location for two different periods. We also set 9 different VQoSS change evaluation tasks to quantify the performance of MLLMs. Our findings indicate that models such as GPT-4 exhibit remarkable proficiency in discerning temporal street variations for VQoSS evaluation. Moreover, as GPT-4 is not free in use, in order to reduce the cost for large-scale estimation, we introduce the Street Quality GPT (SQ-GPT) for VQoSS change estimation, which can efficiently distill [28,29] GPT-4’s knowledge. Through rigorous experimentation, we demonstrate that the performance of SQ-GPT is close to GPT-4, validating its utility as a cost-effective instrument for large-scale urban street quality surveillance.

We summarized our contributions as follows: (1) Our study pioneers the integration of MLLMs into the automation process for estimating changes in the VQoSS, offering a significant tool for urban research advancements. (2) We have identified 9 critical VQoSS tasks and have rigorously evaluated the effectiveness of MLLMs through both quantitative measures and qualitative analysis. (3) We introduce SQ-GPT, an innovative model that demonstrates performance on par with GPT-4, providing an alternative for large-scale VQoSS estimation tasks.

2. Related Works

2.1. Visual Quality of Street Space

The requirements to quantify the VQoSS changes in urban spaces date back to the latter half of the 20th century. Pioneering works [5] emphasized the importance of street life and social interactions in urban spaces. They underscored the need for empirical studies to connect physical appearance with social attributes. Early efforts were largely qualitative due to the challenges of quantitative data collection over large areas [30].

In the end of last century, there seems to be a shift towards more systematic methodologies for assessing the visual quality of street spaces [31,32]. Researchers began employing a mix of subjective assessments, systematic observations, and technological advancements such as GIS [33] for more objective measurements. Studies offered structured approaches to evaluate the physical and perceptual dimensions of urban spaces. Recent years have marked the introduction of advanced technologies into the realm of urban space assessment. The utilization of Street View Imagery (SVI) [7,34,35] and deep learning algorithms [4,13,14,36], for instance, has allowed for large-scale quantitative analyses [37]. This integration has propelled the field from qualitative descriptions to data-driven, automated evaluations, as evidenced in works [38] that leveraged SVI for extensive urban analysis [39].

Tang and Long [4] built upon this progression by offering a comprehensive methodology that combines physical and perceived evaluations of visual quality using machine learning (SegNet [40]), GIS analysis, and systematic user-based assessments. Their work stands at the intersection of urban planning, computer science, and psychology, showcasing a multifaceted approach to understanding the temporal dynamics of street space quality in historical areas like Beijing’s Hutongs. The current trajectory of research suggests a continuous refinement of methodologies with an increasing emphasis on integrating user perceptions and technological sophistication [1,7]. Upcoming studies will likely focus on enhancing the objectivity in measuring visual quality [41,42], fine-tuning algorithmic approaches [43], and incorporating real-time data to capture the ever-evolving nature of urban spaces [44]. This evolution reflects a broader trend in urban planning towards smart, data-informed decision-making that prioritizes both the physical attributes and the experiential qualities of urban environments. Although previous studies have achieved promising results, large-scale estimation of VQoSS changes is still labor-intensive. Especially methods based on deep learning, their accuracy requires a large amount of data annotation.

2.2. Multimodal Large Language Models

In recent years, the field of multimodal learning [21,45] has experienced rapid development, with researchers striving to construct artificial intelligence models capable of comprehending and generating multiple types of data, including text, images, videos, and more. In this section, we provide a concise overview of the relevant literature that informs our work.

To empower LLMs with the ability to interpret images and engage in user interactions based on visual inputs, a multitude of methods have emerged. These methods aim to establish connections between frozen visual encoders and LLMs while keeping the number of trainable parameters minimal. Prominent approaches in this category encompass BLIP-2 [23], InstructBLIP [46], LLaMA-Adapter [47], LLaMA-Adapter V2 [48], LLaVA [25], MiniGPT-4 [26], MultiModal-GPT [49], and PandaGPT [50]. These models introduce additional trainable parameters between the frozen image encoder (or multimodal encoder in the case of PandaGPT) and the LLMs. For instance, BLIP-2 [23] utilizes a lightweight Transformer (Q-Former) as a bridge between the image encoder and the LLMs. LLaMA-Adapter and LLaMA-Adapter V2 introduce adaptation prompts that are pre-added to higher converter layer inputs, along with new teaching cues injected into LLaMA via a zero-gated zero-initial attention mechanism. MultiModal-GPT leverages low-rank adapters (LoRA) [51] for efficient fine-tuning of OpenFlamingo [52]. MiniGPT-4 and LLaVA simply add Linear layers between the frozen visual encoder and frozen LLMs.

A critical objective in developing LLMs is to ensure that they produce coherent and extended natural language responses, enabling more human-like conversations. Multimodal instruction tuning [53] is a widely adopted technique in this regard. LLaVA [25] is the first attempt to use pure language GPT-4 [22] to generate multimodal language-image instructions following data. They are based on the captions and bounding boxes on the COCO dataset [54]. InstructBLIP [46] and MultiModal GPT [49] also proposed to match the corresponding image descriptions to create instruction fine-tuning data. Using these data, LLMs are trained, resulting in a significant improvement in dialogue performance. However, LLaMA-Adapter v2 [47] and MiniGPT4 [26] only use ready-made image captioning datasets for multimodal instruction tuning, potentially resulting in relatively weak image understanding capabilities. Therefore, it is crucial to curate diverse datasets that cover various tasks and scenarios, providing rich contextual information to support the model in deep reasoning. If the examples in the dataset are too single or simple, the model may learn to produce short and uninformative responses. In summary, high-quality instruction following data is pivotal for enabling effective conversations [53,55].

MLLMs have demonstrated a formidable capacity in bridging various modes of human communication and understanding. These models are adept at processing multiple data types, paving the way for innovative applications across diverse fields. This is particularly evident in tasks involving the VQoSS, where the application of MLLMs is especially critical. The visual quality of street spaces not only impacts the quality of life for residents but is also intricately linked to urban planning and sustainable development. Traditionally, assessing the visual quality of street spaces relies on manual annotations and subjective evaluations, a process that is both time-consuming and inefficient. Against this backdrop, the introduction of MLLMs, such as GPT-4 [22], to automate the assessment of street space visual quality is revolutionary. Firstly, by analyzing street view images from different time periods, MLLMs can identify and track urban features that change over time, offering insights into the evolving visual quality of street spaces. Secondly, MLLMs are capable of processing and integrating data from various sources, such as GIS data, social media inputs, and urban planning records, to provide a more comprehensive assessment of street space visual quality. Additionally, the integration of MLLMs helps to overcome the limitations of subjective evaluations, yielding more objective and consistent assessment results.

Currently, GPT-4 stands as one of the most powerful MLLMs, albeit accessible only through its official API and associated with high costs in large-scale applications. Therefore, we have introduced the SQ-GPT model, which achieves near GPT-4 performance through efficient knowledge distillation. The development of SQ-GPT not only signifies wider accessibility and reduced costs but also represents a more effective use of advanced MLLMs technology in the VQoSS domain. It supports urban planning and community development with data-driven insights, ushering in a new chapter for sustainable urban development.

3. Data and Method

3.1. Data

3.1.1. Data Collection

Our dataset is acquired from Baidu Street View and encompasses a total of 10,598 data points from the center of Nanjing, Gulou district, with each point paired with corresponding geographical coordinates. Figure 1 depicts our study area, the Gulou District of Nanjing. The Gulou District of Nanjing is a vital administrative region situated in the center of Nanjing City, Jiangsu Province. It covers an area of approximately 56.57 square kilometers and has a permanent resident population of about 940,400. Geographically, the district is bordered by the Yangtze River to the north, Qinhuai River to the south, Qinhuai and Pukou districts to the east, Xuanwu and Jianye districts to the south, and is adjacent to Yuhuatai District to the west and Qixia District to the north. As the core area of the city, Gulou District boasts a wealth of historical and cultural heritage, such as the Drum Tower and the Confucius Temple, and serves as Nanjing’s center for commerce, culture, and tourism, with multiple commercial districts and cultural institutions. It offers convenient transportation with several subway lines and major roads running through it. In terms of education, the district houses numerous universities and research institutions, making it one of the most concentrated areas of scientific and technological resources in China. The Gulou District also enjoys Nanjing’s 11-kilometer prime Yangtze River coastline, features diverse terrain, and has a distinct four-season climate, marking it as a region with a long history and profound cultural heritage.

This dataset captures images from two different periods: the previous period, denoted as p, is from September 2017, and the latter period, denoted as l, is from September 2020. In both periods, we collected four images per data point, corresponding to the four cardinal directions (0°, 90°, 180°, and 270°). Therefore, we refer to our dataset as $\mathcal{X}$, and an individual image within this dataset is represented by $x_i^{sd}$, where d stands for the direction, i is the data point index, and s indicates the time period (either p or l). We show a sample of our collection in Figure 2.

To progress with our method and evaluation pipeline, we randomly selected approximately 10% (1,060) of the data points from our dataset to distill knowledge via GPT-4. We denote these split data as $\overline{\mathcal{X}}$. For the purpose of user evaluation, we further randomly chose 200 data points from the remaining dataset to quantify and compare the performance between GPT-4 and our developed model, SQ-GPT.

3.1.2. Estimation Settings

In Figure 3, it can be observed that our pipeline comprises 4 distinct categories, encompassing a total of 9 individual tasks ($T=9$) designed for estimating VQoSS changes. These tasks serve as pivotal criteria for quantifying these changes, providing a comprehensive assessment. For each task, we set three levels of evaluation results: A: Positive Changes, B: Negative Changes, and C: No Changes. Our definitions for these tasks draw inspiration from the work of [4] (some samples are showing in Figure 4), and we elaborate on their specific definitions as follows (we make a concise definition so that models can use them as prompt efficiently during questioning):

Task A: Facade Color. In this task, the focus is on assessing the color schemes used on building facades along a street. It involves evaluating the aesthetic and visual appeal of the colors employed and considering any changes or improvements to enhance the overall quality of the street space.

Task B: Facade Material or Decoration. Task B revolves around evaluating and potentially modifying the materials or decorative elements used in building facades within the street environment. The aim is to enhance the visual and tactile qualities of facades, potentially by introducing new materials or decorations that contribute to the overall street space quality.

Task C: Road Damage. This task refers to any defects or deterioration in the street surface, such as potholes, cracks, rutting, weathering, surface defects, and debris, which impair both the functionality and the appearance of the roadway.

Task D: Lane Refinement. Task G involves assessing and refining the layout and design of lanes within the street. This may include modifications to lane width, markings, or traffic flow to enhance safety, traffic management, and overall street quality.

Task E: Signboards. In this task, the emphasis is on evaluating and optimizing the presence and design of signboards within the street space. This may involve assessing the placement, size, and aesthetics of signboards to improve the overall quality and functionality of the street.

Task F: Surrounding greenery and Facilities Construction. In Task I, the scope extends beyond the street itself to encompass the construction and improvement of greenery and facilities in the surrounding area. This can include parks, plazas, or other amenities that contribute to the quality and attractiveness of the street space and its immediate surroundings.

Task G: Parking Improvement. Task E focuses on evaluating and implementing improvements to parking facilities within the street area. This includes considerations for parking space design, accessibility, and any enhancements that can contribute to the overall quality of the street space.

Task H: Street Furniture Increases. In Task F, the objective is to check the quantity and quality of street furniture. This may involve adding benches, lighting fixtures, waste bins, or other elements that improve the comfort and usability of the street space.

Task I: Greenery Improvement. Task H focuses on evaluating and enhancing the presence of greenery, such as trees, shrubs, or plantings, within the street environment. The goal is to improve aesthetics, air quality, and the overall ambiance of the street.

Each of these tasks is associated with a distinctive question that necessitates a MLLM to provide an answer. Our objective is for the MLLM to effectively discern the differences between images captured during two periods p and l. The question format for each task adheres to the following structure:

$$Q_T=\left\{D\right\}+"Analyze"+\left\{T\right\}+"changesbetweendifferentperiods,"+\left\{C\right\},$$

(1)

where D is the description of input images as "The first four images are street view images taken from four different directions of the street. The last four images are taken from the same location and the same four directions but at a different period." T is the task defined above and C is a multiple choice question as "and finally you must give me a choice from: A: Positive B: Negative, C: No Changes. Make your statement concise". We also demonstrate a sample of question and answer in Table 1.

3.2. Overview of Our Pipeline

The architecture of our pipeline is illustrated in Figure 5. Initially, we distill knowledge from GPT-4 by inputting street-view images from a data point along with corresponding prompts and questions to elicit high-quality answers. This procedure is applied across all 1,060 split data points, generating a new dataset to fine-tune our own model. SQ-GPT comprises a frozen image encoder, a frozen QFormer [23] for image-to-text alignment, a Linear plus Reduction stage for additional refinement, and a frozen LLMs framework for the final answer output. The fine-tuning process is informed by insights drawn from GPT-4, ensuring that the output of SQ-GPT is improved by the distilled knowledge. We provide a detailed description of each component in the subsequent sections.

3.3. Knowledge Distillation From GPT-4

We utilize the GPT-4 API for distillation processes to generate a new dataset for the fine-tuning of our model. Table 1 presents a sample that illustrates the generation pipeline. It can be observed that GPT-4 can describe the details of input images and compare two periods from multiple aspects.

We sequentially feed GPT-4 four images from a data point’s period p, aligned at 0°, 90°, 180°, and 270°. This is followed by four images from period l in the same input order. Subsequently, for a given task, the pertinent question formulated in Equation 1 is inputted (Note that we also set the definition of the corresponding task as the input prompt). To augment the dataset with a diverse set of question-answer pairs, we adjust the temperature to 0.1 and perform $k=5$ iterations of answer generation, ensuring consistency in the choices for a particular data point in a certain task. We employ GPT-4 configured with an 8k token context length. The pricing structure is as follows: $ 0.03 per 1K tokens for input and $ 0.06 per 1K tokens for output. Our calculations indicate that generating a single question-answer pair costs approximately $ 0.07. This cost becomes prohibitive for large-scale operations, underlining the importance and economic value of our research.

For a better understanding, we formulated the generation of new data set $\mathcal{L}$ as:

$$y_{i,t,k}={\int }_k^{5}GPT4(\left[{\oint }_s^2{\oint }_d^4{x}_i^{sd}\right]+Q_t),$$

(2)

$$\mathcal{L}=\{({\widehat{x}}_i,Q_t,y_{i,t,k})\mid i=1,\cdots ,I;t=1,\cdots ,T;k=1,\cdots ,5\},$$

(3)

where ∮ stands the input images in order for period and direction of a data point, ∫ is the repeat generation for k times, I is the data point number in split $\overline{\mathcal{X}}$, ${\widehat{x}}_i$ represent all images for a data point, and t is the index for all tasks T.

$\mathcal{L}$ contains 47,700 data triads, each consisting of image sequences, corresponding questions, and high quality answers. This rich dataset will be instrumental in the fine-tuning process of our model, SQ-GPT. The fine-tuning will be executed with a specific focus on improving the model’s ability to understand and generate contextually relevant responses based on sequential visual inputs, to achieve knowledge distillation of GPT-4.

3.4. SQ-GPT

As depicted in Figure 5’s lower branch, our SQ-GPT is trained in an end-to-end manner. To commence this process, we employ a frozen image encoder $E_{image}$, which extracts image features from all images ${\widehat{x}}_i$, corresponding to a given data point i. This process can be succinctly described as follows:

$${\widehat{f}}_i=E_{image}\left({\widehat{x}}_i\right),\in {\mathbb{R}}^{sd\times hw\times c},$$

(4)

where h and w represent the spatial dimensions of the image features obtained after encoding, while c stands for the feature dimension. During this stage, we independently process the previous four images and the later four images. We stack them in the same period order and direction order as outlined in the GPT-4 distillation 3.3.

Next, we leverage a frozen QFormer architecture (shown in Figure 6), as introduced [23], to facilitate image-text alignment. The choice of QFormer is crucial in our approach, as it excels in capturing intricate dependencies between textual and visual information, enabling us to establish a more robust and meaningful connection between the two modalities. This alignment step is fundamental to the overall effectiveness of our method, as it forms the basis for subsequent fusion and comprehension of image and text data. Except the learned queries $Q^{\ast }$ in the original QFormer, we add a set of learnable queries $Q^{\prime }$ for better ability of alignment of image and text. Thus, the calculation of QFormer is as follows:

$$f_i^{\ast }=Linear\left(QFormer({\widehat{f}}_i,Q^{\ast }+Q^{\prime })\right),\in {\mathbb{R}}^{sd\times hw\times c},$$

(5)

where each image feature of $\widehat{f}$ serves as K and V in cross-attention [56]. $Linear$ is a trainable simple one-layer Fully Connection (FC) layer that can further adjust the output features from QFormer.

After the process of image-text alignment, we further reduce the volume of each image and concatenate them as a visual token entirety as follows:

$$f_i^{\prime }=Cat\left[Reduction\left(f_i^{\ast }\right)\right],\in {\mathbb{R}}^{z\times c},$$

(6)

where the reduction operation takes the form of a convolution layer with stride of 2, which effectively reduces the dimensions of the $hw$ features to one-fourth of their original size. The concatenation of all images is represented as $Cat$, and the resulting total number of visual tokens is denoted as z. The inclusion of the reduction step serves a critical purpose: it shortens the length of visual tokens, mitigating the potential adverse effects of overly long visual tokens on the computations of a frozen LLM. The importance of this reduction step has been substantiated in a prior study [24].

Next we concatenate the visual tokens (as prefix) with text tokens encoded from question $Q_t$ corresponding to a certain task. Then, a frozen LLM will take then as input and generate the answer. It can be formulated as follows:

$$y_{i,t}=LLM\left(Cat[f_i^{\prime },Q_t]\right).$$

(7)

The output answer of $y_{i,t}$ will calculate loss with one of the high quality answer $y_{i,t,k}$. Through this knowledge distillation training, our SQ-GPT can achieve performance levels close to those of GPT-4.

4. Results

4.1. Experimental Settings

Our training pipeline is adapted from LLaVA-1.5 13B [25], with modifications that involve freezing both its image encoder and its LLM components. Additionally, we incorporate a pre-trained QFormer, sourced from BLIP-V2 [23], to enhance the capabilities of our model in handling image-text tasks. We also adopt instruction tuning as LLaMA-Adapter v2 [47] and MiniGPT4 [26] during training. Only ready-made image captioning in $\mathcal{L}$ is utilized, and no extra tuning data [25] is needed for annotation. We train all 9 tasks together to enhance the generalization of the model. The training is completed within 1 day on a GPU server with 4 Nvidia A40.

To evaluate the performance of our SQ-GPT model and benchmark it against other models, we refer to Section 4.2 where a human-centric evaluation was conducted on 200 data points as mentioned in 3.1.1. We utilized LLaVA-1.5, SQ-GPT, and GPT-4 to generate answers for all nine tasks, employing the corresponding questions. During this process, we set the generation temperature to 0 to ensure precise and deterministic outputs. We enlisted three distinct groups of evaluators—50 laypeople, 30 architecture students, and 10 urban environment experts—to assess the quality of responses from all three models. Evaluators were instructed to cast a binary vote on whether the generated answers accurately addressed the task as defined in 3.1.2. Consequently, we define our metric of agreement based on these binary votes as follows:

$$Agreemen{t}_{g,m,t}=\frac{1}{|J_g|}\frac{1}{\left|O\right|}\sum _{j=1}^{J_g}\sum _{o=1}^{O}vote\left(A_{o,m,t}\right),$$

(8)

where g is the group, m is the model, $J_g$ is the number of person in a group, and O is the number of generate answer for a certain model and task.

In Section 4.3, we also provide the visual analysis by mapping the answer results of all data point to the city map. We generate answers using SQ-GPT for all 9 tasks. The answer of A: Positive, B: Negative, and C: No Changes are translated to the value of 1, 2, and 3.

4.2. Answer Quality Analysis

As shown in Figure 7, we analyze the result for human-centric evaluation in all 9 tasks. The data compares the agreement with three models: LLaVA-1.5, SQ-GPT, and GPT-4. Across all tasks, GPT-4 consistently gains the highest agreement rates from all respondent groups, indicative of its superior performance or alignment with human judgment. Our SQ-GPT also has a performance very close to GPT-4, which verifies the effect of our knowledge distillation. In addition, laypeople demonstrate the highest levels of agreement with the models, suggesting AI outputs are more in tune with their evaluations. Conversely, experts consistently show the lowest agreement rates, which may reflect a disparity in understanding or knowledge relative to the models and other groups. LLaVA-1.5 is consistently the model with the lowest agreement rates among all respondents, hinting at it being the least accurate or aligned with human judgment. This pattern suggests that advancements in AI, as seen with GPT-4, although not perfect, are increasingly recognized by common sense.

Among all tasks, A and B seem to be difficult. Experts give a low agreement even for the most powerful GPT-4. We find the most possible reason is the brightness difference when capturing images from two periods. This difference may affect the analysis of models as they can capture the color changes without discriminating the condition of weather. Models perform well in tasks like C ("Road Damage"), G ("Parking Improvement"), and I ("Greenery Improvement"). These tasks pay more attention to some obvious visual forms and layout features, which are what deep models are good at. In general, MLLMs can complete most VQoSS tasks well and is a good means to implement urban analysis. In particular, our SQ-GPT has obtained matching performance after distilling the knowledge of GPT-4, and can be used as a tool for large-scale evaluation.

We also offer some samples of answer comparison between LLaVA-1.5, SQ-GPT, and GPT-4. However, due to the length restriction, only two samples for discriminating changes of "Road Refinement" and "Greenery Improvement" are provided in this manuscript. Referencing Table 2, we tasked the models with interpreting "Road Refinement" based on images from varying periods. The later-period images visibly display faded road markings and numerous repair patches. All models unanimously indicated deterioration with option B: Negative. GPT-4 observed "The roads show signs of heavier use," which corroborates our visual assessment. Interestingly, SQ-GPT’s analysis parallels GPT-4’s but goes a step further by considering the condition of traffic signs, stating, "Despite a consistent presence of traffic signs, the overall impression hints at a deterioration in the road’s organization." This suggests SQ-GPT’s nuanced attention to detail. LLaVA-1.5, while also noting a negative change, linked it to an uptick in transportation in the later period—an associative leap that, although not incorrect, strays from the task definition provided in 3.1.2. Such findings not only demonstrate each model’s unique approach to the analysis of visual changes but also underscore the variability in interpreting contextual information, offering valuable insights into the current capabilities and limitations of MLLMs-based image interpretation.

Referencing Table 3, we tasked the models with interpreting "Greenery Improvement" based on images from varying periods. The first set of images is taken on a sunny day, while the later set are take from the day with clouds. At first glance, there seems to be no obvious greenery improvement changes exist between images from two periods. LLaVA-1.5 analyse two periods in a very fairly good way and give an answer as C: No Changes. However, GPT-4 and our SQ-GPT give the answer as Positive changes. They all concern about "the later images, the trees appear more mature and fuller, indicating growth and increased leaf density, which contributes to a lusher streetscape". When we look closely at these two periods, we can find that the trees in the latter period are indeed more vigorous and the green of the bushes is more obvious. This shows that GPT-4’s attention is more detailed, and our model distills the knowledge of GPT-4 and obtains similar capabilities.

4.3. Visualization of VQoSS Estimation

Our analysis of tasks B, C, and E reveals a persistent blue tone, suggesting that these tasks of urban development have remained relatively stable over the past three years. While stability might imply effective management in these areas, it could also signal a lack of innovation or improvement that warrants further investigation. Task G’s "Parking Improvement" stands in stark contrast with a striking red spectrum dominating the visual data. This is a clear indicator of escalating urban congestion, primarily due to a surge in vehicle ownership. The density of parked cars has outpaced the availability of parking spaces, highlighting a critical area for urban policy intervention. The city must consider implementing multifaceted solutions, such as expanding public transportation, increasing parking infrastructure, and encouraging carpooling to alleviate parking pressures.

The evaluation of Task A presents an intricate scenario, with a balanced mix of colors denoting a complex and uncertain outcome. This color distribution suggests that the strategies employed are not yielding clear results. The task’s low agreement, referenced in Figure 7, corroborates the ineffectiveness of current efforts. This warrants a reevaluation of the task’s goals and methods to better serve the city’s interests. Other tasks reflected by green shades indicate positive strides in the city’s Visual Quality of Service Standards. These improvements are promising, demonstrating successful initiatives that enhance the city’s visual appeal and functionality. However, it is crucial to maintain this momentum with continuous monitoring and updating of urban planning strategies to ensure sustained progress.

While the overall trend points towards an improvement in the city’s visual quality, the looming shadow of increased congestion due to population growth cannot be ignored. This presents a paradox where the city is improving in visual standards while simultaneously grappling with the challenges of density and overuse of urban infrastructure. To address this, city planners and policymakers need to adopt a holistic approach. The above result analysis highlights the actual effect of our SQ-GPT on VQoSS tasks, indicating that our method is significant for future urban planning.

5. Discussion

We verified that using MLLMs, like GPT-4, which are originally designed to understand and generate human-like text, can also be adoptted to interpret visual information. When applied to the estimation of VQoSS, these models can analyze photographic or multi-aspects street-view images to assess various quality indicators such as cleanliness, state of repair, signage clarity, and overall visual appeal. The estimation of VQoSS involves evaluating these images to rate or categorize the street space’s condition and visual characteristics. This could be used for urban planning, maintenance scheduling, or to improve navigational aids by providing more detailed information about the environment. In traditional machine learning, human annotation is necessary to annotate a dataset, which the model then uses to learn and make predictions. For instance, humans might need to annotate images of streets with descriptors like "poorly maintained" or "clean" to train a model to recognize these qualities. However, with MLLMs, the AI can learn from vast amounts of unlabeled data, recognizing patterns and making inferences without explicit human-supervision. This capability is particularly advantageous because it reduces the time and resources needed to prepare data, making it easier to scale up and apply these models to large-scale tasks, such as those needed for assessing the VQoSS across different cities or regions. In addition, we can further use MLLMs with both street imagery and possibly accompanying textual data (like metadata, annotations, or descriptions) to form an assessment of the street’s visual quality, which could inform various urban development and management decisions.

A distillation pipeline is also provided aimed at enhancing cost-efficiency. Given that GPT-4 is not currently free, direct API utilization for large-scale VQoSS estimation is not feasible. Our experiments with the proposed SQ-GPT demonstrated that, after distillation, our model can achieve performance comparable to GPT-4 at a reduced cost. This allows for its application in large-scale estimation tasks across various tasks for VQoSS. As the field of MLLMs evolves rapidly, GPT-4, despite its effectiveness in numerous tasks, may be superseded in the future. Consequently, our pipeline is significant for users wishing to elevate their models to state-of-the-art MLLM technology.

The current version of SQ-GPT presents some limitations. Specifically, the integration of several frozen modules from prior research may hinder performance optimization during fine-tuning. While methodologies such as Lora [51] offer pathways for efficient training, our attempts in this direction yielded suboptimal results. Adjustments to the image encoder or LLMs adversely affected the quality of the generated answers. This could be attributed to data constraints, as the distilled dataset available for fine-tuning may be insufficient for the extensive modules in question. Addressing these issues will be a primary focus in our forthcoming research endeavors.

Furthermore, our analysis indicates that SQ-GPT’s performance in tasks such as "Facade Color" recognition is not good enough, which can be primarily attributed to the challenges posed by varying light intensities and angles during image capturing. These environmental factors can significantly alter the perceived color, leading to inaccuracies in the model’s answering. To counteract these issues, future version of the pipeline could incorporate advanced pre-processing algorithms that normalize lighting conditions or employ more sophisticated techniques for color constancy. Additionally, enhancing the dataset with a wider array of images captured under diverse lighting conditions may improve the model’s robustness to such environmental variables. By addressing these nuances, we can refine SQ-GPT’s capabilities for more accurate and reliable analysis.

An aspect for further investigation is the generalizability of SQ-GPT, as well as our entire pipeline, for estimation tasks to other city areas. Specifically, it would be valuable to assess the adaptability of our methods to diverse urban environments. The unique characteristics and data profiles of different cities may present varying challenges to the robustness and accuracy of our model. Future research should focus on the scalability of our pipeline, ensuring that it maintains high performance across a range of geographically and culturally distinct cities. This expansion could potentially lead to a more versatile and universally applicable tool for urban data analysis and planning.

6. Conclusion

In this paper, we proposed SQ-GPT, aimed at assessing the visual quality of street spaces using the capabilities of MLLMs like GPT-4. We also proposed our own model SQ-GPT which comprises several key components: an image encoder, a QFormer for aligning images with text, and a refinement stage to fine-tune the process. Through efficient knowledge distillation, SQ-GPT manages to closely mirror GPT-4’s performance while significantly cutting down on costs, thereby democratizing the technology for broader use. We utilized a dataset from Baidu Street View in Nanjing, Gulou district, capturing images from two distinct time frames to monitor changes over time.

We also assessed the model’s performance through both human-centric evaluation and visual analysis, which confirmed its effectiveness in various tasks. Nevertheless, we recognized the model’s limitations, such as the challenge of inconsistent lighting conditions affecting color recognition, and proposed future improvements, including the development of sophisticated pre-processing algorithms. In conclusion, we showed that SQ-GPT can perform on par with GPT-4 in quality assessments but at a more affordable rate, which is highly beneficial for large-scale urban planning and quality assessment projects. We also acknowledged the necessity for ongoing research to surmount the existing limitations and to boost the model’s adaptability and sturdiness across various urban settings. We believe that this paper could be a pioneer work that inspires researchers using MLLMs for the future efficient and high-quality urban planning.