SOQ: Theoretical Foundations for Mapping Multimodal Meaning in New Works of Creative Industries

1. Introduction

Contemporary art is characterized by increasing complexity of semantic structures, multimodality of ex- pressive means, and interactive nature of audience interaction. In conditions of digitalization of cultural practices and development of artificial intelligence technologies, there arises a need for developing formal- ized methods of analysis and evaluation of artistic works [1,2]. The global generative AI market in art amounts to $298.3 million in 2023, with projected exponential growth to $8.628 billion by 2033, indicating cardinal changes in creative industries [3]. Traditional approaches to art historical analysis, based predominantly on qualitative interpretation methods, face challenges from contemporary artistic forms. Works of digital, hybrid, and interactive art require new analytical tools capable of accounting for their multilayered semantic structure and dynamic character of meaning formation [4,5].

Sethi demonstrated possibilities of quantitative analysis of artistic style using computer vision and semantic workflows [6]. Elgammal and Saleh developed a network approach to creativity assessment, defining it through parameters of originality and influence in temporal networks of works [7]. Rossetto introduced the concept of visual semantic density (VSD) as a measure of information necessary for complete description of a visual object, which became the foundation for developing formalized approaches to semantic content assessment [8].

Computer vision and machine learning algorithms demonstrate growing ability to interpret artistic works, including style recognition, color palette analysis, and pattern identification that may remain unnoticed by the human eye [9,10,11,12]. Contemporary developments in artificial intelligence open new possibilities for analyzing large collections of artworks using automated classification and authentication methods [13].

Recent research demonstrates that multimodal AI systems exhibit intrinsic capabilities for cross-modal meaning generation, challenging traditional text-centric approaches to AI creativity [47]. Advanced transformer architectures now enable real-time fusion of visual, textual, and auditory modalities in creative applications [48].

Contemporary research shows that generative AI increases human creative productivity by 25% [3]. The development of multimodal AI systems opens new possibilities for analyzing works combining text, images, audio, and interactive elements [14].

This study proposes an integrative SOQ concept – a comprehensive multimodal meaning assessment system that extends existing approaches by accounting for interpretation dynamics, personalized perception, and holistic work integrity.

The problems of quantitative art assessment identified through analysis of the contemporary state require addressing fundamental theoretical foundations capable of ensuring conceptual integrity of the proposed approach. Consideration of semiotic, multimodal, and holistic aspects of art analysis will allow formulating the theoretical foundation of the SOQ system.

2. Theoretical Foundations

2.1. Multimodal Semiotics and Theory of Meaning

The SOQ concept is based on fundamental propositions of Yu.M. Lotman’s semiotics about culture as a semiotic system and mechanisms of meaning formation in various sign structures [15,16]. Of particular significance is Lotman’s understanding of cultural text as a multilayered semantic construction capable of generating new meanings in the interpretation process. Lotman’s concept of the semiosphere as a cultural semiotic continuum provides methodological foundation for understanding how semantic structures function in dynamic cultural environment [1,17].

G. Kress and T. van Leeuwen’s theory of multimodality provides methodological foundation for analyzing interaction of various semiotic resources in contemporary art [18,19]. Their concept of transpositional grammar allows systematizing ways of meaning transmission through different modalities: visual, auditory, textual, spatial, and tactile. Kress and van Leeuwen’s social-semiotic approach emphasizes that social norms, values, and conventions shape the meaning of visual constructions, which is especially important for understanding contemporary art in its sociocultural context [20,21].

Cognitive semiotics offers additional tools for analyzing artistic works, integrating methods of linguistics, cognitive sciences, and semiotics for understanding processes of art perception and interpretation [22,23]. This approach allows accounting for both perceptual and cognitive aspects of artistic perception, which is critically important for developing a comprehensive meaning assessment system.

Contemporary developments in generative AI have revealed the inherently multimodal nature of meaning-making systems, where visual rhetoric and multimodal discourse intersect in novel ways [Pereira & Dondero, 2025]. This intersection provides new opportunities for applying semiotic analysis to AI-generated content across multiple modalities [Bateman, 2025].

2.2. Quantitative Methods in Art Analysis

Development of digital humanities led to emergence of new methodological approaches to art analysis, combining traditional humanistic methods with computational technologies [24]. Sethi’s research demonstrates effectiveness of applying Discrete Tonal Measure, Discrete Variational Measure, and Convolutional Style Measure for quantitative analysis of artistic works [6]. Their approach to assessing brushstroke density, color palette, and textural characteristics creates precedent for formalizing aesthetic categories.

Elgammal and Saleh’s work on quantitative creativity assessment in artistic networks introduces important concepts of originality and influence as measurable parameters [7]. Their creativity formula demonstrates possibility of mathematical modeling of creative processes.

Network analysis in art studies opens new possibilities for revealing connections between works, artists, and cultural contexts [25]. Application of graph theory methods allows visualizing and analyzing complex interrelationships in artistic systems, which is especially important for understanding contemporary art with its intertextual and transmedia characteristics.

Contemporary approaches to automated art analysis include use of neural networks for artistic style analysis [26,27]. Neural network models can learn from large data arrays and identify patterns inaccessible to traditional analysis.

Recent systematic reviews reveal significant advances in deep learning applications for cultural imagery analysis, though challenges remain regarding algorithmic bias and cultural representation [46]. The integration of digital humanities with computer science methodologies shows exponential growth, indicating increasing acceptance of computational approaches in humanities research [43].

2.3. Visual Semantic Density

Rossetto’s work [8] on the concept of visual semantic density (VSD) defines it as "the minimum amount of information needed to fully and uniquely describe an image such that the image content can be inferred from the description." This definition resonates with the goals of the SOQ concept but extends to multimodal works and includes dynamic aspects of interpretation.

Development of methods for assessing semantic density of visual media opens new possibilities for quantitative analysis of informational content of artistic works [28]. These approaches use neural network architectures such as ConvNext for automatic extraction of semantic characteristics of images.

2.4. Holistic Approaches to Art Analysis

Holistic art analysis considers a work as an integral system where meaning arises from interaction of all components [29,30]. This approach opposes reductionist methods that analyze individual elements of a work in isolation. Holistic methodology recognizes that artistic works possess emergent properties that cannot be understood through simple summation of their parts [31].

2.5. Temporal Aspects of Perception and Dynamic Semiotics

The SOQ methodology integrates temporal dimension into artistic work analysis, which is especially critical for assessing interactive and performative art forms where meaning is not fixed but created in dynamic interaction between work, author, and audience.

2.5.1. Modeling Dynamics of Interactive Art

The system uses generalizable multimodal world models capable of predicting future frames of video sequences and analyzing dynamic interaction between work, performer, and audience. This allows tracking meaning evolution over time and capturing ephemeral aspects of performative art.

The SOQ methodology applies the metaphor of "dynamic ornament" to describe the temporal and evolutionary nature of interactive works. Unlike static structures, temporal structure of semantic connections represents a living and changing entity characterized by fluidity, transformation, and emergent effects, providing a more systematic and process-oriented approach to contemporary art analysis.

2.5.2. Temporal Consistency in Multimodal Analysis

SOQ ensures temporal consistency when analyzing works combining various media (video, sound, text) unfolding over time. The system uses autoregressive noise charts for stable generation with extended temporal horizon, generating paired RGB and depth data for enhanced spatial understanding.

2.6. Cinema Language and Advertising Metrics as Multimodal Meaning Benchmarks

Cinema language proposed the first systematized grammar of multimodal storytelling, defining the importance of using tools such as frame, rhythm, transition, and sound motif. Kuleshov’s montage "fifth truth" is an early example of "connected semantic unit" where meaning is born not in the frame itself but on the edge between frames, in the contextual field.

Beyond the film industry, visual content meaning was assessed in advertising, where information ab- sorption indicators were clearly linked to marketing metrics and directly influenced video success. Starting with a more complex system, marketers simplified "absorbed meaning" assessment to three figures: Ad Recall (how many viewers remember the video after 24 hours); Brand Lift (purchase intention change); and AET (Attention-Engagement-Time, seconds of fixed gaze).

Both cinema language and advertising video assessment are examples of how authorial meaning can be systematized and calculated, while serving as theoretical foundation for our work.

3. Conceptual SOQ Model

3.1. Philosophical Justification for Quantitative Meaning Assessment

In our opinion, the meaning of an artistic work is not fixed matter subject to measurement, but a dynamic process arising in the act of interpretation by the viewer and their virtual dialogue with the author. Consequently, any attempt at its objectification faces fundamental semiotic paradox: the more precise the formula, the wider the distance between sign and meaning, between given and givenness (in Lotman’s terms).

SOQ proposes a third path: combining tools for meaning prototyping and rapid hypothesis testing with "living" intersubjective metrics based on verified authorial intent, correlating with expert consensus and audience behavior, rather than chasing abstract truth about meaning.

The SOQ system represents a concept of integral measurement of multimodal meaning in artistic works, accounting for semantic density, internal semantic architecture, interpretation depth, perception accessibility, and communicative effectiveness.

3.2. Definition of Meaning in SOQ

The theoretical SOQ model is based on representing an artistic work as a dynamic macro network of semantic nodes of different size and visibility with internal and external meaning parameters, placed on the temporal axis of interaction with the work.

Each node in the macro network is assigned a specialized AI agent for analysis and tracing interpretation dynamics [32,33]. Contemporary achievements in AI for art analysis demonstrate effectiveness of automated methods for artistic content processing.

In the SOQ system, mapped dynamic meaning has the following characteristics:

1.

Semantic scale includes 4 variants: - Elementary semantic unit: minimal textual "brick" within one node - Block semantic unit: generalized meaning of entire node plus its position in macronetwork - Connected semantic unit: independent meaning arising on edge between two blocks - Aggregated semantic unit: synthetic map of all incoming and outgoing block connections

2.

Vitality: applied only to medium semantic units, includes 2 node states: active and passive

3.

Type of meaning source: indicates how meaning was created in node and system

4.

Repeatability level: indicates novelty - internal, external, general, and absolute

5.

Visibility: related to digestibility and visibility of meaning element for audience

3.3. SOQ Formula

The total SOQ parameter is determined by the formula:

SOQ_total = (SOQ_NL_nodes · Holistic_Factor) + α·Author_Score + β·Audience_Score + γ·Expert_Score + δ·Personal_Score + ε·World_Score

where:

-

SOQ_NL_nodes represents a dynamic system with memory analyzing artwork as living process of node interactions over time

-

Holistic_Factor or integrity coefficient accounts for cross-modal motif coherence - α, β, γ, δ, ϵ are adaptive weight coefficients

The Holistic_Factor is computed as:

HF = (Cross_Modal_Coherence × Temporal_Consistency × Semantic_Integration) / 3

where:

-

Cross_Modal_Coherence measures alignment between visual, auditory, and textual elements (0-1)

-

Temporal_Consistency evaluates meaning stability across time sequences (0-1)

-

Semantic_Integration assesses the degree of meaningful connection between modalities (0-1)

The adaptive weight coefficients are determined through a multi-stage calibration process:

-

α (Author weight): Calculated based on documented authorial intent strength (0.3-0.5)

-

β (Audience weight): Derived from audience engagement metrics and sample size (0.2-0.4)

-

γ (Expert weight): Weighted by expert credentials and consensus level (0.2-0.3)

-

δ (Personal weight): Adjusted for user interaction history and preference consistency (0.1-0.2)

-

ε (World Score): Originality and quality of the artistic world (0.2-0.3)

The coefficients are normalized to sum to 1.0 and updated dynamically based on validation performance against ground truth assessments.

Each component is calculated using specialized neural networks trained on annotated multimodal art datasets.

SOQ_NL_nodes is calculated as:

SOQ_NL_nodes = Σₜ₌₁ᵀ SOQₜ (nonlinear temporal aggregation)

The system creates emergent properties through attention mechanisms and nonlinear processing of semantic nodes.

3.3.1. SOQ_NL_nodes Architecture

The SOQ_NL_nodes system employs a transformer-based architecture with temporal memory:

SOQ_NL_nodes = Σ (t=1 to T) [Attention (Qt, Kt, Vt) × Memory_State(t-1) × Temporal_Weight(t)]

where:

-

Qt, Kt, Vt represents query, key, and value matrices at time t

-

Memory_State maintains context from previous interpretations

-

Temporal_Weight decreases exponentially for older interpretations (λ = 0.9)

The system processes semantic nodes through four specialized attention heads:

1.

Static_Attention: Processes fixed authorial elements

2.

Dynamic_Attention: Tracks audience interpretation changes

3.

Personal_Attention: Adapts to individual user preferences

4.

Cumulative_Attention: Integrates historical interpretation data

3.4. ArtScore Metric for Artistic Quality Assessment

Additionally, to the main SOQ formula, a specialized ArtScore metric is proposed for quantitative assessment of "artistry" in works created using AI technologies:

ArtScore = α · AC + β · SD + γ · CR + δ · IN

where: - AC (Aesthetic Complexity) measures formal work properties - SD (Semantic Depth) evaluates symbolic content saturation and connectivity - CR (Cultural Resonance) reflects work’s connection to sociocultural context - IN (Innovation) measures degree of deviation from existing norms

Contemporary research supports multi-dimensional approaches to creativity assessment, integrating empirical aesthetics with psychological measures of creative evaluation [36]. Automated evaluation systems demonstrate moderate agreement with expert assessments when properly calibrated [40], though significant limitations remain in style replication capabilities [34].

3.5. Validation and Evaluation Framework

The SOQ system requires comprehensive validation across multiple dimensions to ensure reliability and validity of assessments. We propose a three-tier validation framework:

3.5.1. Technical Validation

-

Cross-modal consistency testing using established multimodal datasets

-

Temporal stability assessment through repeated measurements

-

Inter-rater reliability testing with expert panels (target κ > 0.7)

3.5.2. Cultural Validation

-

Cross-cultural testing across diverse artistic traditions

-

Bias detection and mitigation protocols

-

Inclusive representation in training data

3.5.3. Longitudinal Validation

-

Tracking assessment stability over time

-

Adaptation to evolving artistic practices

-

Continuous learning and model updating protocols

The validation process employs both quantitative metrics (correlation coefficients, statistical significance tests) and qualitative assessments (expert interviews, focus groups) to ensure comprehensive evaluation of the SOQ system's performance.

4. Ethical Considerations and Social Responsibility

Development and implementation of the SOQ system raise critically important ethical questions requiring systematic approach to ensuring social responsibility of technological solutions in culture and art sphere.

4.1. Preventing Commercialization of Artistic Meaning

The process-oriented SOQ approach serves as key mechanism for counteracting reduction of art to commodity characteristics. Focusing on dynamic nature of creative process, the system shifts attention from final product to experience of creating and interpreting meaning.

4.2. Ensuring Inclusivity and Cultural Justice

Fighting cultural prejudices is implemented through use of diverse and representative training data promoting recognition and interpretation of marginalized culture art. The system actively counteracts "silencing" of underrepresented group voices through mechanisms for identifying and correcting cultural stereotypes in AI-generated content.

4.3. Privacy Protection and Ethical Data Handling

The personalized SOQ layer implements comprehensive user data protection measures including encryption, access control, and personal information anonymization.

5. Applications and Development Prospects

5.1. Application Areas

-

Generative art: SOQ model can be used not only for analysis but also for creating artworks

-

Art historical expertise: integration with digital collection management systems allows automating cataloging and analysis processes

-

Curatorial activity: SOQ system can help curators make decisions about including works in exhibitions

-

Education: applying SOQ in educational programs can promote more systematic approach to art study

-

Creative industries: SOQ offers overcoming gap between semantic accuracy and aesthetic quality

Human-AI collaborative systems show particular promise for expanding creative frontiers, though effects vary significantly across different user populations [47]. Interactive co-creation interfaces demonstrate measurable improvements in design quality and user satisfaction [45], while real-time collaborative drawing systems enable novel forms of synchronous creative expression [44].

5.2. Limitations and Challenges

-

Interpretation subjectivity: despite objectification aspiration, SOQ remains dependent on cultural and individual interpretation factors

-

Technological limitations: current AI and computer vision development level may not fully capture subtle semantic nuances

5.3. Future Research Directions

-

Expansion to temporal media: adapting SOQ for video, performance, and interactive installation analysis

-

Cross-cultural validation: testing concept applicability in various cultural contexts

-

Blockchain technology integration: creating decentralized artwork assessment and verification system

-

User interface development: creating accessible tools for artists, curators, and researchers

-

Biometric integration: developing methods for integrating viewer physiological and emotional reactions

6. Conclusion

Integrating achievements in semiotics, computer vision, network analysis, and multimodality theory, the SOQ concept proposes formalized methodology and tools for analyzing complex artistic works and quantitative authorial assessment of their meaning with addition of reception channels and interpretation layers from AI agents, critics, and users.

SOQ should be considered as an authorial system for mapping, vectorization, visualization, and marking meaning elements, where the authorial layer is the reference point, and receptive layers are channels for observing and correcting communication with audience, fans, and critics, rather than normative sources of "truth" instead of the author.

Main concept advantages include:

-

Holistic approach to meaning analysis

-

Accounting for interpretation dynamics

-

Possibility of quantitative work comparison

-

Adaptivity to analysis technology development

The SOQ concept represents a significant advancement in computational approaches to art analysis, offering a theoretically grounded, empirically validatable framework for assessing multimodal meaning in contemporary artistic works. While acknowledging the inherent limitations of quantifying aesthetic and semantic dimensions, SOQ provides valuable tools for researchers, curators, and artists navigating the increasingly complex landscape of AI-augmented creative practices. Future research should focus on empirical validation across diverse cultural contexts and continued refinement of the system's technical components to ensure its relevance and accuracy in rapidly evolving creative industries.