MetaHire AI-Driven 3D Avatar Platform for Automated Interview Simulation and Multimodal Communication Evaluation

1. Introduction

Communication skills are central to hiring decisions, yet their evaluation in traditional interviews is subjective, inconsistent, and difficult to scale. Human interviewers introduce implicit bias, and their assessments vary with fatigue, prior experience, and subjective impression. These limitations are magnified in high-volume hiring, where hundreds of candidates must be processed quickly and fairly. AI-driven analysis of verbal and non-verbal behaviour offers a path to objective and reproducible candidate profiles, and a growing body of research confirms its feasibility in interview settings.

Automated interview assessment was studied by Naim et al. [1], who demonstrated that prosody, facial cues, and language jointly predict interviewer ratings. Subsequent systems combined audio and visual signals for automated scoring [2,3]. AI-driven mock interview platforms have also been explored as candidate training tools [4]. A survey of multimodal emotion recognition confirms that fusion of visual, acoustic, and textual channels consistently outperforms any single modality [5,6].

Facial expression recognition has been advanced substantially by the DeepFace architecture of Taigman et al. [7], which achieves near-human accuracy on the LFW benchmark through a nine-layer deep network trained on over four million labelled images. Gaze estimation was addressed by Abdelrahman et al. [8], whose L2CS-Net model applies a combined classification-regression loss to predict gaze angles accurately under unconstrained webcam conditions.

Speech emotion recognition has benefited from the availability of the RAVDESS corpus [9], which provides acted recordings across eight balanced emotion classes for model training and evaluation. For adaptive dialogue generation, the DeepSeek-V3 model [10] provides a Mixture-of-Experts architecture with 671 billion parameters that supports instruction-following and context-aware question generation in real time.No existing platform integrates all four analysis modalities with a live 3D avatar and adaptive generative AI questioning under a unified feedback framework. This paper presents MetaHire, a platform that addresses this gap. The principal contributions are as follows.

• A fully integrated AI interview platform combining a 3D generative avatar with four real-time analysis modules under a unified scoring architecture. This extends a preliminary three-modality system previously reported by the authors [11].
• A proposed CNN-LSTM hybrid model for speech emotion recognition trained on RAVDESS, achieving 87.67% accuracy across eight emotion classes.
• Two original datasets: GazeEval-5, a webcam-based gaze zone corpus for interview conditions, and SentEval-Pro, an expert-labelled professional sentiment corpus of 899 sentences.
• A weighted multimodal fusion scoring engine that produces five interpretable candidate metrics from raw deep learning outputs.
• Experimental validation through benchmark evaluations on four datasets and eight complete live interview simulations.

The remainder of the paper is organised as follows. Section 2 describes the proposed MetaHire system and its 3D avatar interview pipeline. Section 3 presents the four AI analysis modules and the scoring engine. Section 4 describes the datasets. Section 5 reports experimental results and analysis. Section 6 concludes the paper.

2. The Proposed MetaHire System

MetaHire rests on four interdependent components: a 3D avatar interview engine, a generative AI dialogue layer, a multimodal analysis pipeline, and a cloud deployment infrastructure. Figure 1 shows the overall system workflow from candidate input through AI processing to feedback delivery.

2.1. 3D Avatar and Interview Delivery

At the centre of MetaHire is a photorealistic 3D avatar built with HeyGen, a neural video synthesis platform that renders talking-head video from text and audio streams. HeyGen maps a text-to-speech audio signal onto a base 3D facial model using a neural renderer that synchronises phoneme timing with jaw and lip kinematics. The result is a conversational avatar that delivers any question script with natural prosodic variation, consistent appearance, and synchronised facial animation. Unlike a pre-recorded video, the avatar renders each question dynamically from the generated script, ensuring that no two sessions appear identical. The avatar is presented to the candidate through a web interface. It takes the role of an HR interviewer, maintaining professional demeanour and appropriate response cadence throughout the session. Its consistent behaviour eliminates the within-interviewer variability that affects human evaluators and reduces candidate anxiety associated with human observation.

2.2. Generative AI Dialogue and Adaptive Questioning

Question generation is handled by DeepSeek-V3 [10], a Mixture-of-Experts large language model with 671 billion total parameters, of which 37 billion are activated per forward pass. DeepSeek-V3 builds on the transformer architecture introduced by Vaswani et al. [12], extending it with a Mixture-of-Experts design that activates only a fraction of parameters per inference step. Large language models for instruction-following tasks have evolved from early bidirectional transformer pre-training to the current generation of scaled Mixture-of-Experts architectures [13]. The model receives a structured prompt specifying the interview domain, the candidate's previous response transcript, and a behavioural instruction to generate a contextually relevant follow-up question. This mechanism ensures that technical depth is probed progressively and that the question sequence adapts to each candidate's answers rather than following a fixed script. Figure 2 illustrates the DeepSeek API integration workflow. The candidate's spoken response is first transcribed by Azure Cognitive Services, which delivers real-time phoneme-level alignment and punctuation restoration. The clean transcript is passed simultaneously to the DeepSeek-V3 question generator and to the sentiment analysis module. The generated question text is then routed to the HeyGen TTS engine, which accepts SSML-tagged input and applies domain-appropriate prosody and speaking rate before rendering the avatar's next turn.

2.3. User Interaction Workflow

Users access MetaHire either as registered accounts, which enable persistent session history and progress tracking, or as anonymous guests. After domain selection and device verification, the avatar begins the session and all four AI modules start in parallel. On session completion, the scoring engine aggregates sub-scores and the feedback dashboard is rendered with five performance metrics. Figure 3 shows the full user interaction flow from domain selection to feedback delivery.

2.4. Cloud Infrastructure

The platform is deployed on Huawei Cloud using a GPU-accelerated Elastic Cloud Server with 24 vCPUs, 96 GiB RAM, and an NVIDIA A30 GPU running Ubuntu 22.04. A Relational Database Service instance stores user credentials, session records, and feedback reports. Deployment in the AF-Cairo region provides low-latency access for regional users. The backend uses Python 3.11 with Flask, the frontend uses HTML5 and Next.js, and database access uses PyMySQL..

3. AI Analysis Modules and Scoring Engine

Camera frames feed the facial expression and eye tracking modules simultaneously. Audio windows feed the tone recognition module. Transcribed text feeds the sentiment module. Each module produces a normalised sub-score that enters the scoring engine. Figure 4 shows the data flow from multimodal inputs through the four AI modules to the scoring engine.

3.1. Facial Expression Recognition

Facial expressions are captured frame-by-frame from the webcam and processed by the pretrained DeepFace model [7]. The model applies a nine-layer deep neural network with over 120 million parameters to classify each frame into one of seven universal emotion categories: Happy, Neutral, Surprised, Sad, Fearful, Angry, and Disgust. Each frame produces a dominant emotion label and a full probability distribution, from which session-level statistics are computed.

Let f_i denote the frequency of emotion i over the session and s_i its mean intensity on a 0 to 100 scale. Each emotion is assigned a contextual weight w_i, ranging from +0.8 for Happy to −0.8 for Angry and Disgust, based on its perceived appropriateness in professional interview settings. Figure 8 shows the weighting scheme. The facial sub-score is:

S_emotion = 50 + 50 × Σ_i [ w_i (0.7 · f_i / Σf + 0.3 · s_i / 100) ]

(1)

The baseline of 50 places a candidate displaying only neutral affect at the midpoint of the scale. The 0.7 weight on frequency rewards consistent positive affect, and the 0.3 weight on intensity moderates the penalty for brief negative episodes.

The DeepFace model processes each frame through sequential stages of face detection, alignment, and feature extraction before reaching the emotion classification layer. Face detection localises the candidate's face within the webcam frame and crops it to a standardised input region. Alignment normalises the crop against a canonical facial geometry to reduce the effect of head pose variation. The aligned crop then passes through the convolutional feature extractor, which produces a compact embedding that the classification layer maps to the seven-emotion probability distribution. Figure 5 illustrates this analysis flow.

3.2. Eye Tracking and Gaze Engagement

Gaze direction is estimated using L2CS-Net [8], which predicts yaw and pitch angles through a combined classification-regression loss applied to webcam frames after face detection and facial landmark extraction. The predicted gaze vector is projected onto the screen plane and classified into one of three zones: Center, Screen Corners, or Outside Screen. These zones capture the attentional states most relevant to a webcam-based interview. Figure 6 shows the eye tracking module flow.

Let p_outside and p_corner denote the proportions of frames in the Outside and Corner zones. The gaze engagement score is:

S_gaze = 100 · (1 − (0.8 · p_outside + 0.3 · p_corner))

(2)

A candidate maintaining central gaze throughout scores 100. The asymmetric penalties reflect that outside-screen gaze is a stronger indicator of disengagement than brief peripheral glances toward the screen corners.

3.3. Tone Recognition

Audio is captured in overlapping 2.4-second windows. Each window is transformed into a 20-coefficient Mel-Frequency Cepstral Coefficients (MFCC) representation [14], which encodes the spectral envelope of speech in a form robust to speaker identity and ambient noise variation. The MFCC matrix feeds a proposed hybrid CNN-LSTM architecture. The architecture combines time-distributed Conv1D layers, which extract local spectro-temporal patterns, with an LSTM layer that models temporal dependencies across the sequence. The model classifies each audio segment into one of eight emotions: Calm, Happy, Neutral, Surprised, Sad, Fearful, Angry, and Disgust. Using the softmax probability p_j and contextual weight w_j for class j, the tone sub-score is:

S_tone = 100 × Σ_j (w_j · p_j)

(3)

Weights range from +0.9 for Calm to −0.9 for Angry, emphasising composure and clarity as the primary vocal virtues in professional communication. Figure 7 illustrates the tone recognition analysis flow from audio capture through MFCC extraction and CNN-LSTM classification to tone score output.

3.4. Speech Sentiment Analysis

Sentiment analysis adds a semantic layer beyond acoustic content, capturing how a candidate frames responses linguistically. Each transcribed speech segment is passed to DeepSeek-V3 with a structured prompt instructing the model to classify the text as Confident, Neutral, or Worried in a professional interview context. The model applies its instruction-following capability to interpret interview-specific phrasing, such as hedged language or assertive declaratives. The output probability over the three classes is mapped to a sentiment score. Confident responses receive the highest weight, worried responses the lowest. The sentiment score feeds into the Confidence Score and Clarity Score on the feedback dashboard. Figure 8 illustrates the sentiment analysis flow from speech transcription through LLM classification to sentiment score output.

Figure 8. Sentiment analysis module flow from transcribed text to sentiment score.

Figure 8. Sentiment analysis module flow from transcribed text to sentiment score.

3.5. Unified Scoring Engine

After each session the scoring engine aggregates the four sub-scores into a final communication performance score:

S_final = 0.35 · S_emotion + 0.35 · S_gaze + 0.30 · S_tone

(4)

Emotion and gaze each carry 35% because non-verbal signals hold higher diagnostic weight in interview assessment research [1]. Tone carries 30%, consistent with its supporting role. Sentiment contributes to derived dashboard metrics rather than S_final directly. The dashboard presents five metrics: Communication Score from Equation (4), Confidence Score weighted toward tone and sentiment, Clarity Score from LLM evaluation of response structure, Technical Score from keyword matching against domain question banks, and Boost Score as a composite for overall ranking. All scores are normalised to 0 to 100 and assigned a performance badge: High (76 to 100%), Moderate (46 to 75%), or Low (0 to 45%).

4. Datasets

Four datasets support the evaluation of the four analysis modules. Two are established public benchmarks and two are original contributions of this work. Table 1 provides an overview.

The LFW dataset contains more than 13,000 face images collected from the web under unconstrained conditions, augmented here with seven emotion category labels for facial expression evaluation [15]. RAVDESS provides 1,440 audio recordings from 24 professional actors covering eight balanced emotion classes, segmented into 2.4-second clips for the tone recognition model [9]. GazeEval-5 is an original dataset constructed specifically for this work. It comprises 1,800 webcam-captured images from five participants, recorded under controlled conditions to represent three gaze zones relevant to remote interview settings: Center, Screen Corners, and Outside Screen. All frames were manually annotated by the authors following a predefined labelling protocol. SentEval-Pro is an original text corpus developed to address the absence of professional-context sentiment data suitable for interview assessment. It contains 899 sentences constructed to reflect candidate interview responses across a range of confidence levels, and annotated into three categories, Confident, Neutral, and Worried, by domain experts following inter-annotator agreement verification.

5. Experimental Results and Analysis

5.1. Benchmark Evaluation

Each module was evaluated independently on its corresponding dataset using accuracy, precision, recall, and F1 score. Table 2 reports the results.

The facial emotion module achieves 97.35% accuracy, consistent with the benchmark result reported by Taigman et al. [7] for DeepFace on LFW. Precision and recall are balanced at 96% and 95%, confirming reliable detection without class-specific bias. Figure 9 shows the confusion matrix, where near-diagonal concentration confirms correct classification across all seven emotion classes.

The eye tracking module reaches 92.73% accuracy on GazeEval-5. The Screen Corners class shows slightly lower precision-recall balance than the Center and Outside classes, which is expected given the geometric ambiguity between corner and center gaze near zone boundaries. Figure 10 shows the confusion matrix for facial emotion recognition on the LFW dataset.

The proposed CNN-LSTM tone recognition model achieves 87.67% accuracy on RAVDESS, with precision, recall, and F1 all equal at 88.00%, indicating balanced performance across the eight emotion classes. This symmetry confirms that the architecture generalizes without over-fitting to the majority classes. Figure 11 shows the confusion matrix for eye tracking classification on the GazeEval-5 dataset.

The sentiment module reaches 94.10% accuracy on SentEval-Pro, with an F1 score of 91.20%. The slight gap between accuracy and F1 reflects a mild imbalance in the Worried class, which is the least frequent category in typical interview scenarios. Figure 12 shows the confusion matrix for sentiment classification on the SentEval-Pro dataset.

5.2. Live Interview Evaluation

Eight participants completed full simulated interviews on the deployed MetaHire platform. Each session ran for approximately 12 to 15 minutes and covered both technical and behavioural questions delivered by the avatar. Table 3 reports the raw AI module outputs and Table 4 the interpreted dashboard metrics.

Four of the eight sessions received High badges. Trial 1 produced the highest communication score of 93.1, driven by sustained central gaze (92), a predominantly calm vocal tone (92.5), and consistently positive facial affect (96). Trial 5 received the lowest communication score of 57.3, attributable to gaze and tone scores of 55 and 56, which reflect frequent off-screen attention and an anxious vocal delivery. Notably, Trial 5 achieved a clarity score of 96.4, indicating well-structured verbal responses despite poor non-verbal engagement. This dissociation between verbal articulation and non-verbal behaviour demonstrates the diagnostic value of the four-modality architecture over any single-channel approach.

Trials 3 and 4 received Moderate badges primarily due to low technical scores (49.33 and 47.67) and low behavioural scores (23.5 and 25.5), even though their non-verbal communication metrics were acceptable. These candidates present as verbally composed but technically underprepared, a profile that is indistinguishable from stronger performers when using non-verbal signals alone. The LLM-based technical and sentiment scoring components therefore add discriminative capability not available in acoustic and visual modalities.

5.3. Limitations

Language support is currently limited to English; non-native speakers with strong accents may receive reduced accuracy from the speech recognizer and tone model. GazeEval-5 and SentEval-Pro were collected from a small, demographically homogeneous group, which may limit generalization across cultures and age groups. The modality weights in Equations (1) to (4) were set heuristically based on interview research rather than through data-driven optimization. The HeyGen avatar introduces an average rendering latency of approximately 2.3 seconds, and Azure speech transcription adds 1.2 to 1.6 seconds, which may disrupt conversational flow in timed interview formats. The live evaluation involved eight participants, sufficient for a proof-of-concept but too small for statistical inference over a broader population.

6. Conclusions and Future Work

This paper presented MetaHire, an AI-driven platform integrating a 3D generative avatar, adaptive LLM-based questioning, and a four-modality real-time analysis engine for virtual interview simulation and candidate communication assessment. Benchmark evaluations on LFW, RAVDESS, GazeEval-5, and SentEval-Pro confirmed strong module-level performance, with accuracy between 87.67% and 97.35%. Eight live interview simulations validated the integrated system, yielding communication scores between 57.3 and 93.1 and revealing interpretable candidate profiles that single-modality analysis could not recover. Future work will pursue multilingual support through accent-robust speech models and translated question banks, data-driven calibration of modality weights using a large and diverse participant corpus, enterprise integration enabling HR professionals to access comparative candidate dashboards, and VR deployment where the avatar operates in an immersive environment with spatial audio and motion tracking.