

% Template for PLoS
% Version 3.5 March 2018
%
% % % % % % % % % % % % % % % % % % % % % %
%
% -- IMPORTANT NOTE
%
% This template contains comments intended 
% to minimize problems and delays during our production 
% process. Please follow the template instructions
% whenever possible.
%
% % % % % % % % % % % % % % % % % % % % % % % 
%
% Once your paper is accepted for publication, 
% PLEASE REMOVE ALL TRACKED CHANGES in this file 
% and leave only the final text of your manuscript. 
% PLOS recommends the use of latexdiff to track changes during review, as this will help to maintain a clean tex file.
% Visit https://www.ctan.org/pkg/latexdiff?lang=en for info or contact us at latex@plos.org.
%
%
% There are no restrictions on package use within the LaTeX files except that 
% no packages listed in the template may be deleted.
%
% Please do not include colors or graphics in the text.
%
% The manuscript LaTeX source should be contained within a single file (do not use \input, \externaldocument, or similar commands).
%
% % % % % % % % % % % % % % % % % % % % % % %
%
% -- FIGURES AND TABLES
%
% Please include tables/figure captions directly after the paragraph where they are first cited in the text.
%
% DO NOT INCLUDE GRAPHICS IN YOUR MANUSCRIPT
% - Figures should be uploaded separately from your manuscript file. 
% - Figures generated using LaTeX should be extracted and removed from the PDF before submission. 
% - Figures containing multiple panels/subfigures must be combined into one image file before submission.
% For figure citations, please use "Fig" instead of "Figure".
% See http://journals.plos.org/plosone/s/figures for PLOS figure guidelines.
%
% Tables should be cell-based and may not contain:
% - spacing/line breaks within cells to alter layout or alignment
% - do not nest tabular environments (no tabular environments within tabular environments)
% - no graphics or colored text (cell background color/shading OK)
% See http://journals.plos.org/plosone/s/tables for table guidelines.
%
% For tables that exceed the width of the text column, use the adjustwidth environment as illustrated in the example table in text below.
%
% % % % % % % % % % % % % % % % % % % % % % % %
%
% -- EQUATIONS, MATH SYMBOLS, SUBSCRIPTS, AND SUPERSCRIPTS
%
% IMPORTANT
% Below are a few tips to help format your equations and other special characters according to our specifications. For more tips to help reduce the possibility of formatting errors during conversion, please see our LaTeX guidelines at http://journals.plos.org/plosone/s/latex
%
% For inline equations, please be sure to include all portions of an equation in the math environment.  For example, x$^2$ is incorrect; this should be formatted as $x^2$ (or $\mathrm{x}^2$ if the romanized font is desired).
%
% Do not include text that is not math in the math environment. For example, CO2 should be written as CO\textsubscript{2} instead of CO$_2$.
%
% Please add line breaks to long display equations when possible in order to fit size of the column. 
%
% For inline equations, please do not include punctuation (commas, etc) within the math environment unless this is part of the equation.
%
% When adding superscript or subscripts outside of brackets/braces, please group using {}.  For example, change "[U(D,E,\gamma)]^2" to "{[U(D,E,\gamma)]}^2". 
%
% Do not use \cal for caligraphic font.  Instead, use \mathcal{}
%
% % % % % % % % % % % % % % % % % % % % % % % % 
%
% Please contact latex@plos.org with any questions.
%
% % % % % % % % % % % % % % % % % % % % % % % %

\documentclass[10pt,letterpaper]{article}
\usepackage[top=0.85in,left=2.75in,footskip=0.75in]{geometry}

% amsmath and amssymb packages, useful for mathematical formulas and symbols
\usepackage{amsmath,amssymb}

% Use adjustwidth environment to exceed column width (see example table in text)
\usepackage{changepage}

% Use Unicode characters when possible
\usepackage[utf8x]{inputenc}

% textcomp package and marvosym package for additional characters
\usepackage{textcomp,marvosym}

% cite package, to clean up citations in the main text. Do not remove.
\usepackage{cite}

% Use nameref to cite supporting information files (see Supporting Information section for more info)
\usepackage{nameref,hyperref}

% line numbers
\usepackage[right]{lineno}

% ligatures disabled
\usepackage{microtype}
\DisableLigatures[f]{encoding = *, family = * }

% color can be used to apply background shading to table cells only
\usepackage[table]{xcolor}

% array package and thick rules for tables
\usepackage{array}



\usepackage{balance}
\usepackage[pdftex]{graphicx}
\usepackage{algorithmic}

\usepackage{tabularx}
\usepackage{fixltx2e}
\usepackage{subcaption}
\usepackage{stfloats}
\usepackage{url}
\usepackage{float}
\usepackage{comment}
\usepackage{booktabs}

% create "+" rule type for thick vertical lines
\newcolumntype{+}{!{\vrule width 2pt}}

% create \thickcline for thick horizontal lines of variable length
\newlength\savedwidth
\newcommand\thickcline[1]{%
  \noalign{\global\savedwidth\arrayrulewidth\global\arrayrulewidth 2pt}%
  \cline{#1}%
  \noalign{\vskip\arrayrulewidth}%
  \noalign{\global\arrayrulewidth\savedwidth}%
}

% \thickhline command for thick horizontal lines that span the table
\newcommand\thickhline{\noalign{\global\savedwidth\arrayrulewidth\global\arrayrulewidth 2pt}%
\hline
\noalign{\global\arrayrulewidth\savedwidth}}


% Remove comment for double spacing
%\usepackage{setspace} 
%\doublespacing

% Text layout
\raggedright
\setlength{\parindent}{0.5cm}
\textwidth 5.25in 
\textheight 8.75in

% Bold the 'Figure #' in the caption and separate it from the title/caption with a period
% Captions will be left justified
\usepackage[aboveskip=1pt,labelfont=bf,labelsep=period,justification=raggedright,singlelinecheck=off]{caption}
\renewcommand{\figurename}{Fig}

% Use the PLoS provided BiBTeX style
\bibliographystyle{plos2015}

% Remove brackets from numbering in List of References
\makeatletter
\renewcommand{\@biblabel}[1]{\quad#1.}
\makeatother



% Header and Footer with logo
\usepackage{lastpage,fancyhdr,graphicx}
\usepackage{epstopdf}
%\pagestyle{myheadings}
\pagestyle{fancy}
\fancyhf{}
%\setlength{\headheight}{27.023pt}
%\lhead{\includegraphics[width=2.0in]{PLOS-submission.eps}}
\rfoot{\thepage/\pageref{LastPage}}
\renewcommand{\headrulewidth}{0pt}
\renewcommand{\footrule}{\hrule height 2pt \vspace{2mm}}
\fancyheadoffset[L]{2.25in}
\fancyfootoffset[L]{2.25in}
\lfoot{\today}

%% Include all macros below

\newcommand{\lorem}{{\bf LOREM}}
\newcommand{\ipsum}{{\bf IPSUM}}

%% END MACROS SECTION


\begin{document}
\vspace*{0.2in}

% Title must be 250 characters or less.
\begin{flushleft}
{\Large
\textbf\newline{An AI-Based Framework for Improving Efficiency and Fairness in the Interview Process} % Please use "sentence case" for title and headings (capitalize only the first word in a title (or heading), the first word in a subtitle (or subheading), and any proper nouns).
}
\newline
% Insert author names, affiliations and corresponding author email (do not include titles, positions, or degrees).
\\
Mohannad~Taman\textsuperscript{1,3},
Yahia~Khaled\textsuperscript{2,3},
Dalia~Sobhy\textsuperscript{3*},
%Name4 Surname\textsuperscript{2},
%Name5 Surname\textsuperscript{2\ddag},
%Name6 Surname\textsuperscript{2\ddag},
%Name7 Surname\textsuperscript{1,2,3*},
%with the Lorem Ipsum Consortium\textsuperscript{\textpilcrow}
\\
\bigskip
\textbf{1} AI Engineer, Obeikan Digital Solutions, Cairo, Egypt,  m.taman@obeikan.com.sa
\textsuperscript{1,2},
\\
\textbf{2} SOC Analyst, Banque Misr, Cairo, Egypt, yahiakhaled49@gmail.com
\\
\textbf{3} Arab Academy of Science and Technology and Maritime Transport, Alexandria, Egypt.
\\
\bigskip

% Insert additional author notes using the symbols described below. Insert symbol callouts after author names as necessary.
% 
% Remove or comment out the author notes below if they aren't used.
%
% Primary Equal Contribution Note
%\Yinyang These authors contributed equally to this work.

% Additional Equal Contribution Note
% Also use this double-dagger symbol for special authorship notes, such as senior authorship.
%\ddag These authors also contributed equally to this work.

% Current address notes
%\textcurrency Current Address: Dept/Program/Center, Institution Name, City, State, Country % change symbol to "\textcurrency a" if more than one current address note
% \textcurrency b Insert second current address 
% \textcurrency c Insert third current address

% Deceased author note
%\dag Deceased

% Group/Consortium Author Note
%\textpilcrow Membership list can be found in the Acknowledgments section.

% Use the asterisk to denote corresponding authorship and provide email address in note below.
* dalia.sobhi@aast.edu

\end{flushleft}
% Please keep the abstract below 300 words
\section*{Abstract}
Artificial intelligence (AI) technologies have advanced to the point where they can help human resource specialists, such as recruiters, by automating major parts of the hiring process and filtering the list of candidates. However, little research has evaluated the use of AI in virtual interviews. This paper presents \textit{InstaJob}, an AI-powered framework designed to improve efficiency and fairness in the hiring process. It uses deep learning models for face emotion detection, text emotion analysis, and filler word detection in interviews to evaluate candidates' soft skills, ensuring unbiased assessments. The proposed face emotion detection model achieved a validation accuracy of 77\%, which outperforms the other state-of-the-art approaches.


% Please keep the Author Summary between 150 and 200 words
% Use first person. PLOS ONE authors please skip this step. 
% Author Summary not valid for PLOS ONE submissions.   
%\section*{Author summary}
%Lorem ipsum dolor sit amet, consectetur adipiscing elit. Curabitur eget porta erat. Morbi consectetur est vel gravida pretium. Suspendisse ut dui eu ante cursus gravida non sed sem. Nullam sapien tellus, commodo id velit id, eleifend volutpat quam. Phasellus mauris velit, dapibus finibus elementum vel, pulvinar non tellus. Nunc pellentesque pretium diam, quis maximus dolor faucibus id. Nunc convallis sodales ante, ut ullamcorper est egestas vitae. Nam sit amet enim ultrices, ultrices elit pulvinar, volutpat risus.

\linenumbers

% Use "Eq" instead of "Equation" for equation citations.
\section*{Introduction}
Emotion is a complex and dynamic state that arises in response to various stimuli, including experiences, thoughts, or social interactions. It includes subjective feelings, cognitive processing, behavioral responses, physiological changes, and communication cues. Therefore, the ability to identify emotions is essential in a variety of domains, including marketing, human-robot interaction, mental health evaluation, and employment interviews \cite{jekauc2024recognizing, khare2024emotion}.

Using virtual interviews in employment processes has become more common recently. Virtual video interviews offer several advantages to interviewers and interviewees alike: (1) they enable HR personnel to assess a large number of job applications; (2) they enable HR personnel to review and make decisions offline; and (3) they facilitate cost-effective long-distance interviews. However, virtual video interviews pose several challenges, particularly in areas like assessing non-verbal communication, building rapport, and ensuring consistency across candidates. Studies have indicated that in virtual environments, biases related to race, gender, and socioeconomic class may develop \cite{dando2023interviewing, coleman2024lessons}. In order to minimise bias by standardising the interview process and delivering data-driven insights for more equitable decisions, there is growing interest in the usage of AI-driven tools that objectively analyse applicant responses and non-verbal cues. These technologies can help make virtual interview recruitment fairer by ensuring that hiring decisions are based on qualifications rather than opinions.

According to recent surveys \cite{mckinseyState2023, yadav2024adopting}, AI in recruiting is becoming more widely accepted, with 60\% of HR professionals believing it will have a major impact. The global market for AI recruiting is expected to reach \$7.1 billion by 2025, highlighting its growing influence in the field. HR professionals can save up to 80\% of their time during the hiring process by implementing AI, while candidates who practice using AI-powered interview tools have a higher chance of securing employment opportunities \cite{mckinseyState2023, yadav2024adopting}. In this context, the use of automated systems for assessing video interviews to evaluate applicants has grown in popularity in recent years across a range of industries \cite{fernandez2020ai}. Hiring managers can save time with these tools, and candidates can schedule interviews at their convenience \cite{lukacik2022into}. Nowadays, automatic video interview assessment systems are in use all over the world. More than 80 million interviews have been assessed according to HireVue, a well-known startup in the automated hiring sector \cite{hirevueAboutCompany}.


Emerging applications have made use of social signal processing (SSP) \cite{vinciarelli2011bridging} to assist in the study and assessment of interview performance. Nevertheless, emotional cues from the voice and face have not been fully investigated in prior studies. 
%For example, most previous studies, detailed in the following section, only focused on detecting smiles.




%\IEEEPARstart{T}{he} job market has undergone significant changes with the rise of AI technologies aimed at improving recruitment efficiency and objectivity. Traditional methods of interviewing and hiring are often prone to bias, inefficiencies, and subjective evaluations. 

%*InstaJob* addresses these issues by leveraging AI models to analyze candidates' facial expressions, emotions in text, and speech patterns, thus offering a streamlined and unbiased recruitment process.


This paper addresses the following research question: \textbf{RQ:} \textit{How can artificial intelligence (AI) be applied to ensure the fairness of the applicant's evaluation and the efficiency of the interview process using facial and speech cues?}
    

The remainder of the paper is structured as follows: Section \ref{related} presents related work. Section \ref{desc} describes the proposed AI-driven framework. Section \ref{eval} provides the experimental evaluation. Finally, Section \ref{conc} provides conclusions and future work.


% \begin{eqnarray}
% \label{eq:schemeP}
% 	\mathrm{P_Y} = \underbrace{H(Y_n) - H(Y_n|\mathbf{V}^{Y}_{n})}_{S_Y} + \underbrace{H(Y_n|\mathbf{V}^{Y}_{n})- H(Y_n|\mathbf{V}^{X,Y}_{n})}_{T_{X\rightarrow Y}},
% \end{eqnarray}

%\section*{Materials and methods}

\section*{Related Work}\label{related}
AI-based recruitment techniques can promote better hiring results, decrease bias, and accelerate the hiring process \cite{signore2023digital, fabris2023fairness, qin2023comprehensive}. However, the effectiveness of AI in hiring depends on the approaches adopted and the implementation environment. This section presents an overview of research on AI-based recruitment practices.


\subsection*{AI in Recruitment}
The use of AI in recruitment has grown significantly over the years. Various studies have explored automated systems for CV parsing, skill assessment, and candidate filtering, with most approaches focusing on text analysis \cite{albert2019ai, kathiravan2023modern}. However, few works have focused on emotional detection from video interviews.

%the proposed platform focuses on incorporating face emotion detection and speech analysis to provide a holistic evaluation of candidates.


In recent years, several recruitment platforms have been proposed \cite{naim2016automated,mhadgut2022vrecruit,avanish2022interview}. For example, a multi-fine-grained method based on recurrent neural networks (RNN) and keyword-question attention mechanisms is proposed for interviews \cite{naim2016automated}. This method scores different personality characteristics of interviewees, obtaining a comprehensive interview score. The approach leverages a two-stage model learning mechanism and a keyword-question-level attention mechanism to reliably predict personality traits after removing words and phrases that are associated with those traits. This work has focused on determining personality traits based on speech without considering facial emotion recognition. Further, ``vRecruit" \cite{mhadgut2022vrecruit}, a machine learning-based web application for virtual recruitment, was proposed, which incorporates a client-specific interview process and a text-based sentiment analysis engine. In \cite{avanish2022interview}, the authors used CNN to classify a candidate's emotions based on images and nervousness from blink rate during a virtual interview process, with an accuracy of 60\%. However, the focus of these works was on evaluating candidates using still images, which may introduce significant bias compared to video interview analysis.






% For figure citations, please use "Fig" instead of "Figure".
%Nulla mi mi, Fig~\ref{fig1} venenatis sed ipsum varius, volutpat euismod diam. Proin rutrum vel massa non gravida. Quisque tempor sem et dignissim rutrum. Lorem ipsum dolor sit amet, consectetur adipiscing elit. Morbi at justo vitae nulla elementum commodo eu id massa. In vitae diam ac augue semper tincidunt eu ut eros. Fusce fringilla erat porttitor lectus cursus, \nameref{S1_Video} vel sagittis arcu lobortis. Aliquam in enim semper, aliquam massa id, cursus neque. Praesent faucibus semper libero.

% Place figure captions after the first paragraph in which they are cited.
% \begin{figure}[!h]
% \caption{{\bf Bold the figure title.}
% Figure caption text here, please use this space for the figure panel descriptions instead of using subfigure commands. A: Lorem ipsum dolor sit amet. B: Consectetur adipiscing elit.}
% \label{fig1}
% \end{figure}


\subsection*{Emotion Detection in AI}
Emotion detection has been a significant area of AI research, especially with the use of deep learning models such as convolutional neural networks (CNN) for facial expression analysis. The 2013 Facial Expression Recognition (FER) Challenge data set (FER-2013 dataset) \cite{goodfellow2013challenges}, a widely used benchmark in this field. It provides a diverse range of facial expressions necessary for training and evaluating emotion detection models. 

The first model, the Local Learning Bag of Words (BOW) was designed for a competition application \cite{goodfellow2013challengesrepresentationlearningreport}. This was later improved by the Local Learning Deep+BOW model, which combined deep learning features with traditional BOW and was applied to mental disorder detection \cite{Georgescu_2019}. In 2020, Ensemble ResMaskingNet, an ensemble learning approach, demonstrated the effectiveness of combining multiple CNNs to improve CNN performance \cite{9411919}. Another CNN-based model from the same year, aimed at enhancing human-computer interaction \cite{9154121}.

By 2022, efforts to improve emotion detection methods using data augmentation with the VGG16 model was introduced, which was used to improve the quality of images in the FER-2013 dataset \cite{9647334}. However, it achieved a low accuracy as compared to previous approaches. In 2023, custom architectures and other methods achieved moderate improvements, with one such custom architecture reaching a considerable accuracy, though the application details were unspecified \cite{10.1007/978-981-99-7025-4_26}. Other studies from 2023 reported proposed some models with applications remaining unspecified \cite{10.1145/3587038} \cite{10071193}.

To the best of our knowledge, none of these approaches have used face emotion detection to enhance the recruitment process. In this context, we aim to introduce a novel deep learning model specifically tailored for this purpose.

In parallel with facial emotion recognition, text-based emotion recognition has evolved significantly. Models like BERT and EmoRoBERTa have been employed to analyze sentiment and emotional tone in textual data. These models have been particularly useful in assessing emotional intelligence in scenarios such as interviews, where the analysis of both facial expressions and text plays a pivotal role.

The evolution of emotion detection models, from traditional handcrafted feature extraction techniques to advanced deep learning and ensemble methods, underscores the continual advancements in this field. The shift towards deeper architectures and hybrid approaches, along with the integration of multiple datasets, represents a significant leap forward in the accuracy and reliability of emotion detection systems. Ensemble models, in particular, have proven highly effective, combining diverse architectures to achieve superior performance. As emotion detection continues to mature, it will remain an essential component in applications ranging from human-computer interaction (HCI) to personalized content recommendation systems.


\section*{Proposed Framework}\label{desc}

%In this section, the proposed framework is presented.

\subsection*{Full System Architecture}\label{collect}
This system analyzes video interviews through two parallel paths (Fig. \ref{fig:Interview process system Architecture}). The left path focuses on facial analysis, using HaarCascade for face detection and CNN techniques to assess candidate expressions. The right path processes audio, transcribing it with IBM Watson, then analyzing it for fillers and emotions using IBM Watson and EmoRoBERTa. Each analysis produces a score, which are combined to generate a comprehensive assessment of the candidate's performance.



    \begin{figure}
        \centering        %\includegraphics[width=1\linewidth]{facearchitecture.png}
%\includegraphics[width=0.8\linewidth]{ARCH.png}
\includegraphics[scale =0.9]{ARCH.png}
        \caption{Interview process system Architecture.}
        \label{fig:Interview process system Architecture}
    \end{figure}

\subsection*{Face Emotion Detection}\label{collect}      
\subsubsection*{System Architecture}\label{collect}  
The system architecture is shown in Fig. \ref{fig:Face Emotion Detection System Architecture}. The Face Emotion Detection module aims to analyze facial expressions to detect emotions such as happiness, sadness, anger, and neutrality. It provides users with feedback on their emotional responses during video interviews or interactions, enhancing self-awareness and interview performance. This analysis is performed offline after the video is uploaded by the candidate. Finally, the recruiter receives a score indicating the candidate's performance.

    \begin{figure}
        \centering        %\includegraphics[width=1\linewidth]{facearchitecture.png}
\includegraphics[width=1\linewidth]{model_insta.jpg}
        \caption{Face Emotion Detection System Architecture.}
        \label{fig:Face Emotion Detection System Architecture}
    \end{figure}



\subsubsection*{HaarCascade For Face Detection}\label{collect}
In our video processing pipeline, we utilize haarcascade\textunderscore frontalface\textunderscore default.xml from the OpenCV library for face detection. This pre-trained HaarCascade classifier is specifically designed to identify frontal faces within images or video frames (Fig. \ref{fig:Face Detection}). The classifier operates by evaluating patterns of pixel intensities that correspond to the characteristic features of a human face, such as the arrangement of eyes, nose, and mouth. Upon detection, the classifier marks the location of each identified face with a bounding box, effectively isolating facial regions from the rest of the image or frame. This bounding box ensures that only the essential facial features are extracted as inputs for further processing, such as emotion detection using machine learning models. Integrating HaarCascade face detection enhances the precision and efficiency of our system in capturing and analyzing facial expressions within video streams.


 \begin{figure}[!h]
        \centering
        \includegraphics[scale=0.3]{1.png}
        %\includegraphics[width=0.5\linewidth]{1.png}
        \caption{Sample of Face Detection using Haar Cascade.}
        \label{fig:Face Detection}
    \end{figure}


\subsubsection*{Emotion Detection}\label{collect}    
After detecting faces using {haarcascade\_frontalface\_default.xml} in the video processing pipeline, the extracted facial regions (bounded by boxes) are passed to the emotion detection model. These regions, containing key facial features (Fig. \ref{fig:Face Emotion Detection}) are preprocessed through resizing and pixel normalization to ensure consistent input formatting.We developed a Convolutional Neural Network (CNN) model for face emotion detection. The CNN consists of multiple convolutional and pooling layers followed by fully connected layers. The input shape of images is 48x48 pixels, grayscale. The architecture of our CNN model is shown in Fig. \ref{fig:CNN_model}. The deep learning model then classifies emotions—such as happiness, sadness, anger, surprise, and neutrality—based on these features. By combining Haar Cascade face detection with the emotion recognition model, the system efficiently captures and interprets emotional cues from video streams, enabling meaningful real-time insights.

  \begin{figure}
        \centering
        %\includegraphics[width=0.3\linewidth]{SadFull.png}
        \includegraphics[scale = 0.3]{SadFull.png}
        \caption{Face Detection Detection.}
        \label{fig:Face Emotion Detection}
    \end{figure}
    
\begin{figure}
    \centering
    \includegraphics[width=0.8\linewidth]{model_CNN_2.png}
    \caption{The CNN model for the face emotion detection feature.}
    \label{fig:CNN_model}
\end{figure} 



%\begin{comment}
    % \begin{figure}[!h]
    %     \centering
    %     \includegraphics[width=0.5\linewidth]{haarscade.png}
    %     \caption{Face Detection}
    %     \label{fig:Face Detection}
    % \end{figure}

   
%\end{comment}

  

\begin{comment}
\subsection{Emotion Detection}\label{collect}    
After detecting faces using haarcascade\textunderscore frontalface\textunderscore default.xml in our video processing pipeline, the identified facial regions, delineated by bounding boxes, serve as inputs to our emotion detection model. These regions contain essential facial features necessary for analyzing emotional expressions. Each cropped face image undergoes preprocessing, including resizing and normalization of pixel values, to ensure consistency and optimal input format for the deep learning model. Subsequently, the model leverages its trained architecture to classify emotions such as happiness, sadness, anger, surprise, and neutrality based on the extracted facial features. By integrating Haar Cascade face detection with our emotion detection model, we achieve a streamlined workflow that effectively captures and interprets emotional cues from video streams, enhancing the system's capability to provide meaningful insights in real-time applications.
\end{comment}


    

\subsection*{Speech Analysis}\label{collect}
It consists of three modules. First, the \textit{Speech-to-Text Conversion} module, which is implemented using the IBM Watson Speech-to-Text API. It inputs the candidate's speech, performs some preprocessing (e.g., sentence segmentation using punctuation (.,?)), and then provides the transcribed text. Second, the \textit{Speech Emotion Detection} module uses the pre-trained EmoRoBERTa (28 emotion classes). It performs three main processes: contraction expansion (e.g., ``can't'' $\rightarrow$ ``cannot''), sentence-level emotion classification, and dominant emotion flagging. The third module is \textit{Filler Word Detection}, which uses regex-based pattern matching. It searches for some fillers (e.g. 'uh', 'um', 'like', etc) and provides the timestamped filler occurrences. At the end, it calculates a score for speech detection and filler detection. 

\subsection*{Scoring Methodology}
The score $S$ is calculated as a weighted sum of negative emotion instances ($E$) and filler word occurrences ($F$) from three independent models ($S = \sum_{i=1}^{3} w_i \cdot T_i$). $T_i$ represents the count of triggers (emotions/filler words) for model $i$, whereas $w_i$ is the weight per model (inverse of triggers needed to increment score by 1). Table \ref{tab:scoring} shows the scoring rules per model.


%\subsection{Model-Specific Triggers}
\begin{table} [!h]
\centering
\caption{\textbf{Scoring rules per model.}}
\label{tab:scoring}
%\begin{tabular}{@{}lll@{}}
\begin{tabular}{|l|l|l|}
\hline
\textbf{Model} & \textbf{Trigger} & \textbf{Score Increment} \\ \thickhline
\hline
Face Emotions Detection & 4 negative emotions & +1 \\
\hline
Speech-to-Text Emotions AI & 6 filler words & +1 \\
\hline
Text Emotions AI & 2 negative emotions & +1 \\ \hline
\end{tabular}
\end{table}



%     \begin{figure}
%         \centering
% \includegraphics[width=0.7\linewidth]{dataset-cover.png}
%         \caption{FER2013 Seven Emotions\cite{fer2013_dataset}.}
%         \label{fig:FER2013}
%     \end{figure}

\section*{Face Emotion Detection Experimental Evaluation}\label{eval}
In this section, we will discuss the experimental evaluation. 

\subsection*{Dataset}
The dataset consists of facial expressions obtained from the Facial Expression Recognition 2013 (FER2013) dataset \cite{fer2013_dataset}, which was provided by Kaggle at the International Conference on Machine Learning (ICML). The FER-2013 dataset contains a set of \textit{35,887} grayscale face images, all standardized to 48×48 pixels. Each image is annotated with one of seven emotion categories: \textit{Angry, Disgust, Fear, Happy, Sad, Surprise,} or \textit{Neutral} (Fig. \ref{fig:FER2013}). These emotions cover a wide spectrum of affective states that are essential for examining human reactions in interview situations, where nonverbal clues such as facial expressions offer crucial information about applicants' emotional reactions, degrees of engagement, and behavioral indicators.

The dataset is divided into training, validation, and test sets, facilitating model development and performance evaluation across different stages. There are \textit{28,709} images in the training set and \textit{3,589} images in the test set. In the training set, there are \textit{4,953} images of anger, \textit{547} images of disgust, \textit{5,121} images of fear, \textit{8,989} images of happiness, \textit{6,077} images of sadness, \textit{4,002} images of surprise, and 6,198 images of neutrality. This imbalance presents an additional challenge in training models to perform accurately across all classes. The dataset's low resolution and grayscale format add further complexity, as models must learn to identify subtle emotional cues from limited visual data.

\begin{figure}
    \centering
    \includegraphics[width=0.9\linewidth]{dataset-cover.png}
    %\includegraphics[scale =0.3]{dataset-cover.png}
    \caption{FER2013 Seven Emotions \cite{fer2013_dataset}.}
    \label{fig:FER2013}
\end{figure}

\subsection*{Dataset Preprocessing and Augmentation}\label{collect}
% To fit the interview context, a data cleaning step was mandatory. We encountered label inaccuracies, particularly in the ``happy" class, where images mislabeled as ``angry" were identified and removed. Additionally, emotions such as fear, surprise, and disgust were excluded from the dataset as they are less commonly expressed in typical interview contexts. In this context, the focus was on four expressions: \textit{angry, sad, happy,} and \textit{neutral} as shown in \ref{fig:Cleaned FER2013}.


% In the generated dataset after cleaning, we observed an imbalance in the distribution of instances across different emotional states, as shown in Figure \ref{fig:Cleaned FER2013 Class Distribution}. For example, the ``happy" category had the highest number of instances, followed by ``neutral" and ``sad," while ``angry" had a notably lower amount. This mismatch is problematic since the model can be biased to identify dominant emotions like happiness and neutrality, which could result in less precise identification and categorization of less common emotions like anger. This imbalance should be addressed to ensure the model can successfully capture and distinguish a wide variety of emotional expressions seen in actual interview situations.

To fit the interview context, data cleaning was essential. Label inaccuracies, particularly in the ``happy" class where images were mislabeled as ``angry," were identified and removed. Additionally, emotions such as fear, surprise, and disgust were excluded, focusing on four expressions: angry, sad, happy, and neutral (Fig. \ref{fig:Cleaned FER2013}). After cleaning, the dataset showed an imbalance in emotional state distribution, with "happy" having the most instances, followed by ``neutral" and ``sad," while ``angry" had the least (Fig. \ref{fig:Cleaned FER2013 Class Distribution}). This imbalance could bias the model towards recognizing dominant emotions like happiness and neutrality, potentially affecting its ability to accurately identify less common emotions like anger. Addressing this imbalance is crucial for the model's performance in real interview contexts.

%The "happy" category exhibited the highest number of instances, followed by "neutral" and "sad," while "angry" had significantly fewer instances in comparison. This imbalance poses a challenge as the model may lean towards better recognizing over-represented emotions like happiness and neutrality, potentially leading to less accurate detection and classification of less frequent emotions such as anger. Addressing this imbalance is crucial to ensure the model's ability to effectively capture and differentiate a diverse range of emotional expressions encountered in real-world interview scenarios.


%We have performed data cleaning to include the expressions that more expressed in typical interview contexts. We 

%After selecting the FER2013 dataset for our face emotion detection model, we 
%After selecting the FER2013 dataset for our face emotion detection model, thorough cleaning was essential to ensure dataset accuracy and relevance. We encountered label inaccuracies, particularly in the "happy" class, where images mislabeled as "angry" were identified and removed. Additionally, emotions such as fear, surprise, and disgust were excluded from the dataset as they are less commonly expressed in typical interview contexts. This meticulous cleaning process aimed to refine the dataset, enhancing its suitability for training a deep learning model to accurately classify essential facial expressions like happiness, sadness, anger, surprise, and neutrality.



%\subsection{About the dataset}\label{collect}

%In our cleaned dataset for face emotion detection, we observed a notable imbalance in the distribution of instances across different emotional states. The "happy" category exhibited the highest number of instances, followed by "neutral" and "sad," while "angry" had significantly fewer instances in comparison. This imbalance poses a challenge as the model may lean towards better recognizing over-represented emotions like happiness and neutrality, potentially leading to less accurate detection and classification of less frequent emotions such as anger. Addressing this imbalance is crucial to ensure the model's ability to effectively capture and differentiate a diverse range of emotional expressions encountered in real-world interview scenarios.

    \begin{figure}
        \centering
        \includegraphics[width=0.8\linewidth]{cleanedexample.png}
        \caption{Cleaned FER2013.}
        \label{fig:Cleaned FER2013}
    \end{figure}


    \begin{figure}
        \centering
        \includegraphics[width=0.8\linewidth]{unbalanced.png}
        \caption{The cleaned FER2013 class distribution.}
        \label{fig:Cleaned FER2013 Class Distribution}
    \end{figure}

    

%\subsection{Data Augmentation}\label{collect}

The train\textunderscore data\textunderscore generator is an instance of ImageDataGenerator configured to augment image data during the training phase of our machine learning model. This configuration includes several augmentation techniques aimed at enhancing the model’s ability to generalize from the training data. The images are first rescaled to a range of 0 to 1 to normalize the pixel values. During training, the generator randomly applies a rotation of up to 15 degrees, zooms images by up to 10\%, performs horizontal flips, and shifts images horizontally and vertically by up to 5\% of the total width and height, respectively, using a wrapping fill mode to handle gaps. These augmentations not only help expose the model to a wider variety of image variations, reducing overfitting and improving its ability to learn robust features from the data, but also effectively address the initial class imbalance by generating synthetic samples for under-represented classes like ``angry." This approach ensures a more balanced distribution of training instances across different emotional states, thereby enhancing the model’s overall performance and generalization capability.

        
    %\begin{figure}[H]
     %   \centering  \includegraphics[width=0.6\linewidth]{Dataaugmentation.png}
        %\caption{ImageGenerator Data Augmentation.}
        %\label{fig:ImageGenrator Data Augmentation}
    %\end{figure}
    

    
    \begin{figure}
        \centering
\includegraphics[width=0.8\linewidth]{augmentedexamples.png}
        \caption{Examples of the augmented dataset.}
        \label{fig:Examples Of The Augmented Dataset}
    \end{figure}

%\subsection{Evaluation Metrics}

\subsection*{Results and Discussion}

To evaluate the effectiveness and robustness of our face emotion detection model, we conducted an extensive series of simulations utilizing the FER-2013 dataset. The FER-2013 dataset is well-regarded in the field for its diverse range of facial expressions and comprehensive annotations, making it an ideal benchmark for assessing model performance. Our simulations were designed to rigorously test the model’s ability to accurately identify and classify a wide array of facial emotions across various conditions, including different lighting, angles, and occlusions.

Through these simulations, we aimed to visualize and analyze the model's predictions in comparison to the actual ground truth labels. This process not only helps in quantifying the model's accuracy but also in identifying specific instances where the model excels or struggles. By showcasing these visualizations, we provide a detailed insight into the operational dynamics of our face emotion detection system.

Fig. \ref{fig:Face Emotion model examples} illustrates sample results from our simulations, demonstrating the model's capability to interpret and categorize facial emotions. Each image is accompanied by the predicted emotion label and the corresponding ground truth, allowing for a clear comparison. These visualizations highlight the practical applicability of our model in real-world scenarios, underscoring its potential for integration into various applications such as AI-powered interviews, mental health assessments, and human-computer interaction systems.

In addition to evaluating the model’s accuracy, these simulations also shed light on its limitations and areas for improvement. For instance, certain emotions may be more challenging to detect under low-light conditions or when the face is partially obscured. Understanding these limitations is crucial for guiding future enhancements and ensuring that the model performs reliably across all intended use cases.

%##################################################################################################
\begin{figure}[htbp]
    \centering
    \begin{subfigure}[b]{0.25\textwidth} % Adjust width for two columns
        \centering
        \includegraphics[width=\textwidth]{1.png}
        \caption{Happy}
        \label{fig:Happy1}
    \end{subfigure}%
    \begin{subfigure}[b]{0.25\textwidth} % Adjust width for two columns
        \centering
        \includegraphics[width=\textwidth]{2.png}
        \caption{Neutral}
        \label{fig:Neutral1}
    \end{subfigure}%
    
    \begin{subfigure}[b]{0.25\textwidth} % Adjust width for two columns
        \centering
        \includegraphics[width=\textwidth]{3.png}
        \caption{Sad}
        \label{fig:Sad1}
    \end{subfigure}%
    \begin{subfigure}[b]{0.25\textwidth} % Adjust width for two columns
        \centering
        \includegraphics[width=\textwidth]{4.png}
        \caption{Happy}
        \label{fig:Happy2}
    \end{subfigure}

    \begin{subfigure}[b]{0.25\textwidth} % Adjust width for two columns
        \centering
        \includegraphics[width=\textwidth]{5.png}
        \caption{Happy}
        \label{fig:Happy3}
    \end{subfigure}%
    \begin{subfigure}[b]{0.25\textwidth} % Adjust width for two columns
        \centering
        \includegraphics[width=\textwidth]{6.png}
        \caption{Happy}
        \label{fig:Happy4}
    \end{subfigure}

    \begin{subfigure}[b]{0.25\textwidth} % Adjust width for two columns
        \centering
        \includegraphics[width=\textwidth]{7.png}
        \caption{Neutral}
        \label{fig:Neutral2}
    \end{subfigure}%
    \begin{subfigure}[b]{0.25\textwidth} % Adjust width for two columns
        \centering
        \includegraphics[width=\textwidth]{8.png}
        \caption{Neutral}
        \label{fig:Neutral3}
    \end{subfigure}

    \begin{subfigure}[b]{0.25\textwidth} % Adjust width for two columns
        \centering
        \includegraphics[width=\textwidth]{9.png}
        \caption{Angry}
        \label{fig:Angry1}
    \end{subfigure}%
    \begin{subfigure}[b]{0.25\textwidth} % Adjust width for two columns
        \centering
        \includegraphics[width=\textwidth]{10.png}
        \caption{Sad}
        \label{fig:Sad2}
    \end{subfigure}

    \begin{subfigure}[b]{0.25\textwidth} % Adjust width for two columns
        \centering
        \includegraphics[width=\textwidth]{11.png}
        \caption{Sad}
        \label{fig:Sad3}
    \end{subfigure}%
    \begin{subfigure}[b]{0.25\textwidth} % Adjust width for two columns
        \centering
        \includegraphics[width=\textwidth]{12.png}
        \caption{Angry}
        \label{fig:Angry2}
    \end{subfigure}

    \caption{Face Emotion model examples.}
    \label{fig:Face Emotion model examples}
\end{figure}

%##################################################################################################

Our model achieved a validation accuracy of 77\%, a validation loss of 0.6, a validation precision of 0.82, and a validation recall of 0.71. Table \ref{tab:fer-summary} shows that our model outperforms the other state-of-art approaches. 

\begin{table}[!htbp]
\caption{\textbf{Summary of the state-of-the-art approaches and their performance as compared to the proposed model on FER dataset.}}
\label{tab:fer-summary}
\centering
\renewcommand{\arraystretch}{1.2}
\begin{tabular}{|c|c|p{3cm}|c|p{3cm}|}
\hline
\textbf{Study} & \textbf{Year} & \textbf{Method} & \textbf{Accuracy} & \textbf{Application} \\ \thickhline
\hline
\cite{goodfellow2013challengesrepresentationlearningreport} & 2014 & Local Learning BOW & 67.48\% & Competition \\
\hline
\cite{Georgescu_2019} & 2018 & Local Learning Deep+BOW & 75.42\% & Mental Disorder Detection \\
\hline
\cite{9411919} & 2020 & Ensemble ResMaskingNet & 76\% & Boost the Performance of CNNs \\
\hline
\cite{9154121} & 2020 & CNN & 70.14\% & Enhance Human Computer Interaction \\
\hline
\cite{9647334} & 2022 & VGG16 & 58.6\% & Improve Quality of Images in FER2013 \\
\hline
\cite{10071193} & 2023 & N/A & 61.88\% & N/A \\
\hline
%\cite{10.1145/3570837} & 2023 & N/A & 63.25\% & N/A \\
%\hline
\cite{10.1007/978-981-99-7025-4_26} & 2023 & Custom Architecture & 66.6\% & N/A \\
\hline
\cite{BHAGAT20242079} & 2024 & Deep CNN & 65.68\% & VR, Robotics, Marketing and Mental Health \\
\hline
\textbf{Ours} & \textbf{2025} & \textbf{Advanced CNN} & \textbf{77\%} & \textbf{AI Recruitment} \\
\hline
\end{tabular}
\end{table}

While validation accuracy provides a general measure of the model's performance, precision and recall offer more nuanced insights into its behavior. Precision, the ratio of correctly predicted positive observations to the total predicted positives, is crucial in scenarios where the cost of false positives is high. Recall, the ratio of correctly predicted positive observations to all actual positives, is essential in situations where it is important to capture as many true positives as possible. These metrics help us understand the trade-offs between different types of errors, guiding us to optimize the model for its intended applications.

Hence, to provide a balanced measure of the model's performance, especially when dealing with imbalanced datasets or when the costs of false positives and false negatives are different, we compute the F1 score. The F1 score is the harmonic mean of precision and recall, calculated as 

\begin{center}
\( F1 = \frac{2 \cdot \text{precision} \cdot \text{recall}}{\text{precision} + \text{recall}} = 0.761 \)
\end{center}

The F1 score balances precision and recall, offering a single metric that accounts for both types of errors. It is especially important in applications like emotion detection, where both false positives and false negatives can have significant implications. For instance, incorrectly identifying a negative emotion when there is none (false positive) can cause unnecessary concern, while missing a genuine negative emotion (false negative) can overlook someone in need of help. Thus, the F1 score provides a more reliable metric for assessing the overall effectiveness of our face emotion detection model.


\begin{comment}

\subsection{Experiment Setup}
The AI models were tested on real interview videos. The facial emotion detection model was trained using FER-2013, achieving 75\% accuracy, while text emotion detection using EmoRoBERTa on the GoEmotions dataset achieved an accuracy of 89\%. Filler detection accuracy, powered by IBM Watson, reached 85%.

\subsection{Results and Discussion}
The experiment results reveal that the AI models effectively reduced human bias and improved the objectivity of candidate evaluations. By analyzing both emotional responses and speech patterns, *InstaJob* ensures comprehensive soft skills assessment.

\subsubsection{Metrics Comparison}
- **Facial Emotion Detection Accuracy**: 75\%
- **Text Emotion Detection Accuracy**: 89\%
- **Filler Word Detection Accuracy**: 85\%

These results demonstrate the efficiency of the models in providing objective evaluations, significantly reducing the time spent on manual assessments.

\section{Scoring Methodology}
The score $S$ is calculated as a weighted sum of negative emotion instances ($E$) and filler word occurrences ($F$) from three independent models ($S = \sum_{i=1}^{3} w_i \cdot T_i$). $T_i$ represents the count of triggers (emotions/filler words) for model $i$, whereas $w_i$ is the weight per model (inverse of triggers needed to increment score by 1).


\subsection{Model-Specific Triggers}
\begin{table}[h]
\centering
\caption{Scoring Rules per Model}
\label{tab:scoring}
\begin{tabular}{@{}lll@{}}
\toprule
\textbf{Model} & \textbf{Trigger} & \textbf{Score Increment} \\ \midrule
Face Emotions Detection & 4 negative emotions & +1 \\
Speech-to-Text Emotions AI & 6 filler words & +1 \\
Text Emotions AI & 2 negative emotions & +1 \\ \bottomrule
\end{tabular}
\end{table}
\end{comment}

\subsection*{Implementation Example on Scoring Methodology}
For a session with 8 negative emotions (Face), 12 filler words (Speech), and 3 negative emotions (Text), the score is computed as follows (with respect to Table \ref{tab:scoring}):
%\begin{itemize}
 %   \item 8 negative emotions (Face), 
  %  \item 12 filler words (Speech), 
  %  \item 3 negative emotions (Text),
%\end{itemize}
%The score is computed as:
\[
S = \left\lfloor \frac{8}{4} \right\rfloor + \left\lfloor \frac{12}{6} \right\rfloor + \left\lfloor \frac{3}{2} \right\rfloor = 2 + 2 + 1 = 5
\]
Generally, higher scores reflect cumulative performance issues. The main limitation is that it assumes uniform impact of triggers; calibration may vary by context, where other advanced scoring methodologies will be addressed in the future.

\begin{comment}


\begin{itemize}
    \item \textbf{Interpretation}: Higher scores reflect cumulative performance issues.
    \item \textbf{Limitations}: Assumes uniform impact of triggers; calibration may vary by context.
\end{itemize}


    
\section{Threats to Validity}\label{sec:ThreatsToValidity}

\subsection{Internal Threats}
One internal threat to the validity of the InstaJob platform is the variability in data quality. The system processes video and audio inputs from candidates, and inconsistencies in video resolution, lighting, and audio clarity could impact the performance of the AI models. For example, poor lighting could affect the facial emotion detection model, while background noise might interfere with speech-to-text transcription, leading to inaccurate emotion analysis or filler detection.

Another internal threat is the risk of overfitting in the AI models. While the models have been trained on large datasets like FER-2013 and GoEmotions, their ability to generalize well to new, unseen data might be compromised if the training datasets do not fully represent the diversity of real-world candidates. This could result in biased or inaccurate assessments when applied to interviewees from different cultural or linguistic backgrounds.

\subsection{External Threats}
One external threat to validity is the generalization of the AI models to different cultural contexts. The facial emotion detection and speech analysis models are trained on datasets that may not capture the full range of emotional expressions or speech patterns seen in diverse populations. For instance, cultural differences in facial expressions or speech intonations might not be well-represented in the training data, leading to potential bias in the model’s predictions. This raises concerns about the fairness of the system when evaluating candidates from different regions or with different accents.

Another external threat is the realism of the virtual interview environment. While *InstaJob* aims to simulate real interview conditions, the online nature of the platform could lead to different behavioral patterns from candidates compared to in-person interviews. Candidates may act more or less naturally in front of a camera, which could impact the accuracy of the AI's emotional and behavioral assessments. Moreover, technical issues like poor internet connections could cause delays or distortions in the video or audio, which would affect the quality of the analysis.

\subsection{Model Generalization}
There is also a threat to the generalizability of the models used in *InstaJob*. Although the system has demonstrated strong performance in controlled tests, the AI models may not perform as well when exposed to drastically different environments or job types not represented in the training data. Additionally, the use of specific datasets such as FER-2013 for facial emotion detection and GoEmotions for text emotion classification limits the system’s ability to generalize to real-world settings where the diversity of emotional expression and language use is broader.

\subsection{Ethical Considerations}
Another important factor to consider is the ethical implications of using AI in recruitment. While *InstaJob* aims to reduce bias, there is still the potential for the models to inadvertently reinforce biases if not properly calibrated or tested across diverse populations. There is also a risk of over-reliance on AI decision-making, which could lead to candidates being unfairly rejected based on erroneous model outputs.

In summary, while *InstaJob* provides significant advancements in AI-driven recruitment, it is essential to mitigate these threats by improving model diversity, ensuring data quality, and conducting further testing in real-world environments with diverse candidate pools.

\section{Conclusion}\label{conc}

This paper presents InstaJob, an AI-powered platform that enhances fairness and efficiency in the recruitment process by integrating emotion detection, speech analysis, and AI-based technical assessments. Future work will focus on improving the model's generalizability and expanding the system's features to cater to a broader range of industries.

\end{comment}
\section*{Conclusion}\label{conc}

This paper presents \textit{InstaJob}, an AI-powered framework that enhances fairness and efficiency in the recruitment process by integrating facial emotion detection, speech analysis, and AI-based technical assessments. The system utilizes datasets such as FER-2013 and GoEmotions to power its models and offers a scalable solution for conducting initial candidate evaluations.

While the platform demonstrates promising results, several limitations must be acknowledged. The variability in video and audio quality can affect the accuracy of the models, and there is a risk of biased outcomes due to cultural differences not fully represented in the training data. Additionally, the virtual interview format may not perfectly replicate in-person interactions, potentially impacting candidate behavior and model performance.

Future work will focus on improving model generalizability, expanding dataset diversity to reduce cultural and linguistic bias, and testing the platform further in real-world scenarios across various industries. It is also essential to address the ethical considerations of using AI in recruitment by ensuring transparency, fairness, and inclusivity in the decision-making process.






%\section*{Acknowledgments}
%Cras egestas velit mauris, eu mollis turpis pellentesque sit amet. Interdum et malesuada fames ac ante ipsum primis in faucibus. Nam id pretium nisi. Sed ac quam id nisi malesuada congue. Sed interdum aliquet augue, at pellentesque quam rhoncus vitae.

\nolinenumbers

\bibliography{ref}


% Either type in your references using
% \begin{thebibliography}{}
% \bibitem{}
% Text
% \end{thebibliography}
%
% or
%
% Compile your BiBTeX database using our plos2015.bst
% style file and paste the contents of your .bbl file
% here. See http://journals.plos.org/plosone/s/latex for 
% step-by-step instructions.
% 
% \begin{thebibliography}{10}

% \bibitem{bib1}
% Conant GC, Wolfe KH.
% \newblock {{T}urning a hobby into a job: how duplicated genes find new
%   functions}.
% \newblock Nat Rev Genet. 2008 Dec;9(12):938--950.

% \bibitem{bib2}
% Ohno S.
% \newblock Evolution by gene duplication.
% \newblock London: George Alien \& Unwin Ltd. Berlin, Heidelberg and New York:
%   Springer-Verlag.; 1970.

% \bibitem{bib3}
% Magwire MM, Bayer F, Webster CL, Cao C, Jiggins FM.
% \newblock {{S}uccessive increases in the resistance of {D}rosophila to viral
%   infection through a transposon insertion followed by a {D}uplication}.
% \newblock PLoS Genet. 2011 Oct;7(10):e1002337.

% \end{thebibliography}



\end{document}

