An AI-based framework for improving efficiency and fairness in the interview process

Mohannad Taman, Yahia Khaled, Dalia Sobhy

Abstract


Artificial intelligence (AI) technologies have advanced to the point where they can help human resource specialists, such as recruiters, by automating major parts of the hiring process and filtering the list of candidates. However, little research has evaluated the use of AI in virtual interviews. This paper presents InstaJob, an AI-powered framework designed to improve efficiency and fairness in the hiring process. It uses deep learning models for face emotion detection, text emotion analysis, and filler word detection in interviews to evaluate candidates’ soft skills, ensuring unbiased assessments. The proposed face emotion detection model achieved a validation accuracy of 77%, which outperforms the other state-of-the-art approaches.

Received on, 27 April 2025

Accepted on, 25 May 2025

Published on, 18 June 2025


Keywords


Artificial intelligence, virtual interviews, Facial emotion recognition, speech processing, Deep learning applications.

Full Text:

PDF

References


D. Jekauc et al., “Recognizing affective states from the expressive behavior of tennis players using convolutional neural networks,” Knowl Based Syst, vol. 295, p. 111856, Jul. 2024, doi: 10.1016/j.knosys.2024.111856.

S. K. Khare, V. Blanes-Vidal, E. S. Nadimi, and U. R. Acharya, “Emotion recognition and artificial intelligence: A systematic review (2014–2023) and research recommendations,” Information Fusion, vol. 102, p. 102019, Feb. 2024, doi: 10.1016/j.inffus.2023.102019.

C. Dando, D. A. Taylor, A. Caso, Z. Nahouli, and C. Adam, “Interviewing in virtual environments: Towards understanding the impact of rapport-building behaviours and retrieval context on eyewitness memory,” Mem Cognit, vol. 51, no. 2, pp. 404–421, Feb. 2023, doi: 10.3758/s13421-022-01362-7.

V. Coleman et al., “Lessons Learned From Conducting Virtual Multiple Mini Interviews During the COVID-19 Pandemic,” The Journal of Physician Assistant Education, vol. 35, no. 3, pp. 287–292, Sep. 2024, doi: 10.1097/JPA.0000000000000606.

“Artificial Intelligence Insights & Articles | QuantumBlack | McKinsey & Company,” 2023. [Online]. Available: https://www.mckinsey.com/capabilities/quantumblack/our-insights/

S. Yadav and S. Kapoor, “RETRACTED ARTICLE: Adopting artificial intelligence (AI) for employee recruitment: the influence of contextual factors,” International Journal of System Assurance Engineering and Management, vol. 15, no. 5, pp. 1828–1840, May 2024, doi: 10.1007/s13198-023-02163-0.

C. Fernández-Martínez and A. Fernández, “AI and recruiting software: Ethical and legal implications,” Paladyn, vol. 11, no. 1, pp. 199–216, May 2020, doi: 10.1515/pjbr-2020-0030.

E.-R. Lukacik, J. S. Bourdage, and N. Roulin, “Into the void: A conceptual model and research agenda for the design and use of asynchronous video interviews,” Human Resource Management Review, vol. 32, no. 1, p. 100789, Mar. 2022, doi: 10.1016/j.hrmr.2020.100789.

HireVue, “About the Company | Leadership & CEO | HireVue.” [Online]. Available: https://www.hirevue.com/about

A. Vinciarelli et al., “Bridging the Gap between Social Animal and Unsocial Machine: A Survey of Social Signal Processing,” IEEE Trans Affect Comput, vol. 3, no. 1, pp. 69–87, Jan. 2012, doi: 10.1109/T-AFFC.2011.27.

C. Signore, B. Della Piana, and F. Di Vincenzo, “Digital Job Searching and Recruitment Platforms: A Semi-systematic Literature Review,” 2023, pp. 313–322. doi: 10.1007/978-3-031-42134-1_31.

A. Fabris et al., “Fairness and Bias in Algorithmic Hiring: A Multidisciplinary Survey,” ACM Trans Intell Syst Technol, vol. 16, no. 1, pp. 1–54, Feb. 2025, doi: 10.1145/3696457.

C. Qin, L. Zhang, R. Zha, and D. Shen, “A Comprehensive Survey of Artificial Intelligence Techniques for Talent Analytics,” arXiv preprint , 2023, doi: https://dx.doi.org/10.48550/arXiv.2307.03195.

E. T. Albert, “AI in talent acquisition: a review of AI-applications used in recruitment and selection,” Strategic HR Review, vol. 18, no. 5, pp. 215–221, Oct. 2019, doi: 10.1108/SHR-04-2019-0024.

M. Kathiravan, M. Madhurani, S. Kalyan, R. Raj, and S. Jayan, “A modern online interview platform for recruitment system,” Mater Today Proc, vol. 80, pp. 3022–3027, 2023, doi: 10.1016/j.matpr.2021.06.459.

I. Naim, Md. I. Tanveer, D. Gildea, and M. E. Hoque, “Automated Analysis and Prediction of Job Interview Performance,” IEEE Trans Affect Comput, vol. 9, no. 2, pp. 191–204, Apr. 2018, doi: 10.1109/TAFFC.2016.2614299.

S. Mhadgut, N. Koppikar, N. Chouhan, P. Dharadhar, and P. Mehta, “vRecruit: An Automated Smart Recruitment Webapp using Machine Learning,” in 2022 International Conference on Innovative Trends in Information Technology (ICITIIT), IEEE, Feb. 2022, pp. 1–6. doi: 10.1109/ICITIIT54346.2022.9744135.

A. K. A, A. H, N. P. Nair, V. A, and A. T, “Interview Performance Analysis using Emotion Detection,” in 2022 4th International Conference on Inventive Research in Computing Applications (ICIRCA), IEEE, Sep. 2022, pp. 1424–1427. doi: 10.1109/ICIRCA54612.2022.9985667.

I. J. Goodfellow et al., “Challenges in Representation Learning: A Report on Three Machine Learning Contests,” 2013, pp. 117–124. doi: 10.1007/978-3-642-42051-1_16.

I. J. Goodfellow et al., “Challenges in Representation Learning: A report on three machine learning contests,” arXiv:1307.0414 [cs, stat], May 2013, [Online]. Available: https://arxiv.org/abs/1307.0414

M.-I. Georgescu, R. T. Ionescu, and M. Popescu, “Local Learning With Deep and Handcrafted Features for Facial Expression Recognition,” IEEE Access, vol. 7, pp. 64827–64836, 2019, doi: 10.1109/ACCESS.2019.2917266.

L. Pham, T. H. Vu, and T. A. Tran, “Facial Expression Recognition Using Residual Masking Network,” in 2020 25th International Conference on Pattern Recognition (ICPR), IEEE, Jan. 2021, pp. 4513–4519. doi: 10.1109/ICPR48806.2021.9411919.

A. Jaiswal, A. Krishnama Raju, and S. Deb, “Facial Emotion Detection Using Deep Learning,” in 2020 International Conference for Emerging Technology (INCET), IEEE, Jun. 2020, pp. 1–5. doi: 10.1109/INCET49848.2020.9154121.

J. Luo, Z. Xie, F. Zhu, and X. Zhu, “Facial Expression Recognition using Machine Learning models in FER2013,” in 2021 IEEE 3rd International Conference on Frontiers Technology of Information and Computer (ICFTIC), IEEE, Nov. 2021, pp. 231–235. doi: 10.1109/ICFTIC54370.2021.9647334.

S. Hassan, M. Ullah, A. S. Imran, and F. A. Cheikh, “Attention-Guided Self-supervised Framework for Facial Emotion Recognition,” 2024, pp. 294–306. doi: 10.1007/978-981-99-7025-4_26.

Y. Wu, L. Zhang, Z. Gu, H. Lu, and S. Wan, “Edge-AI-Driven Framework with Efficient Mobile Network Design for Facial Expression Recognition,” ACM Transactions on Embedded Computing Systems, vol. 22, no. 3, pp. 1–17, May 2023, doi: 10.1145/3587038.

Y. Mao, “Optimization of Facial Expression Recgonition on ResNet-18 using Focal Loss and CosFace Loss,” in 2022 International Symposium on Advances in Informatics, Electronics and Education (ISAIEE), IEEE, Dec. 2022, pp. 161–163. doi: 10.1109/ISAIEE57420.2022.00041.

Dumitru, I. Goodfellow, W. Cukierski, and Y. Bengio, “Challenges in Representation Learning: Facial Expression Recognition Challenge,” 2013. [Online]. Available: https://www.kaggle.com/c/challenges-in-representation-learning-facial-expression-recognition-challenge

D. Bhagat, A. Vakil, R. K. Gupta, and A. Kumar, “Facial Emotion Recognition (FER) using Convolutional Neural Network (CNN),” Procedia Comput Sci, vol. 235, pp. 2079–2089, 2024, doi: 10.1016/j.procs.2024.04.197.




DOI: https://dx.doi.org/10.21622/ACE.2025.05.1.1317

Refbacks

  • There are currently no refbacks.


Copyright (c) 2025 Mohannad Taman, Yahia Khaled, Dalia Sobhy

Advances in Computing and Engineering

E-ISSN: 2735-5985

P-ISSN: 2735-5977

 

Published by:

Academy Publishing Center (APC)

Arab Academy for Science, Technology and Maritime Transport (AASTMT)

Alexandria, Egypt

ace@aast.edu