Abstract

Music recommendation systems are transforming the way users interact with music streaming services by personalizing experiences. This paper presents a novel approach that leverages facial emotion recognition (FER) to suggest music dynamically based on users\' emotional states. Using machine learning and deep learning models, facial expressions are analyzed in real-time through video input. The system integrates emotion-based recommendations with collaborative and content- based filtering techniques to enhance accuracy. This approach offers a more intuitive and human- centered interaction compared to traditional systems. The proposed solution can be used across streaming platforms, personalized wellness apps, and therapeutic environmen ts.

Introduction

I. INTRODUCTION

Music has a profound effect on emotions, making it a valuable tool for personal well-being. Traditional recommendation systems rely heavily on listening history, metadata, or user preferences. However, they fail to adapt instantly to a user’s current emotional state. In this paper, we propose an emotion-based music recommendation system that analyzes users' facial expressions to suggest songs in real-time. Facial Emotion Recognition (FER) models help capture emotions such as happiness, sadness, anger, and surprise using video feed input. By combining content-based and collaborative filtering techniques, this recommendation engine personalizes playlists that align with the user’s mood. This work explores how integrating machine learning and computer vision can deliver a seamless musical experience tailored to momentary emotions, surpassing traditional recommendation systems.

Music plays a vital role in regulating emotions, enhancing moods, and offering a therapeutic effect. With the advent of streaming platforms, music recommendation systems have gained popularity by suggesting songs based on user preferences, listening history, or genre similarity. However, traditional recommendation models have limitations since they depend primarily on static data and cannot adapt instantly to the user’s current emotional state. A              more human-centric approach involves recognizing the user’s emotions in real-time and generating music recommendations accordingly. Facial Emotion Recognition (FER) provides a powerful tool to bridge this gap by capturing emotions like happiness, sadness,     anger,or neutrality through video input, allowing the system to deliver contextually relevant music recommendations. This combination enhances the user experience by aligning song suggestions with emotional needs.

The goal of this project is to create a dynamic, real- time music recommendation system that analyzes facial expressions and uses collaborative filtering and content-based filtering algorithms to suggest relevant music tracks. The system leverages computer vision technologies like OpenCV to detect facial features and pre-trained deep learning models to classify emotions accurately. Once the emotion is identified, the system matches it to a corresponding music category and fetches appropriate songs using APIs such as Spotify’s API. The seamless integration of these technologies ensures personalized music suggestions that reflect the user’s emotional state, making the experience more engaging. This innovation can have significant applications in entertainment platforms, wellness solutions, and therapeutic environments, demonstrating how artificial intelligence can enhance personalization beyond conventional recommendation techniques.

II. LITERATURE SURVEY

Kumar et al. (2022) discussed integrating facial emotion recognition with recommendation systems to improve engagement by suggesting mood- specific content. The system used convolutional neural networks (CNN) for real-time face detection and emotion extraction.[1]

Zhang & Chen (2021) demonstrated how FER models coupled with collaborative filtering  improved user satisfaction by addressing emotional needs. They found that FER-based systems outperformed traditional models in delivering personalized content.[2]

Wen et al. (2020) explored the use of OpenCV with machine learning algorithms for emotion classification, achieving promising results with the Haar cascade classifier for face detection.[3]

Li et al. (2019) focused on building lightweight FER models for mobile platforms to enhance user experience by delivering personalized services such as playlists and wellness tools.[4]

Park et al. (2021) developed a hybrid filtering system combining collaborative and content-based methods to improve recommendation performance for emotional well-being applications.[5]

Deng et al. (2018) presented a comparative analysis of various deep learning models for emotion recognition, concluding that CNNs performed better on emotion datasets such as FER-2013.[6]

Soleymani et al. (2019) discussed how emotion recognition technologies enhance user engagement across multimedia platforms, including music streaming services.[7]

Singh & Gupta (2020) utilized deep neural networks to classify seven emotions, suggesting that emotion- based recommendations significantly improve engagement metrics.[8]

Chowdhury et al. (2023) integrated Spotify’s API with FER systems to deliver music recommendations, showing improved user satisfaction through emotion-based interactions.[9]

Mollah et al. (2022) investigated the impact of real- time emotion detection on music selection, showing that such systems can be applied in therapy and meditation applications.[10]

III. PROBLEM STATEMENT

Traditional music recommendation systems rely on historical data, such as user preferences, playlists, and past listening patterns. However, these methods fail to consider real-time emotional states, resulting in a mismatch between users’ moods and the recommended music. This paper addresses the challenge of providing emotion-based music recommendations by developing a system that captures users' facial expressions in real-time to suggest music that aligns with their mood.

IV. METHODOLOGY

The proposed system leverages Facial Emotion Recognition (FER) and machine learning-based recommendation algorithms to suggest music dynamically. The workflow starts with capturing a live video feed of the user's face through a webcam. OpenCV processes the frames, and Haar cascade classifiers or MTCNN detect facial features. A pre- trained deep learning model (like VGGFace or FER- 2013) classifies the emotions (happy, sad, angry, neutral, etc.).

The detected emotion triggers a recommendation algorithm—combining content-based filtering (matching songs with emotional metadata) and collaborative filtering (using other users’ preferences for similar moods). Spotify’s API or another music streaming API integrates the system with a wide database of songs. The final step is delivering an emotionally-aligned playlist to the user through a web or mobile interface.

The architecture ensures real-time performance by caching frequent data using Redis and performing asynchronous processing for smooth user experience. Docker containers manage dependencies and ensure seamless deployment on cloud platforms.

V. MODULES USED IN THE PROJECT

1. Facial Emotion Recognition Module: Uses OpenCV and CNN-based models to classify emotions.
2. Recommendation Module: Combines collaborative and content-based filtering.
3. API Integration Module: Uses Spotify API to fetch music tracks based on recommendations.
4. Database Module: Stores user preferences and music metadata (MySQL/MongoDB).
5. Frontend Module: Web interface to capture video input and display music suggestions.
6. Backend Module: Flask/Django server to handle requests and process video data in real-time.

VI. ALGORITHM

Step 1: Capture video feed from the webcam using OpenCV.

Step 2: Detect the face using a Haar Cascade or MTCNN.

Step 3: Extract facial landmarks and classify the emotion using a pre-trained CNN.

Step 4: Match the detected emotion to a suitable music category.

Step 5: Use collaborative and content- based filtering to select songs.

Step 6: Fetch relevant tracks from Spotify API.

Step 7: Display recommendations on the web interface.

VII. SYSTEM ARCHITECTURE

Fig: System Architecture

Conclusion

This paper presents a facial emotion recognition- based music recommendation system that dynamically suggests songs based on the user’s emotional state. The system overcomes the limitations of traditional recommendation engines by focusing on real-time emotional context, enhancing user experience. The combination of machine learning, deep learning, and API integration delivers a personalized, intuitive make the system even more versatile by including voice recognition and context-awareness.

References

[1] Kumar, R., et al. (2022). Facial Emotion Recognition and Personalized Music Recommendation Systems. IEEE Transactions on Affective Computing. [2] Zhang, X., & Chen, L. (2021). Hybrid Filtering for Music Recommendations Using Facial Expressions. ACM Multimedia. [3] Wen, Y., et al. (2020). Real-Time FER Models for Music Streaming Services. IEEE Access. [4] Li, H., et al. (2019). Lightweight FER Models for Mobile Platforms. Elsevier Neurocomputing. [5] Park, J., et al. (2021). Hybrid Music Recommendation Systems. IEEE Transactions on Multimedia. [6] Deng, J., et al. (2018). Comparative Analysis of CNNs for Emotion Recognition. Springer Multimedia Tools. [7] Soleymani, M., et al. (2019). Impact of Emotion Recognition on User Engagement. IEEE Transactions on Multimedia. [8] Singh, S., & Gupta, P. (2020). Deep Neural Networks for Emotion Classification. Elsevier Procedia Computer Science. [9] Chowdhury, K., et al. (2023). Integration of FER with Music APIs. IEEE Transactions on Affective Computing. experience that can be utilized across various [10] Mollah, A., et al. (2022). Emotion Detection in domains, including wellness applications and music streaming platforms. Future enhancements could Therapy Applications. IEEE Access.

Copyright

Copyright © 2024 Avadhut P. More, Swanand P. Gholap, Aniket A. Gayke, Uddhav D. Hon, Sharad M. Rokade. This is an open access article distributed under the Creative Commons Attribution License, which permits unrestricted use, distribution, and reproduction in any medium, provided the original work is properly cited.