Recipe Recommendation by Ingredients Detection

Abstract

Cooking is a daily chore for many individuals and hence can become a monotonous activity. Sometimes it is difficult for one to decide what new & interesting can be cooked from the available ingredients. Lack of culinary expertise can lead to disproportionate use of ingredients resulting in wastage. Also while cooking, one’s hands are unclean to be able to look the recipe in their smart device or recipe books. For beginners cooking can also become a hassle because they might not know names of many ingredients. Our proposed system intends to help everyone with their daily cooking. Using our application one will be able to escape their monotonous cooking routine by using ingredients detection system along with recommendation model. Using our application user will be able to scan the available ingredients through which we will recommend best suitable recipe for them. For detecting the ingredients we will apply CNN algorithm with parameter optimization. CNN is the state-of-the-art approach to the deep learning, with significantly higher accuracy then traditional support-vector-machine-based algorithm. Our application will store the detected ingredients of the user in the database and recommend recipe accordingly in the home page. Further recommendation on the home page would be based on the historical patterns of the user, while recommending we will be using various factors such as cuisine preference, ingredients, cooking time, etc. Using NLP we will serve the voice control feature to the users through which they will be able to hear the recipe at their own pace, in case their hands are unclean. Through NLP our motive is to overcome the regional barrier so anyone unfamiliar with English will also be able to use our application and can experience the culture of different regions through their cuisine.

Introduction

I. INTRODUCTION

Cooking is a daily chore for many individuals and hence can become a monotonous activity. Sometimes it is difficult for one to decide what new & interesting can be cooked from the available ingredients. Lack of culinary expertise can lead to disproportionate use of ingredients resulting in wastage. Due to most of the information available in English it becomes difficult to cater regional population. For beginners cooking can also become a hassle because they might not know names of many ingredients.

One person households have increased in the recent times worldwide making it necessary for an individual to learn to cook for themselves.

A report by business wire [7] shows that single-person household will record 128% growth “between” 2000-2030. According to a report by The Washington Post [6] nearly 60% of millennials don’t know how to prepare a simple salad and around 37% of them didn’t even knew what butter knife was.

So the point being that in the world right now where the necessity of being independent is increasing, percentage of people possessing the skill of cooking, which is a very basic and daily chore is significantly low. Also the pandemics such as corona virus has shown the world that being independent in terms of cooking can be highly beneficial for their own health. To tackle these problems it would be efficient to provide a platform that is easy to use and beginner friendly so that individuals would be more motivated to cook for themselves. It is only possible to accomplish this if we consider the fact that many beginners may not know the name of various ingredients. Since we are making a platform whose target audience is worldwide, the fact that many ingredients will have different names according to the location is also to be considered and since we are providing the ingredients detection feature, these problems gets completely eradicated. Even providing these features would not be enough, to encourage the user we would be providing recipes from various cuisines based on their historical data to keep them motivated. Factors such as preferred cuisines, cooking time and ingredients would be considered to provide relevant suggestions. Based on our target audience which may consist of many non-English speakers, providing those recipes in English would not make any sense and hence we provide multilingual feature in our platform through users will be able to view recipes in their own preferred language. Natural language Processing has been around for nearly five decades and has evolved overtime till a point that now it can very accurately translate between various languages. By harnessing the power of NLP we wish to accurately translate our recipes and provide user with the most relevant data.

II. LITERATURE SURVEY

Keiji Yanai et al. have proposed to cook using generic object recognition technology. They assume that the proposed system works on a smartphone which has builtin cameras and Internet connection such as Android smartphones and iPhones. They intend a user to use their system easily and intuitively during shopping at grocery stores or supermarkets as well as before cooking at home. A user can instantly acquire a recipe list from internet cooking recipe databases by pointing food ingredients with a mobile phone's built-in camera. With their system, a user can get to know the cooking recipes related to various kinds of food ingredients unexpectedly found in a grocery store including unfamiliar ones and bargain ones on the spot. To do so, the system recognises food ingredients in photos shot with built-in cameras and searches online culinary recipe databases for recipes that has the same ingredients. As an object recognition method, they adopt bag-offeatures with SURF and color histogram extracted from not single but multiple images as image features and linear kernel SVM with the one-vs-rest strategy as a classifier [1].

ZhengXian Li et al. have proposed a scalable recommendation system based on Apache Spark are designed and implemented for personalized recipe recommendation for mobile apps. In the recommendation system, they also propose a hybrid recommendation algorithm, which combines content-based filtering and collaborative filtering, to improve the recipe recommended effectiveness. In the end, they evaluate the performance of the recommendation system [2].

Lipi Shah et al. have proposed the aim of this work is to find which algorithm is suitable for personalized recipe recommendation by comparing the evaluation parameter RMSE of different conventional algorithms. They have focused on two types of data. In these approaches, they divided each recipe into different features like ingredients, different preparation directions like less than 30 minutes, three steps or less, less than 60 minutes etc., different cuisines like Indian, Asian, American, Italian etc., different dietary like low fat, high protein, high carbohydrate etc., different occasions like diwali, birthday, summer, dinner party etc., different courses like dessert, main dishes, salad etc...One is fine grained data of recipes for their different features and second is the high level data that is rating for recipes by each user. Here they have proposed, two hybrid approaches using content and rating information on recipes [3].

S.T Patil et al. have proposed Recommendations based on Recipe Ingredients Recipe ingredients are used to calculate user ingredients in recipe ingredients based recommendation approach. The user and recipe vectors are constructed based on ingredients as orthogonal terms, or the dimension of the vectors. The following diagram depicts a high-level overview of the recipe ingredient-based recommendation approach. Extract all recipe ingredients from recipe data, Obtain unique ingredient terms with corresponding quantities. Ingredients should be spitted into two sets. For example, ingredients separated with and/or/instead are split into unit set ingredients. Quantity associated with ingredients for the recipes are considered as the frequency of ingredients occurrence in the recipe, Frequencies from all recipes as terms and quantities will be constructed Calculate the maximum ingredient frequencies of the recipes, as well as the total number of recipes, as an inverted index, and display all recipes depending on ingredients [4].

Alkatai Gholve et al. have proposed In Proposed system they analysing the Android application. The user can register the application. Once the registration is successful, user can login the application. Now the User can upload images of ingredients. These images are displayed in Admin dashboard. The role of an admin is to view this image and substantiate the image. User can also add a new recipe and give the ratings for a particular recipe [5].

III. PROPOSED SYSTEM

Whilst using Makeat, the users can scan the ingredients available with them which will be used to provide them with various recipes that could be made by them. The ingredients detected along with the recipe that the user cooked amongst the various recommended would be stored in the database. Through the historical information of the user stored in the database considering various factors, new recipes can recommended to the user on the home page. The user will also have option of viewing the recipes in different languages supported by the platform according to their preference.

YOLO v5: It is state of the art and newest version of the YOLO object detection series, YOLOv5 set the benchmark for object detection models very high, it already beats the EfficientDet and its other previous YOLOv5 versions. The objective of Object Detection in Artificial Intelligence is to detect items in photographs. Yolo V5 is currently one of the proficient models for Object Detection. The beautiful thing about  Deep Neural Network is that retraining it on your own custom dataset is a breeze.

Hybrid based recommendation system: Hybrid based recommendation systems combine two or more recommendation strategies in different ways to benefit from their complementary advantages. Hybrid techniques can be applied in a variety of ways, including producing content-based and collaborative-based predictions separately and then merging them; adding content-based capabilities to a collaborative-based approach (and vice versa); and unifying the approaches into a single model.

Several studies comparing the performance of hybrid methods to pure collaborative and content-based methods have shown that hybrid methods can deliver more accurate suggestions than pure approaches. These methods can also be used to overcome some of the common problems in recommendation systems such as cold start and the sparsity problem, as well as the knowledge engineering bottleneck in knowledge-based approaches. A hybrid based recommendation system can use any of the recommendation system approaches such as collaborative filtering, content based filtering, demographic based filtering, knowledge based filtering, etc.

Natural Language Processing: Natural language processing enables computers to converse with humans in their native tongue and scales other language-related activities. NLP allows computers to read text, hear voice, analyse it, gauge sentiment, and identify which bits are significant, for example. Machines can now analyse more language-based data than people, without becoming fatigued, and in a consistent and fair manner. Considering the staggering amount of unstructured data that’s generated every day, from medical records to social media, automation will be critical to fully analyse text and speech data efficiently.

IV. SYSTEM ARCHITECTURE

Ingredients Detection Model: The ingredients detection model will be initialized when the user opens the camera, as soon as the camera is opened the frames are inputed into the YOLOv5 model, YOLOv5 is state of the art object detection algorithm which we have custom trained to detect various fruits, vegetables, etc. The YOLOv5 will try to detect the ingredients in the frames with the help of convolution neural networks, Residual blocks, Bounding box regression, Intersection Over Union (IOU), etc and give the names of the detected ingredients as output. This output will be further transferred to the recipe recommendation model.

Recipe Recommendation Model: The names of the detected ingredients are used to query the recipes from firebase. Firebase is a NoSQL database which is flexible and scalable in terms of size, it is also a real time database which provides offline access and real time updates. A string matching library called as fuzzywuzzy would be used to query the recipes from firebase to give the most efficient results as it will match ingredients in recipes such as “chopped tomatos” with “tomatos”. The result of the query would be displayed to the user, the user can select any preferred recipe from the suggested recipes to cook. The user preferred recipe would be stored in firebase for hybrid recommendation model.

NLP Model: After the user has selected the preferred recipe, the user will have a choice to change the language of the recipe to a desired language. Natural language processing would be used to change the language of the recipe to a user preferred language. If the language is already user preferred than the user can use the voice controlled instructions feature to read out the instructions of the recipe. The user can also use this feature to control the instructions and to go back or move forward a instruction. This feature will help user to cook seamlessly without having to interact with the screen with messy hands hence, improving the overall user experience.

Hybrid Recommendation Model: This model is used to further recommend recipes without ingredients detection model based on historical data of the user. The result of this model would be displayed on the home page. Every time a user interacts with a recipe the data related to that recipe would be stored in firebase. This model will first query the historical data of the user from the firebase, this data would then be used in the hybrid recommendation model built upon recommendation systems such as collaborative filtering, content based filtering, knowledge based filtering, etc. The final combination of recommendation systems to be used in our hybrid recommendation model would be decided upon after various testing. Our hybrid recipe recommendation model will try to tackle problems of non-hybrid recommendation systems such as cold start problem, grey sheep problem, popular bias, etc.

The modules are:

A. User Registration

For this module, we will build a cross platform app on which the user will register themselves. This module will encapsulate all the backend required for the project along with storing the user data which can be used in the other modules. This module will also include GUI part of the project.

B. Ingredients Detection

This module will include the scanning and detection of ingredients. A CNN based algorithm will be used for this feature. The user would be able to scan the ingredients in real time using the application and the frames collected during the procedure will be fed to the detection algorithm which will in return output the detected ingredients.

???????C. Personalised Recommendation

This module will come into play when the user has detected some ingredients and interacted with few of the recommended recipes. All this data will be stored in the database to give the user personalised recipe recommendations. More the data collected, more the efficiency of the system to recommend recipes. The module will be designed considering the cold start and grey sheep problem.

The most crucial module of the app, ingredients detection module is built by custom training the weights of YOLOv5 model. For training the model we created our own custom dataset consisting of 1296 unique images which accumulated for a total of 39 classes consisting of 20 fruits and 19 vegetables. Each class consists of 25-35 training images and 5-10 testing images. After training the YOLOv5 model for 300 epochs, we achieved an accuracy of 87%.

Elaborating on the system we created as a whole, the user will start in the app module in which the user will either create a new account or login in an existing one. The interface of the app is created using the flutter framework and all user data is stored in firebase. After logging into the app the user can detect the available ingredients with them, this process will require user to use the camera and detect via the YOLOv5 model used with tflite plugin. The frames captured from the camera would be continuously fed in the inference of the YOLOv5 custom trained model, The model will in return output the names of the detected ingredients. Based on the detected ingredients some recipes will be recommended to the user, We used fuzzywuzzy plugin to match the ingredient names with the ingredients in the recipe dataset so that detected names like ‘Apple’ could be matched with ‘fresh cut Apple’ in the recipe dataset. The user can use the NLP feature to voice control the recipe instructions as well as change the text language. On the home page of the app module the user will get some recipe recommendations based upon the historical data of the user and the output of the Hybrid based recommendation system, based on user preferences set in the app and the recipes liked by the user.

The fig 8 represent splash screen followed by home page of the app. Where waiting time of splash screen will be about 8-9 seconds to load assets and required app data to the user interface. Further the home page contains search and advance search for the recipes considering title, time, calories and much more parameters of the recipe. Along with search there is responsive card view of the recipes with title and some parameters. Bottom bar to navigate through different sections of the app.

The fig 9 represent swipe cards of the recipes, which will be evaluate and display according to ingredients scanned by the users. Here we’ve took care of each end-users by providing instructions of how swipe cards will work as by left and right tags over the recipe swipe cards. Left swipe will directly discard that recipe from the current list of recipes. Right swipe will open the detailed recipe page which is present in the next figure.

The fig 10 represent recipe page and profile page. Recipe page contains image and detailed instructions about the recipe. Along with the detailed instructions app will fetch bunch of parameters of the recipe like cook time, calories, ratings, and many nutrition’s, etc. The page also has the features like bookmark and like the recipe along with the Chef bot (Voice Assistant) which will use NLP to step by step readout recipe instructions. Profile page where users can change their details, profile picture, etc. The page also contains preferences of various parameters where user can set their time and nutrition’s along with Vegan, Veg and Non-veg preference of the recipe.

VI. ACKNOWLEDGMENT

We take this opportunity to express our deep sense of gratitude to our project guide Mrs. Silviya D’monte for her continuous guidance and encouragement throughout the duration of our final year project work. It is because of her experience and wonderful knowledge, we can fulfil the requirement of completing the final year project within the stipulated time. We would also like to thank Dr. Jitendra Saturwar, Head of computer engineering department and Mrs. Vishakha Shelke, Mr. John Kenny, Project Coordinator for their encouragement, whole-hearted cooperation and support.

We would also like to thank our Principal, Dr. J. B. Patil and the management of Universal College of Engineering, Vasai, Mumbai for providing us all the facilities and the work friendly environment. We acknowledge with thanks, the assistance provided by departmental staff, library and lab attendants.

Conclusion

In the world, the number of single person household is going to increase tremendously in the next decade. Also the number of millennials not having basic knowledge of cooking is large. Since the number of single person household is going to keep on increasing, it becomes necessary for any individual to have a basic skill of cooking. Many beginners who are just starting out to learn to cook don’t know the name of many ingredients as well as recipes and hence it may become a challenge for them to cook. Also the present technology which provides recipes are usually not beginner friendly as well as are in English and don’t cater to regional population. With our proposed system Makeat, we have provided a beginner friendly platform which does not require to have any knowledge of cooking as well as names of ingredients. To encourage the user to try out various dishes and eat self-made food we have recommended the user with relevant recipes based on their historical data. Through NLP, we have also provided Multilanguage support, so regional users can experience different cultures through their cuisines. Overall Makeat can boost the percentage of people who can cook for themselves and encourage people to eat self-made food. In the future, the number of supported ingredients can be increased by providing a broad dataset. The accuracy of the model can be improved drastically if more than 300 images are used per class to train the model. Language translation can be implemented on every stage of the system so that it communicates with diverse group of people.

References

[1] Keiji Yanai, Takuma Maruyam, Yoshiyuki Kawano (2013), “Recipe Recommendation System with Visual Recognition”, IEEE. [2] ZhengXian Li, Jinlong Hu, Jiazhao Shen, Yong Xu, (2016), “A scalable recipe recommendation system for mobile application”, IEEE. [3] Lipi Shah, Hetal Gaudani, Prem Balani (2016), “Recipe Recommendation System using Hybrid Approach”, IJARCCE. [4] S.T Patil, Rajesh D. Potdar (2019), “Recipe Recommendation Systems”, IOSR Journals. [5] Alkatai Gholve, Nidhi Mishra, Revati Gaikwad, Shruti Patil (2019), “Recipe Recommendation System”, IJARCCE. [6] Washington Post, “Do millennials really not know how to cook? With technology they don’t really have to”, washintonpost.com [7] Business Wire, “Single-Person Households to Record 128 Percent Growth by 2030”, businesswire.com [8] https://www.kaggle.com/irkaal/foodcom-recipes-and-reviews

Copyright

Copyright © 2022 Mrs. Silviya Dmonte, Mrs. Parth Chudasama, Mrs. Meet Turakhia. This is an open access article distributed under the Creative Commons Attribution License, which permits unrestricted use, distribution, and reproduction in any medium, provided the original work is properly cited.