Smart Trolley System Using OpenCV

Abstract

The shopping atmosphere of the malls and commercial places nowadays is confronted by problems such as long queues at checkout counters and inefficiencies in tracking purchased items. This paper introduces the “Smart Trolley System using OpenCV,” a frontier of computer vision and the IOT present in a retail. Through a unique configuration of the web cameras and IoT devices, it is now possible to track shoppers via colour bands, rejuvenating inventory management and curtailing checkout queues. This system revamps the shopping experience with a new approach of offering convenience and saving time in the crowded malls. It deals with the important issues of checkout queues, manual item tracking, limited automation, exit management, and real-time data integration. The study analyses various technologies such as pre-trained neural networks, Arduino, Xbee, and barcode scanners, and RFID technology to improve the shopping experience and customer satisfaction.

Introduction

I. INTRODUCTION

The retail industry, mainly in large malls and commercial places, is constantly confronting various major problems which influence customer satisfaction and the smooth running of operations. These include delayed payments, manual item tracking, poorly used automation, problems with exit procedures, and lack of integration of real-time data. Such issues imply that shopping is not as it ought to be, and this irritates customers and it is a hindrance to the operation of some retailers. Our Smarter Shopping Trolley using OpenCV, a solution that employs advanced computer vision techniques and IoT technologies, would be the best way to deal with these various problems.

The system is made up of a complex assembly of parts that have been developed recently, such as webcams for tracking the customers through the use of colour bands, pre-trained neural networks for the recognition of products, Arduino and Xbee modules for product scanning in a seamless manner and also, real-time data display, and RFID technologies combined with Wi-Fi for the case of automated billing and immediate data synchronization. The automation of core activities and the uninterrupted flow of data between the shopping trolleys and the stores inventory system is what the Smart Trolley System intends to do in order to transform the retail sector.

This will most likely lead to lesser checkout times, better accuracy of inventory management, enhanced customer convenience with payment options online directly from the trolley, and lastly, it will increase the overall operational efficiency for the retailers. This all-around strategy, in fact, presents solutions to the widespread problem areas shoppers usually encounter and at the same time it allows retailers to keep up with the constantly changing preferences of such modern consumers.

II. LITERATURE REVIEW

To make billing process more efficient author have suggested a way to use RFID tag to tag items and use email application and wife module to manage bill. The primary aim is to reduce staff dependence while also providing an efficient and enjoyable process [1]. The system developed by authors mainly tackles the problem which was faced during covid that is social distancing, the proposed system includes the use of barcode scanner instead of RFID. Further with the use of LCD to display products price, quantity and total amount and also using raspberry pie for payment which   streamline payment process [2]. Authors have focussed mainly on making the billing process as simple as possible, with the use of Arduino and Xbee module with RFID for scanning item a system is implemented which shows customers real time cost and also allow them to pay the bill [3]. The authors have mainly focused on helping retail shopkeepers to enhance their operation so that they don’t lose their business to online retail store. Many technologies like iBeacons, Bluetooth and interactive displays, Natural language processing and speech-to-text technology have been suggested to authors to enhance in house experience of retail marts [4]. Another approach is suggested by authors which include use of RFID, LCD display, Wifi module ESP8266 and amazon cloud services to streamline the billing process and make shopping experience in malls efficient and enjoyable [5].

Shoplifting is a major problem which needs to be addressed, So in the light of it the authors have come up with a  design ShopSmart that  tackles  the problem while  also automating the billing process with the use of  RFID tags ,mobile application and iot components[6]. While using RFID and iot devices network connectivity is an important factor. But sometimes the connectivity issue can happen which will hinder the streamlined billing process so to tackle this author have Proposed a system that utilizes OpenCV for object detection, eliminating the need for network connectivity. [7]. Sometimes the use of RFID for an item may not be enough as sometimes it may happen that item did not get tagged which can leads to inefficient billing, to overcome this the authors had proposed the use of load cell which can help in preventing this problem [8]. Post covid-19 there was an increase demand of touchless technology, to overcome these need authors have designed a way to make intelligent cart which follows the customer, the system includes the use of inbuilt sensor with solar based lights [9]. To make trolly smarter the authors have proposed the use of technologies like RFID, magnetometers and wireless communication protocols like MQTT which allows real time tracking and interaction [10]. In making the cart more automated authors have suggested a way to use ultrasonic devices, barcode scanner and Channel and spatial reliability tracking (CSRT) algorithm for tracking shopper [11]. Another way to make cart follow shopper is to do it through iot by using mobile application with integrating RFID scanner.[12] Since the integration of technologies like barcode, LIDAR and many more makes it more expensive. To reduce its cost author suggests the use of smartphone sensor pedestrian (PDR) combined with map-matching algorithm [13]. Using computer vision in tracking a moving object come with many problems like when the tracking has to be done in crowded area, to address this problem authors suggest the integration of YOLO (You Only Look Once) detection algorithms with CSRT [14]. Authors have suggested a way to Ultra-Wideband (UWB) range-based localization and RFID technology to improve library operation that includes shelfing and following [15].

The papers highlight various technologies which enhances retail operations with a focus on RFID for item tagging and billing efficiency. Many technologies like touchless systems, smart carts with bar scanner and OpenCV, and real-time tracking using algorithms YOLO and PDR have been proposed. The integration of these technologies aims to improve accuracy, reduce manual labour and enhance customer satisfaction which reflects a trend towards smarter and more efficient retail solutions.

III. METHADOLOGY

Fig-1 showing the working of billing system

In fig 1. The billing system is built around the ESP32 microcontroller which is the one that controls the different devices. RFID modules gather information from the RFID tags fixed on the products and thus enable the real-time monitoring of the items that are in or out of the trolley. The ESP32 processes this data and shows it on a display, allowing customers to check how many purchases and the total bill should be. Through a push button, the shopper can communicate with the system e.g. by either confirming the purchases or clearing the items while a buzzer is giving sound feedback for the actions like adding or removing items or giving wrong alerts. The whole apparatus is battery-operated and, therefore, can function independently without the constant power supply of an external source. This RFID technology incorporation with a friendly interface smoothest the shopping process, minimizes the checkout time, and at the same time it makes the customers more satisfied with its overall service.

Fig -2 Showing system integration with cv

 Fig 2 shows the working model of a complicated Smart Trolley System with OpenCV that is meant to bring automation to and improve the retail shopping experience. The main piece of equipment in the system is the microcontroller ESP8266, which is responsible for the proper functioning of all the components connected to it. An ultrasonic sensor is used to detect obstacles and thus steer the vehicle clear of the obstacles. The ESP8266 sends commands to a motor driver that controls two motors (Motor 1 and Motor 2), allowing the trolley to move. In order to track and item recognition, a phone camera is used together with the Droid Cam application, which streams video to an OpenCV-based system over Wi-Fi. This arrangement makes it possible to process video feeds in real-time for the efficient tracking of shoppers and items. The system is powered by a battery; therefore, it does not need any external power sources to operate. Wireless fidelity (Wi-Fi) enables flawless correspondence among the phone camera, Droid Cam, and the OpenCV system, thus ensuring precise and agile tracking. This integrated strategy not only automates the trolley's movement and item recognition but also delivers a powerful real-time solution to speed up shopping processes, minimize wait times, and raise customer satisfaction thus augmenting the overall shopping experience.

Fig-1 circuit diagram for billing system

The circuit diagram above illustrates a smart shopping cart system that employs an ESP-32 microcontroller and an EM-18 RFID reader module. The ESP-32 is a multifunctional MCU that has Wi-Fi and Bluetooth built-in and it communicates with the EM-18 to read RFID tags which have the unique IDs of the products. The information is processed by the ESP-32 and sent to a 16x2 I2C LCD screen, which displays the item's name and price on the screen. The system also contains LEDs and a buzzer for visual and auditory feedback; thus, user interaction is enhanced. The switch (CMD1) gives customers the chance to insert items to their electronic cart first of all showing a price that constantly changes as they make purchases. This arrangement is to make the shopping experience more convenient by minimizing errors, making the process faster and easier for the customers, and providing the customers with the accuracy of their purchases by enabling them to scan the products as they shop.

This undertaking entails the interconnection of several individual parts to come up with a fully-fledged system. The components are, for instance, a Liquid Crystal I2C LCD (16x2) for display, an RFID Reader for identification, LEDs as visual feedback, a buzzer for audio alerts, a push button for user input, and the internal Wi-Fi module of the ESP32 for connectivity. The second part of the prototype entails a human-following shopping cart using let Arduino Uno, USB Camera, DC Motors with Motor Driver (L298N), and LED, Arduino handled serial communication and motor control via specific digital and PWM pins, phone camera interfaced with computer running Python script and Droid Cam application for Wi-Fi-based communication.

IV. RESULTS

By making use of our model, shopping time got a lot quicker and customer service got a lot more convenient. The connection of the ESP8266 microcontroller, ultrasonic sensors, motor drivers, and an OpenCV-based visual tracking system made it possible for the trolley to navigate through obstacles and avoid collisions, thus, the trolley moved without any interruption. The phone camera with Droid Cam contributed to the project significantly by allowing real-time item recognition and shopper tracking which enabled checkout times to be reduced by automating the billing process and allowing direct payments from the trolley. The real-time data integration over Wi-Fi made it possible for the inventory management to be updated accurately and in an instant, the transaction was reflected in the store's central database, thus, improving the overall operational efficiency. The easy-to-use interface composed of a display, push button, and a buzzer as auditory feedback made the customer experience intuitive and engaging even from the first hand. The deployment of the prototype was overall effective in overcoming typical retail difficulties, such as the prolonged waiting lines and the improper tracking of items, at the same time, it offered a cutting-edge, swift, and pleasurable method of shopping, hence the invention of further developments in retail automation was the first step.

References

[1] T. K. Das, A. K. Tripathy and K. Srinivasan, \"A Smart Trolley for Smart Shopping,\" 2020 International Conference on System, Computation, Automation and Networking (ICSCAN), Pondicherry, India, 2020, pp. 1-5, doi: 10.1109/ICSCAN49426.2020.9262350. [2] S. K. Shankar, S. Balasubramani, S. A. Basha, S. Ariz Ahamed and N. S. Kumar Reddy, \"Smart Trolley for Smart Shopping with an Advance Billing System using IoT,\" 2021 5th International Conference on Computing Methodologies and Communication (ICCMC), Erode, India, 2021, pp. 390-394, doi: 10.1109/ICCMC51019.2021.9418348. [3] A. Kumar, A. Gupta, S. Balamurugan, S. Balaji and R. Marimuthu, \"Smart Shopping Cart,\" 2017 International conference on Microelectronic Devices, Circuits and Systems (ICMDCS), Vellore, India, 2017, pp. 1-4, doi: 10.1109/ICMDCS.2017.8211723. [4] I. Rogojanu, G. Suciu, M. -C. Ditu and A. Pasat, \"Smart Shopping Technologies for Indoor Markets,\" 2018 IEEE International Conference on Computational Science and Engineering (CSE), Bucharest, Romania, 2018, pp. 99-103, doi: 10.1109/CSE.2018.00020. [5] A. Sutagundar, M. Ettinamani and A. Attar, \"Iot Based Smart Shopping Mall,\" 2018 Second International Conference on Green Computing and Internet of Things (ICGCIoT), Bangalore, India, 2018, pp. 355-360, doi: 10.1109/ICGCIoT.2018.8752971. [6] B. B. G. Reddy, S. Keerthi Vasan, S. Sundar, S. Sachin Ramsangu and T. Anjali, \"ShopSmart – Smart Shopping Application,\" 2020 International Conference on Communication and Signal Processing (ICCSP), Chennai, India, 2020, pp. 0818-0822, doi: 10.1109/ICCSP48568.2020.9182286. [7] D. Belibov and D. S. Tudose, \"Smart Shopping Cart,\" 2019 18th RoEduNet Conference: Networking in Education and Research (RoEduNet), Galati, Romania, 2019, pp. 1-6, doi: 10.1109/ROEDUNET.2019.8909680. [8] S. Shailesh, P. Shrivastava Deb, R. Chauhan and V. Tyagi, \"Smart Trolley,\" 2021 International Conference on Advance Computing and Innovative Technologies in Engineering (ICACITE), Greater Noida, India, 2021, pp. 242-245, doi: 10.1109/ICACITE51222.2021.9404582. [9] S. Y. Wong et al., \"An Intelligent Shopping Cart with Automatic Following Function,\" 2023 IEEE International Conference on Cybernetics and Intelligent Systems (CIS) and IEEE Conference on Robotics, Automation and Mechatronics (RAM), Penang, Malaysia, 2023, pp. 50-54, doi: 10.1109/CIS-RAM55796.2023.10370018. [10] S. Sanghavi, P. Rathod, P. Shah and N. Shekokar, \"Design and implementation of a human following smart cart,\" 2020 Third International Conference on Smart Systems and Inventive Technology (ICSSIT), Tirunelveli, India, 2020, pp. 1142-1149, doi: 10.1109/ICSSIT48917.2020.9214217. [11] N. D. Perez, J. Villaverde, W. Cereneo, A. Custodio, S. B. Conde and L. Bulawan, \"Design and Development of a Raspberry Pi-Based Shopping Cart Following Robot through Computer Vision and Object Tracking Method,\" 2022 IEEE Region 10 Symposium (TENSYMP), Mumbai, India, 2022, pp. 1-6, doi: 10.1109/TENSYMP54529.2022.9864520. [12] G. Pradeepkumar, V. Ramesh, A. Karthika, A. Karthikeyan, R. Sacithraa and K. Vanitha, \"Smart Shopping Trolley based on IoT with Mobile Application,\" 2023 International Conference on Sustainable Computing and Data Communication Systems (ICSCDS), Erode, India, 2023, pp. 1221-1225, doi: 10.1109/ICSCDS56580.2023.10104651. [13] Y. F. Tang, Y. X. Zhang, S. W. Hing and S. L. Kan, \"Follow-me Shopping Cart,\" 2019 IEEE 8th Global Conference on Consumer Electronics (GCCE), Osaka, Japan, 2019, pp. 259-262, doi: 10.1109/GCCE46687.2019.9015608. [14] R. J. C. Aquino, C. K. C. Beltran, J. W. A. Fajardo, J. L. A. Lopez, N. E. Sambajon and R. E. Tolentino, \"Image Processing Based Human Following Cart Using 360° Camera,\" 2020 International Conference on Electronics and Sustainable Communication Systems (ICESC), Coimbatore, India, 2020, pp. 375-380, doi: 10.1109/ICESC48915.2020.9155956. [15] X. Zhang et al., \"Special Library Cart with Automatic Following Function,\" 2019 3rd International Conference on Electronic Information Technology and Computer Engineering (EITCE), Xiamen, China, 2019, pp. 1299-1302, doi: 10.1109/EITCE47263.2019.9095078.

Copyright

Copyright © 2024 Ashutosh Marathe, Omkar Kanadkhedkar, Kunal Jadhav, Kshitij Gedam, Swadha Joshi. This is an open access article distributed under the Creative Commons Attribution License, which permits unrestricted use, distribution, and reproduction in any medium, provided the original work is properly cited.