Cloud Resource Cleanup and Monitoring Dashboard

Abstract

Due to various features and economic benefits cloud platform has rapidly become famous and is widely used for giving services on over internet. Due to immediate increase in the number of users on cloud platform, data centers are also developed rapidly in terms of hardware resource, traffic volume, virtual resources, and storage. Thus, it is a making cloud platform or solution architectures very difficult handle all the resources on cloud like how many resources are running without using it, duration, by user, etc. To manage the unused resources which are running on the cloud platform which will cost for the organization, this system is going to help to cleanup unused resources and gives the notification to the users and helps organization to save the cost of those resources. The monitoring dashboard system will show all the resources of cloud platform which are running in respect to each service and can be easily monitor on dashboard and resources cost can also be shown on dashboard and as well as on chart system with respect to various services. This paper discusses about how unused cloud resources will get cleanup and monitoring the resources using dashboard for useful insights which will be helpful for decision making process in the organization.

Introduction

I. INTRODUCTION

Cloud data centers are rapidly growing and continue to grow. Due to this, the services which the cloud platform provides to the users are numerously expand day by day. The resources used by users are also increased more than expected. Maintaining few resources in an organization can be done but maintaining countless number of resources is very complex process.

The resources which are running but not in use will cost the company. This is major issue which is facing by each organization now a days. Paying the amount for unused resources is loss for the organization. To overcome this, the developed system will assist an organization to clean up the resources automatically based on certain conditions which is set by the organization. Based on that condition, the resources are get terminated automatically or restricted users to use the resources for further in which it will save a lot of amounts to the company.

The monitoring dashboard shows the resources which are running currently with respect to each service provided by the cloud provider. On this dashboard, admin can monitor all the resources which are being running, terminated, cost, etc.

The dashboard also contains charts for the resources of each service in which it will show the cost for the resources being charged. The resource cleanup and monitoring dashboard will help the organizations in various ways to save the time, the cost of the resources and monitor the resources of each service so that it will help the organization to have the information on all the services which are being used by the users in the organization.  Admin can also create a token for IAM user where users can have access to the shared one to use the resources depending upon the organization permits and amount of usage.

Amazon is an existing system in which all the resources have the basic functionalities where every service has various resources and can be monitored and managed separately. The unused resources will be running on the cloud will affect the organization. There is no single dashboard in which all the resources information is gathered, and it won’t show the resources which are used separately. Creation of IAM user is manual and will not have management on the same as till how much cost the user can use the resources.

As we conduct AWS cloud trainings here at an organization, Attendees who come and attend the training with us get a voucher with some credits and limited time period of validity so they can do the Labs for their trainings and do practice. Once the Vouchers gets over, the resources must be deleted manually by someone by now. Deleting those resources manually it is time and manpower consuming system. After the completion of the Resource Deletion Automation project the resources and services that are being created by the attendees who are getting the training with an organization will get auto tagged. After that whenever the voucher gets expired, we can know which resources are created by the user who is having that expired voucher. Then the system is using AWS Lambda to automatically delete those resources created by that user to stop the unwanted cloud charges.

The proposed system aims to is to develop a module for automatically tagging newly created resources and services and delete them from the organization website resources once the user gets expired. Auto-decommission of resources once the certain limit of voucher is reached or voucher get expired. Provide facility to monitor the resources consumption by the vouchers. Provide facility to disable voucher if consumption goes higher.

II.  LITERATURE REVIEW

Paying the amount for unused resources on cloud is one of the biggest challenges for organizations in which facing since the cloud platform come into existence. In addition, the dashboard which is given are not effective as they don’t show what exactly organization requires. In this circumstance, this system helps to clean up the unused resources by manually terminating or by automatically.

This system can send the notification of each resource cost at the end of the day through various communication such as email. The dashboard helps to show the number of resources currently being running and show the total cost of each service. [5]

The cloud providers, provides manual termination of resources which is created by user. If the user forgets or didn’t terminate the resources which are not required, then those resources will charge as per the duration of the usage in which it is cost for the company.

If the organization wants to restrict the user based on the user’s usage with respect to cost, this option is not available. The admin cannot see all the resources of all availability zones in a single place and also cannot see how many terminated. So, this system tracks all logs and shows the admin the running instances, terminated instances, have an option to terminate the instances, list the instances, total cost, chart wise, etc. The dashboards will have all the instances, services, cost, user’s details, services, charts, terminated resources. This is helpful for the organization to have useful insights to make great decisions for the future. Unlike cloud providers, this system will help the admin to create the token to access resources on cloud to the users depending upon the permission given.

The cloud providers, provides manual termination of resources which is created by user. If the user forgets or didn’t terminate the resources which are not required, then those resources will charge as per the duration of the usage in which it is cost for the company.

If the organization wants to restrict the user based on the user’s usage with respect to cost, this option is not available. The admin cannot see all the resources of all availability zones in a single place and cannot see how many terminated.

So, this system tracks all logs and shows the admin the running instances, terminated instances, have an option to terminate the instances, list the instances, total cost, chart wise, etc. The dashboards will have all the instances, services, cost, user’s details, services, charts, terminated resources. This is helpful for the organization to have useful insights to make great decisions for the future. Unlike cloud providers, this system will help the admin to create the token to access resources on cloud to the users depending upon the permission given.[9]

III. IMPLEMENTATION

The Cloud resource cleanup and monitoring dashboard mainly have three development phases.

1. Auto Tagging Resources: In phase 1 the user created resources are getting tagged automatically.
2. Fetch Resources of Expired Users: In phase 2 the system fetches the resources created by user who is having the voucher code which is getting expired on any given day.
3. Deletion Automation: In phase 3 the deletion of fetched resources takes place.

The system requirements are AWS Services for: Backend- CloudWatch, CloudFormation, IAM, and Lambda; Datastore- S3 bucket, DynamoDB and Cognito; Middleware- API Gateway and Cognito. It is feasible in terms of operation, technical, economic and financial.

1. Phase 1: In this the users will create the resources in our AWS account by taking the voucher from organization website resources for their labs. There the users will create the resources and then CloudTrail has a unique functionality that is collecting all the events which are done on the AWS account and then it will put all the logs in s3 in every 15 mins and there is one put event in s3 whenever the object put in s3. It will trigger the lambda function where we have the code for tagging and in that function firstly it will download the file from s3 and get the details of each service and its resource and its username and the company and then tag the services with two key-value pairs that are owner and company and meanwhile it will pass all the details collected for tagging in DynamoDB which will be future used in deletion script. 

In Figure 1, User Can purchase voucher and redeem it for creating IAM user in AWS. IAM Credentials will provided to user after redeem voucher. User will be able to login to AWS console with IAM Credentials. User can create resources on AWS after login. User can manually delete the resources created by them. Admin Can manually delete the resources created by the IAM user. Admin is the Root or IAM user with administrator access. Admin can delete the IAM user created by the voucher redemption.

2. Phase 2: The system fetches the resources created by the user who is having the voucher code which is getting expired on any given day.

3. Phase 3: The resources which are fetched is automatically deleted.

Conclusion

Cloud resource cleanup will help the organizations to reduce its costs on resources, set the limitations to users, terminate the resources which are unused, list the resources from various availability zones, show the number of instances running and terminated, create a token for new users to access to the resources, etc. Monitoring dashboard helps to display various running resources to services with cost to each resource and total cost for each service and total cost for all the services. It also shows, charts, day wise cost of resources, various services and under it various resources. Stores the logs and files in S3 for extraction. This saves a lot of time, effective utilization of resources, helps in decision making process, reduces cost, reliable, etc. The future enhancement are single sign on using Microsoft Office365, user wise deletion of resources, user wise resource tracking, isolated policies and responsive system.

References

[1] Wiebe de Roos, “clean up your unused cloud resources to reduce your cloud bill”, 19 November 2019. [2] Karthik Katti, Jayasimha SR, “Monitoring of cloud resources through dashboard”, 1 May 2020. [3] Sara Kardani Moghaddam, Rajkumar Buyya, Kotagiri Romamohananrao, “Performance-Aware Management of Cloud Resources: A Taxonomy and future directions”, August 2018. [4] Sourav Mukherjee, “Benefits of AWS in modern cloud”, March 2019. [5] Nelson Mimura Gonzalez, Tereza Cristina Melo de Brito Carvalho & Charles Christian Miers, “Cloud resource management: towards efficient execution of large-scale scientific applications and workflows on complex infrastructure”, 19 June 2017. [6] Suganya G, Simeen Sheikh, Premalatha,” Automated Resource Management on AWS Cloud Platform”, 2020. [7] Mahiba T, Jayashree, “A review on resource management in cloud computing”, May 2013. [8] Harvinder Singh, Anshu Basin, Parag Ravikant Kaven, Vinay Chavan, “cloud resource management: comparative analysis and research issues”’, 6, June,2020. [9] Mahantesh N. Birje, Chetan Bulla, “Cloud Monitoring System: Basics, Phases and challenges” September, 2019. [10] Christopher B. Hauser, Stefan Wesner, “Reviewing Cloud Monitoring: Towards Cloud Resource Profiling”.

Copyright

Copyright © 2022 Yerraballi Suresh Kumar Reddy, Dr. Ganesh D. This is an open access article distributed under the Creative Commons Attribution License, which permits unrestricted use, distribution, and reproduction in any medium, provided the original work is properly cited.