Indian Railway Tweet Monitoring System

Title of Paper

Short Description

1.Micro-Electronics and Telecommunication Engineering - Smart Government E-Services for Indian Railways Using Twitter

The Paper shows us a methodology of analyzing tweets which is done by first collecting the data, where all the tweets addressed to the Official Railway Ministry of India are collected. Then using the Micro Post Enrichment Algorithm which is used to clean the tweets and ensuring that the tweets are apt for the various classifiers which are to be applied on it. It is a multistep iterative process. It will take a raw tweet as an input and provide a syntactic and semantically enriched tweet. Next come the Feature Extraction  step that chooses the top N unigrams and bigrams from frequency distribution of the unigrams and bigrams present in the dataset  we also perform a comparison, between Naïve Bayes, SVM, decision tree where we reach at a conclusion that the best accuracy is achieved using Naïve Bayes.

2. Benefits of AWS in Modern Cloud

The paper shows us the benefits of using AWS, some of them being:

Data Protection – Data Protection is the major concern of most organizations and applications. The article states various ways in which we can use AWS for data protection and storage and prevent data leakage or any other damage to sensitive data. Some of them being Implementation of data hashing on the device and server, use of remote wipe APIs, etc.

Increased Productivity – Cloud has solved most of the time-consumption issues of software installation, maintenance of the product and backing up on a regular basis. Cloud computing provides more flexibility. The users can access files from anywhere at any time using web-enabled devices such as laptops, smartphones, notebooks etc

Backup options  – With automated backup helps in avoiding manual errors and also helps in time and money and ensures all application data is properly backed up.

It speeds up and minimizes the workload in my ways and hence by AWS CodeDeploy, it easier for to rapidly release new features, and avoid downtime during application deployment.

3. Using TF-IDF to Determine Word Relevance in Document Queries

This Paper gives us information about the results that are achieved when Term Frequency Inverse Document Frequency (TF-IDF) is applied to a document. It calculates values for every word by an inverse proportion of the frequency of the word in a particular document to the percentage of documents that the word appears in. This paper also provides proof of how simple algorithm can be used to categorize relevant words that can enhance query retrieval. The paper goes ahead to show us how TF-IDF returns documents that are relevant to a particular query. Once a user inputs a query for a particular topic, TF-IDF can find documents that contain relevant information on the query. So we can use it in our project to measure the importance of a word in data.

4. Understanding Regression Testing Techniques

The paper informs us about the types of regression testing and also tells us why it is important to include regression testing in order to determine all defects in our project.

In the maintenance phase of the software development life cycle we need to retest the software for the modifications it underwent and for this purpose we use various types of regression testing techniques. The paper shows us the complete structure of Regression Testing, also further classifying each one of them as explained by various authors, explaining Regression Test Selection and Test Case Prioritization in detail with Search Algorithms for Test Case Prioritization.

5. A framework for massive Twitter data extraction and analysis

This study illustrates the capabilities of twitter with two study cases in Spanish, one related to a high impact event and another one related to regular political activity on Twitter. It provides a framework designed mainly to ease the process of DATA EXTRACTION AND ANALYSIS which is followed a sentiment analysis which is divided into three subsystems:  Trainer, classifier, and tester and applies in on the respective two case studies. For the analysis it performs both sentiment analysis as well as quantitative analysis. We also understand how supervised classification is very  expensive and time consuming.

We could use it as a tool which could analyse the data in real time and use it to predict  trends or mass opinion.

6. An Improved Text Sentiment Classification Model Using TF-IDF and Next Word Negation

This paper is focused on testing the TF-IFD(Term Frequency-Inverse Document Frequency) algorithm, which is a technique to quantify words in a set of documents. A score is computed for each word to signify its importance in the document and corpus. In order to proceed with our project we have to be able to identify the tweets also based on their semantic analysis which in-turn can help classify the tweet.

So what we would be extracting from this paper is how we can use TF-IDF to our advantage by successfully identifying the importance of the post as quickly as possible and take the appropriate actions.

7. Response Time analysis for XAMPP Server based on Different Versions of Linux Operating System

Taking in consideration the technologies we would be using for the implementation of this project, XAMPP is a software distribution which provides the Apache web server, MySQL database, Php and Perl (as command-line executables and Apache modules) all in one package.

The activity performed in this paper is the calculation of the response time of XAMPP for various LINUX OS. For our project, we are aiming to deploy it on AWS and for that, XAMPP instance will be the perfect solution for an all-in-one software.

8. Security and Safety in Amazon EC2 Service – A Research on EC2 Service AMIs

After going through this paper, we learnt about functionalities of the AWS EC2 Service, including what and how the safety and security are kept in this system. We also learnt about how we could use 3 EC2 systems for our project by keeping 1 Master and 2 Slave machines.

This paper does not contain a research gap as this reference paper will be used solely to grasp the features of the service.

9. A Review Study of Apache Spark in Big Data Processing

This paper sheds light on how Apache Spark is a fast-growing new generation technology in the field of Big Data Analytics and due to its features such as lightning-fast cluster computing which is specifically designed for fast computation. It also compares between spark and hadoop, which has led us to believe that each has its own advantages, but since we would like our application to have a high processing speed we would go with Apache Spark.

10. Research Paper on AWS Cloud Infrastructure vs Traditional On-Premise

This research paper is solely based on the advantages of AWS Cloud such as, avoiding direct capital expense for variable capital expense, less complication to run and maintain data centres, an upper hand for global deployment, and many more.

So, for us to build a financially feasible web application we can use this service provided by Amazon for an efficient and flexible working. The author of this paper has also conducted surveys on the awareness of people about cloud computing, AWS and a few other factors. This survey helped us even more in determining the best technology for our project.

11. Learning Applications Exploring the Cost-benefit of AWS EC2 GPU Instances for Deep Learning Applications

This paper presents the implications, in terms of runtime and cost, of running two different deep learning problems on AWS GPU-based instances, and it proposes a methodology, based on the previous study cases, that analyses instances for deep learning algorithms by using the information provided by the Keras framework. Our experimental results indicate that, despite having a higher price per hour, the instances that contain the NVIDIA V100 GPUs (p3) are faster and usually less expensive to use than the instances that contain the NVIDIA K80 GPUs (p2) for the problems we analysed. Also, the results indicate that the performance of both applications did not scale well with the number of GPUs and that increasing the batch size to improve scalability may affect the final model accuracy. Finally, the proposed methodology provides accurate cost and estimated runtime for the tested applications on different AWS instances with a small cost. Hence, on observing the whole experiment it is clear that AWS EC2 instance would be the best choice for our project.

12. Understanding Users’ Satisfaction towards Public Transit System in India: A Case-Study of Mumbai

In this paper we got to know the quality of public transit system in resource constrained regions using user-generated contents. With growing urban population, it is getting difficult to manage travel demand in an effective way. Due to resource constraints, developing cities have limited infrastructure to monitor transport services. To improve the quality and patronage of public transit system, authorities often use manual travel surveys. But manual surveys often suffer from quality issues. To do this, we assumed that, if a tweet is relevant to public transport system and contains negative sentiment, then that tweet expresses user’s dissatisfaction towards the public transport service. From this paper we can learn the thing which we can implement is that we can got to know whether the tweet is a general tweet or a complaint tweet related to cleanliness or other services and take the action accordingly.

13. Comparison of Flutter with Other Development Platforms

From this paper we got to know this flutter is a useful toolkit enables easy ways of creating new applications. The basic results in this report indicates flutter has a slight edge as compared to native application development platforms but further more conclusive tests still needs to be carried out to come to a final conclusion.

Flutter has a great potential in future but further tests are needed to be performed as it is easy to use and getting popular day by day. And so, when we start to implement a mobile application we can use Flutter, as it grants the ability to design mobile applications in both iOS and Android with a single codebase, furthermore it also has the potential to make the app in all respects flexible and fluent.

14. Kafka: a Distributed Messaging System for Log Processing Jay Kreps LinkedIn Corp.

From this paper we learnt Kafka is used for processing huge volumes of log data streams.

Unlike typical messaging systems, a message stored in Kafka doesn’t have an explicit message id. Instead, each message is addressed by its logical offset in the log. This avoids the overhead of maintaining auxiliary, seek-intensive random-access index structures that map the message ids to the actual message locations.

As we are developing a real time application, we will be dealing with heavy streaming data, hence we can use Kafka to stream the tweets.

15. Comparing Database Management Systems: MySQL, PostgreSQL, SQLite

This paper introduces the two types of databases, i.e. relational and non-relational. We came to know about three of the databases, which were SQLite, MySQL, and PostgreSQL and also witnessed the advantages and disadvantages of both the database management systems. This also helped us in understanding which database would be much easier to connect and work with the AWS RDS and hence, we came to a conclusion that MySQL will be much more efficient than the other two, so, we will use MySQL for the EC2 Slave servers.

16. Building a replicated logging system with Apache Kafka

This paper instructs us about the online data integration task with the help of event pipelining platform used in Apache Kafka which allows to do many tasks like data streaming, stream storage and message queue.

So in our project we are using the data of twitter for our Indian Railways Tweet monitoring system and, the Apache Kafka provides the real time streaming of data to collect big data  to do real time analysis. So , this technology will help us to analyse the tweet which are tweeted by the passengers for any complaint in the service and with the help of Apache Kafka and the system can help the passenger to get proper and good facilities during travelling.

17. ZOOKEEPER: Wait-free coordination for Internet-Scale Systems

In this paper we learnt about the Zookeeper which provides a simple and high-performance Kernel for building more complex coordination primitive at the client.

It provides guarantee FIFO client ordering. In our project it works based on the timing of the Tweet tweeted by the passengers.  Tweet will be checked and implemented in the manner of FIFO which means First in First Out. So, every passenger will get preference with the timing of their tweet. It also uses simple pipelined architecture that allows us to have hundreds or thousands of requests outstanding while still achieving low latency which naturally enables the execution of operation from a single client in FIFO order.

18. A Collaborative Filtering Approach Based on Naïve Bayes Classifier

This paper shows us about the naïve Bayes classification Algorithm which uses the application of Bayes rule . Bayes rule states about the use of Venn diagram to classify the two types of things and then use of the part which we need in our technology. The Naïve Bayes Algorithm uses the same theory but it is used to differentiate the two types of messages which are coming to the system.

In our project we are using this algorithm to classify the tweets coming from the passengers of the Indian Railways for any negative complaint or positive review so that we can observe all the negative complaint and provide proper solution to that complaint.

19. Measuring user influence on Twitter: A survey Using Twitter API

In this paper we have seen that the Twitter API works to analyse all types of text such as short, long, use of bad words, misspelling and emotions are analysed. It also analyses the real time information from twitter or any social media platform. So, using this information and almost simultaneously observing other social networking applications, we were able to analyse the amount of impact tweets have throughout the whole social media.

So, for our project we came to a decision to use Twitter, above other application because as we observed the influence of a tweet in the world of internet, there are a number of tweets that broke the internet and was instantly the most viral topic around the whole world. Apart from this, the Indian Railways doesn’t have an official account on Instagram, they do have it on Facebook, however, if both the applications are to be compared we could process the posts from Twitter much fluently and take the necessary actions.

20. Web Scraping versus Twitter API: A Comparison for a Credibility Analysis

The study of this paper sheds light on the difference between data extraction models, there are various data extraction techniques such as web scraping, extraction using API and manual extraction. This authors of this paper focus on the differentiation between web scraping and drawing out information using API. As simple as it may sound, both of them have their own set of limitations, yet they both are very useful depending on the type of task they are chosen for.

As for the credibility, Twitter API was made to go through a real-time credibility model where we found out that through an API we could collect more data, web scraping is definitely a faster method but the amount of information it provides won’t be enough for our project. Hence, we came to a decisive statement to use the API of Twitter rather than the web scraping method.