Computer Vision Application in Automobile Error Detection

I. INTRODUCTION

Artificial intelligence can be applied universally across multiple domains of automotive engineering. Be it design, supply-chain, factory production or services that are provided after production. There are also some AI applications that involve "driver assistance" or "driver risk assessment" which don't pertain to vehicle but improve the driving experience in general. There are also services that help gauge the maintenance and the condition of the car well in advance to prevent any last minute mishaps. Problems related to the state and condition of the engine, the working of the battery, etc can be foreseen using AI. As the world move towards self-driven and connected cars AI role become much larger. AI auto market is expected to be 27 billion USD in hardware, software, and services sector.

Chatbots are programmes designed to extract or infer the meaning of user responses. A simple chatbot can perform based on simple one-line response (in the simplest case, a "Yes or No" response). A sophisticated, state-of-the-art chatbot can infer a complicated or elaborate response. Chatbots have been deployed by many companies on messaging apps like Whatsapp, Facebook, etc in order to meet the demand of 24*7 service that is required by customers today. Facebook, with its messenger function, has enabled many companies reach their prospective customers using AI to analyse user data and preferences.

For an industry, even a few negative customer experiences are sufficient to cause a significant drop in revenue as they face a reduction in number of potential customers. A report by Business Insider states that "60% of US customers have declined purchase of product due to poor customer service." Even though this was just US customers, it costs the company billions of dollars' worth potential revenue. Customer service chatbots focus on multi-step processes or input parameters to better aid customers more quickly, rather than asking multiple questions. The built-in natural learning processing (NLP) allows chatbots to understand and handle customer requests faster.

II. EXISTING DATA SCIENCE APPLICATIONS IN THE AUTOMOTIVE SECTOR

The field of automotive engineering is not new to the applications of data science. The idea of using of ML to perform autonomous, connected driving experience had crystallized by 2015. Analyzing a high volume of customer feedback has been practiced across multiple industries, including the automotive.

A. Data collection applications

At a high level of abstraction, the value chain in the automotive industry can broadly be described with the following subprocesses:

1. Development
2. Procurement
3. Logistics
4. Production
5. Marketing
6. Sales, after-sales, and retail
7. Connected consumer

Each of these areas already features a significant level of complexity, has their own data collection methods using sensors, cameras and feedback/reviews collecting platforms.

B. Information dissemination applications

In general, data is collected in a number of different ways. Tools such as GPS, proximity sensors, cameras have been installed in many vehicles. Capturing real-time data insights from these tools, data is then extracted and combined together to provide services. These services allow telematic-service-providing companies, car insurers, and car leasing agencies to predict movement of cars. Companies can use this information to simulate certain business models and thereby understand the demand-supply for their products and services. This gives companies' customers more customized and personalized experiences. A whole new ecosystem, enhancing user experience, is created around the usage of this data.

In general, there are three different types of big data -- Structured, Unstructured and Semi-structured. Let us understand each one of them.

Data that is highly organized in nature, i.e., has a definite number of attributes for all its entries is called structured data. It's well-defined nature eases the process of storage and retrieval. A RDBMS(Relational Database Management System), would be the most typical method for storing structured data. A simple example of structured data would be school records for a particular batch. They would contain the student roll number, the address, contact info, grades, etc in a list for all students of a given batch.

Contrary to the highly well-defined nature of structured data, unstructured data is the type of data that lacks a predefined model. Structured data is best exemplified by records, while unstructured data could be in form of texts, images, video, etc. This type of data is usually stored in a non-relational database such as Not only Structured Query Language (NoSQL). Therefore, the processes such as storage and retrieval are much more difficult with unstructured data. Although, the amount of structured data available for usage in comparison with unstructured data is very miniscule, there are more tools to process structured data compared to unstructured. Commonly used examples of unstructured data are emails and other communications. However, since we are referring to the automotive industry, here's another fun fact: Intel estimated that a car in its eight hours of operation/movement generates terabytes of data. This data, collected from sensors, telemetry, accelerometers and many other devices, can be analysed and one can come up with calculations or adjustments that would enhance the driving safety.

Semi-structured data is a hybrid of structured and unstructured data. This type of data only makes up about 5%-10% of all the data. Modern databases can store both structured and unstructured data together, making them semi-structured. A few examples of semi-structured data are the Extended Markup Language (XML) and open standard JavaScript Object Notation (JSON).

Reuse of legacy systems to store structured data is highly preferable due to its high efficiency and low cost. A structured form of data means clean, meaningful, reusable data that is easier to manage. It is easier to adapt to storage information in a structured form than making repetitive, expensive attempts at using unstructured data.

C. Algorithmic data analysis and decision support applications

The algorithmic data analysis can be broadly divided into four levels as shown in Figure 1. At the highest level, we have prescriptive analysis, which, in essence, is performed when one has a situation that requires good decision-making. A simplified way of thinking about prescriptive analysis is the question "What should I do?". One level below this, in terms of usability in AI, would be predictive analysis. This analysis can be inspired by the question "What will happen?". The lowest two levels of analysis in AI are diagnostic and descriptive analysis, which basically answers the questions "What happened?" "How it happened?" "Why it happened?" and so on.

III. A SPECIFIC EXAMPLE -- AI IN AUTOMOTIVE SERVICES

There are many mobile applications that are being used today for consumer services. One such category that we have selected is people using /driving cars. We have some car users in a situation where car stops in the middle of the road and they need an urgent help. Today they need to keep calling till the right help is available. This is required most by older and younger drivers. We have designed a mobile application that will be available to provide help with AI module. The SmartCarAssist solution provides an intelligent and timely help to anyone driving alone, silent hours and also provides contacts of closest service provider. This will be available as a mobile application where user can send a scan of dashboard and the AI based solution will be able to response to this in few minutes. This will also have Text enabled service for the user to be able to get useful guidance. The problems that are very critical these applications will provide contact of closes service provider based on the location of mobile

Maintenance Recommendations

By recording the data related to previous events that have happened to the vehicle, ML algorithms can be used to provide evolving recommendations to drivers about vehicle maintenance. The information about past occurrences of a certain event can be exploited in order to gather insight about the frequency of this event and predict its occurrence in advance. By gathering insight pertaining future breakdown, the driver can be warned ahead of time to have his vehicle repaired.

Reimagining service through SmartCarAssist

SmartCarAssist allows a customer to find a quick solution to customer problem by utilizing AI. Most BOT use some form or AI to understand customer query and respond accordingly. However due to set of problem which are unique to automotive industry a simple chatbots cannot facilitate customer problem on the go. Less than 7% Indian speaks English, while the number for commercial driver is way too less. India has more than 300million vehicle and more than 750M smartphone users. A vehicle brand with customer all over India will have massive challenge supporting such multi-lingual customer through chat. Having a multi-lingual support center will again be a massive cost for such organization. By introducing Media other than chat for support and using AI for identifying issue and providing solution, SmartCarAssist provide cost saving like never before, customer engagement and customer satisfaction to a global automotive.

A. Data source and data collection process

The data is picked from

1. Kaggle.com
2. Build/Enhance the data with Automobile engineers
3. Build/Enhance the data with Servicing team (vendor specific)
4. Images from Database
5. 1000's of samples - Videos to Frames from different angles
6. Labelling the data in the model
7. Mapping each entry with appropriate responses
8. Correlating each response with appropriate products
9. Transfer learning for Model Development

B. SmartCarAssist -- Information and Services

SmartCarAssist solution is an advanced car-assist AI mobile application for providing services to users while their car got breakdown in the middle of the journey. The state-of-the-art SmartCarAssist computer vision and chatbot AI algorithm will foster diagnosing the Dashboard errors to user by just taking the snap of the dashboard. It gives information about the errors and enables users with data of nearby service stations information who would real solve a particular error of the car. This feature enables customer's experiences during the break down of the car to provide right information and possible to solution to overcome the situations quickly and efficiently. Our Endpoint solutions will solve customer pain points and enables users to come out from the inconvenience situation of car break down. Our Smart APP, scans the car dashboard and provides quick information about the car diagnosis data. With this data, application further enables users below services.

• Silent Serviceability: Extended serviceability information over silent hors of the day and remote places.
• Generative Adversarial Mobile Network: Ability to identify hitherto unknown errors through constant training of detection algorithm using the self-learning capabilities and connected mobile network.
• Self-improvement: Organic ability to improve detections by itself as a combination of self-learning and real data-based recalibration of the model

1. Customer Services BOT for Automotive Industry

Chatbots have enhanced consumer experience in many industries. Here we discuss their roles in the automotive domain.

2. 24*7 Onsite Messenger

The industry is required to demonstrate convenient accessibility in order to provide best consumer experience. The automotive chatbots efficiently help the dealer to catch the ever-evolving understanding of the customers. Chatbots can answer this requirement of continuous availability round the clock. Customers have the liberty to interact with the representatives, along with receiving dedicated support from chatbots.

3. After-sales service

After-sales service comprises of providing a quality consumer experience post sale. Dealers could be involved in more sales and the service stations could have a large queue customers, if it is not organized correctly. In such a scenario, the automated chatbots can help organize schedules that allow customer service to be delivered in a convenient manner. When a customer schedules an appointment, they also need to know the estimated cost and delivery time. Although these tasks might look minimal, however, they create an enriching and valuable customer bond. Automotive industry, due to the nature of their product, is one that generates good revenue from customer maintenance services rather than just sales.

4. Service Plea

As stated earlier, chatbots can automate the process of communication with customer. It can interact with the customer, capture the required data, and set up the service request forward. The staff members can furthermore view the appointments and form the plan accordingly.

As per a survey by Capgemini, conducted on 12000 consumers along with 1000 executives from varied industries, it was seen that 50 percent of automotive companies make use of AI-enabled chat or chat assistants to engage the customers actively.

C. Algorithmic data analysis and decision support

The data analysis for this portfolio will require analysis of images from different angles and isolation and labelling of each image with the right issue. Then along with Domain experts and Service providers each issue is going to be mapped to an appropriate response. This response will be enhanced to include the products required to fix and service providers in the area best known for the mentioned service. This model will also record the new issues for which it will directly provide a global helpline that will offer verbal support and guidance to the vehicle owner. Such new issues and identified responses will be analysed and fed into the model from time to time by a small maintenance team. The team on ground could provide different enhancements -- like depending on the problem identified associating - 'Towing Service, Car booking facility specific to the area (pin code) where the issue is identified.

This product will be developed in iterations where in first 2 iterations, an improvisation on the range of issues getting identified will be improvised. And then as next steps - Linking to Best Product vendors, Best Service Providers and Towing service, Car booking options based on identified area of issue will be added.

D. Model and algorithm

Convolutional Neural Networks (CNN) can be used to perform operations like filtering, image classification, object detection, image segmentation, etc on an input image Apart from doing image processing and manipulation operations, recent machine learning techniques make it possible for engineers to augment the image data as well.

Fig 6 provides the overall flow of SmartCarAssist. The first step is to create a database of dashboard images labelled appropriately with their implied errors. This will create a reference for a CNN, which can then compare the customer's use-case. On finding an accurate match between the customer's issue in the reference database, which will be done using CNNs, the SmartCarAssist would provide a set of questions to the user to confirm the issue. The responses to these questions will be analyzed to provide a set a set of solutions to the user.

The mobile application will also identify the location of the vehicle and have a model that associates service providers with the pincode. This will then share the appropriate service providers in the area to the user.

E. Annotation Tools

1. SuperAnnotate

It is an AI-powered image and video annotation platform that can be used in conjunction with OpenCV. Allows users to create high-quality training datasets providing annotations for computer vision tasks.

• It eases design of project work and distribution of tasks among teams.
• Building large projects at scale.
• Using active learning to accurately annotate images.
• Annotations automation for predefined classes.
• Transfer learning to predict new classes.
• Use of QA automation to detect mislabelled annotations.
• Allows users to view analytics to track the annotation speed and quality.

2. LabelBox

Labelbox is a sophisticated platform with all the tools required for AI-enabled labelling of image data and text data, Has accessibility to a powerful API, along with Python SDK for extensibility.

It is highly recommended for commercial solutions which require creating and lebelling quality training data that comprise of image and texts. A standardized way for organizations to collaborate on the creation, manage, and review of data. Automation labelling to reduce costs, enhance the speed with QA.

3. Playment

Playment helps ML teams build high-quality training data for solutions in image, video, and sensor annotation. It also supports API integration to ML pipelines and GT Studio. Has the best-in-class annotations for Lidar and Radar. A standardized way to manage high-quality training data for computer vision tasks. Has a Ground-truth Studio to serve data labeling for creating diverse, high-quality ground truth datasets at scale streamline data pipelines to enable faster development of AI systems. Auto-scaling Workforce, provisions for customized use cases.

IV. DATA SCIENCE AND CONNECTING VEHICLE SECTOR WITH OTHER SERVICES

Data science is undoubtedly scaling mobility across the automobile and manufacturing industries today. It has played a vital role in reducing the production cost. The induction of data analytics into business models significantly increases the quality of analysis as it helps the analysts identify key areas wherein the performances can be optimized. By storing large amounts of data related to past performance, one can provide insights about the future performance. For instance, with the assistance of IoT, a vehicle can be connected to multiple devices which can gauge the working condition of the car. The technology allows different "connected networks" to communicate with vehicles, such as other cars on the road, mobile phones, and even city crossroads, which could be the first step in the direction of driverless cars. Data science plays a key role in every step of automotive product lifecycle:

A. Product development stage:

Data science in automotive begins with product development. Data science is used for tasks like analysing new model configurations and modelling component part reliability. Data science enables full system simulation as opposed to isolated unit testing, thereby increasing the reliability of the end-product.

B. Data science drives excellence in manufacturing:

One of the key responsibilities of automotive data scientists is to ensure that only high quality products are provided to the end-users. While engineers can test the quality of each vehicle, this has to be performed on an individual basis for each vehicle. Data scientists can analyse an entire population of parts, suppliers, and test data. A close inspection of the financial performance of suppliers - predicting their ability to deliver on time given past performance, in conjunction with econometrics, can provide sufficient information to gauge the overall condition of the supplier

C. Data science drives connected and autonomous vehicles:

Autonomous vehicles are highly dependent on data science for functioning correctly. Data science would be necessary in order to record the internal statistics of the vehicle, like fuel gauge, battery life, engine condition, etc. But that's not all, Data science would be very crucial in sensing the immediate external environment of the vehicle as well. It would not be sufficient to just detect the presence of an obstacle. The algorithm must detect what the obstacle is (person, car or animal) and must also come up with the most appropriate safety response. It is also imperative that the algorithm takes the safety of driver and passengers into consideration as well.

D. Data science drives sustainability initiatives:

Sustainability is a key factor for all automotive manufacturers. Governments set targets for fuel efficiency, but each auto company has its own goals. And each vehicle has a different fuel efficiency, so data science is necessary to optimize the fuel efficiency of a company's entire line of vehicles. For instance, if a company offers both a large gas-guzzling pickup truck and an electric car in its product line, automotive data scientists can perform an optimization to minimize the fuel consumption of the entire fleet while adhering to the company's global sales targets. Such effort into optimization is usually appreciated by the governments and therefore allows industries to claim "fuel efficiency credits" from the government. This allows an industry to generate more income, while showing their sustainability with respect to the environment. With our idea of training the algorithm, reaching out to service providers, having the issue diagnosed with the help of integrating with bot. Once the business improves and we have a series of customers, big data can play a role to boost the efficiency and start providing market intelligence.

Fleet management is perhaps one of the biggest beneficiaries of IoT and connected vehicles technology for Faster Improvement: - Light vehicles or heavy-duty are already packed with sensors and technology. IoT is therefore of utmost importance in maximizing the usage of the leading fleet management technologies, such as GPS and OBD (on-board diagnostics). By leveraging these technologies, companies can effectively extract information from their fleet about routes, maintenance requirements, and driving conditions in real-time.

Faster Decision-Making in Real Time: - A fleet management system that has been properly implemented IoT, enables fleet managers to use real-time resources and make well-informed decisions on the go. Quick decision-making, especially when dealing with changing customer agendas or unexpected events in a chain of events, is crucial to reduce potential "speed bumps" that can affect deliverables and ultimately the business' bottom line the inclination towards IoT, automation in automotive industry is growing fast and the car manufacturers are now realizing the importance of automation.

McKinsey Automation survey from 2018 states that 57% of 1000+ institutions have already began the implementation of fleet management, while another 18% are looking to initiate it in near future.

The adaption of automation in connected cars have already started showing benefits, to name a few like:

• Distinctive insights: Additional factors help gauge and predict driving performance accurately.
• Faster service: The service processing time reduced drastically
• Flexible operational time: Ability to operate 24/7 based on requirement
• Improved quality: Combines the conventional individual unit checking with the detailed analysis of supplier by data scientists to provide a very accurate insight.

V. APPENDIX