{"id":14228,"date":"2020-05-08T13:03:47","date_gmt":"2020-05-08T07:33:47","guid":{"rendered":"https:\/\/www.mygreatlearning.com\/blog\/automation-of-energy-meter-reading-monitoring-and-billing-using-aws\/"},"modified":"2022-11-25T14:29:57","modified_gmt":"2022-11-25T08:59:57","slug":"automation-of-energy-meter-reading-monitoring-and-billing-using-aws","status":"publish","type":"post","link":"https:\/\/www.mygreatlearning.com\/blog\/automation-of-energy-meter-reading-monitoring-and-billing-using-aws\/","title":{"rendered":"Automation of Energy Meter Reading, Monitoring and Billing using AWS"},"content":{"rendered":"\n

PROBLEM STATEMENT<\/strong><\/h3>\n\n\n\n

State Utility and energy supply companies currently use energy meters to measure the electricity consumption and chargeback to the consumers. <\/p>\n\n\n\n

These meters display a cumulative electricity usage in KWH which is recorded manually and reported by the technicians to the power company on a monthly basis. Overall meter reading process requires tremendous manual effort, prone to human errors and is also open to manipulation.<\/p>\n\n\n\n

Many times these energy meters become faulty and cause voltage\/frequency fluctuations. This often goes undetected as there is no real-time monitoring mechanism.<\/p>\n\n\n\n

These inefficiencies often result in losses, power wastage, and increased costs for the utility companies
<\/p>\n\n\n\n

PROPOSED SOLUTION:<\/strong><\/h3>\n\n\n\n

The capstone project aims to leverage \u201csmart meter\u201d technology and AWS platform to achieve the following objectives:<\/p>\n\n\n\n

    \n
  1. Completely automate the meter reading process via MQTT protocol over network<\/li>\n\n\n\n
  2. Develop a monitoring framework to detect faults and power fluctuations to the power company<\/li>\n\n\n\n
  3. Provide accurate and real-time power usage to the consumer<\/li>\n\n\n\n
  4. Optimise the monthly billing cycles<\/li>\n\n\n\n
  5. Create the utilisation matrix for Power Company to prevent meltdowns and predict regional demands for intelligent scale-up of operations<\/li>\n<\/ol>\n\n\n\n

    <\/p>\n\n\n\n

    Smart meters<\/strong><\/h3>\n\n\n\n

    The project leverages smart meter technology, which enables the meters to transmit their power consumption and health metrics (voltage, current and frequency) directly to the IoT services hosted in AWS cloud over the WAN network. Smart meters need to be registered to the IoT core services, only the registered meters can communicate with the IoT core service.
    <\/p>\n\n\n\n

    IoT gateway is a device which resides in the close proximity of the smart meters and collects the data from the smart meter over the local network. The gateway then converts the collected meter data into JSON format and transmits it over the internet to AWS IoT core service using MQTT protocol.
    <\/p>\n\n\n\n

    In the project context, a customised python program has been developed to simulate smart meter and IoT gateway devices. The program can be installed on a laptop and each installation is given a unique meter serial number. This serial number needs to be first registered to the IoT core for it to communicate with the service. 
    <\/p>\n\n\n\n

    IoT Core<\/strong><\/h3>\n\n\n\n

    IoT core service receives the streaming meter data and validates if the sender is a registered device. If the data is from a valid\/registered device, it is passed to the Kinesis delivery stream. <\/p>\n\n\n\n

    Kinesis FireHose Delivery Stream<\/strong><\/h3>\n\n\n\n

    The Kinesis delivery stream transforms the data from JSON to CSV format and dumps it into the S3<\/strong> bucket in the form of CSV flat files. As the streaming data is written into S3, an object create event triggers lambda function which reads the CSV flat file and pushes every record into Dynamodb. <\/p>\n\n\n\n

    S3<\/strong><\/h3>\n\n\n\n

    S3 bucket is the data lake for all the raw and historical meter data. Keeping in mind the daily data volume, data is partitioned into specific dated folders within the S3 bucket, this helps to increase the performance while querying \/searching the data.<\/p>\n\n\n\n

    Athena<\/strong><\/h3>\n\n\n\n

    Database schema and tables have been defined in the AWS Athena whose columns map to the comma-separated field in the flat files. The tables have been partitioned by the \u201ccreated date\u201d, this helps to prune the data while querying and reduces costs. This mapping directly allows to select, filter, aggregate data from the flat files using SQL queries. Once the data is available in SQL format, it can be utilised for reporting and further processing.<\/p>\n\n\n\n

    Redash<\/strong><\/h3>\n\n\n\n

    Redash has been chosen as the reporting and analytics solution. It has a simple easy to use interface, out of box charting and reporting features, and supports all AWS data sources (Athena, RDS, Redshift, Dynamo). It is available as an AMI instance in AWS marketplace. <\/p>\n\n\n\n

    For this project, Redash analytics solution is hosted on an EC2 instance located in the public subnet and connects to Athena data source for fetching data for analytics and reporting.<\/p>\n\n\n\n

    There are two categories of reporting dashboards:<\/p>\n\n\n\n

    1) Consumer dashboards<\/strong>: These are accessed by the consumers for the current and historical power consumption for the last 6 months.<\/p>\n\n\n\n

    2) Monitoring dashboards: <\/strong>These are accessed by the utility companies, these are used to monitor region-wise power consumption, and detect faults and power outages <\/p>\n\n\n\n

    Cloudwatch Event<\/strong><\/h3>\n\n\n\n

    On 1st<\/sup> of every month at 8 AM a cloudwatch event triggers a lambda function. This lambda function aggregates raw consumption data for all meters for the past month from the Dynamodb table, converts them into monthly readings and pushes it into the RDS MYSQL database.  This aggregated monthly consumption is further used to calculate the monthly bills to the consumers.<\/p>\n\n\n\n

    RDS MYSQL<\/strong><\/h3>\n\n\n\n

    RDS MYSQL database has been chosen to host the consumer data, meter master data and aggregated monthly consumption data and consumer billing information. The database is hosted in the private subnet and only accessible within the VPC via 3306 port.<\/p>\n\n\n\n

    Business Challenges & Technical Challenges<\/strong><\/h2>\n\n\n\n

    Technical Challenges:<\/strong><\/h3>\n\n\n\n
      \n
    1. We faced a challenge to simulate smart meter behaviour, the other challenge was to ensure a secure communication channel between smart meters and IoT service.<\/li>\n<\/ol>\n\n\n\n

      Solution<\/span>:<\/strong><\/h4>\n\n\n\n

      We developed a smart meter simulator program in python, which can generate readings at regular intervals. For secured communication, we generated certificates from the AWS IoT service and imported them into smart meters. Only the meters with certificates could now communicate with IoT service.<\/p>\n\n\n\n

      2. Dynamodb does not support standard SQL, hence selecting and aggregating data from Dynamodb was a challenge. Also, the aggregated data had to be put into an RDS database.<\/p>\n\n\n\n

      Solution<\/span>:<\/strong><\/h4>\n\n\n\n

      This was achieved using the lambda function in python environment. Boto3 libraries were used to access data from Dynamodb. <\/p>\n\n\n\n

      RDS database engine was postgres initially, but there is no boto3 inbuilt library available for postgres database thus making it difficult to code in python. <\/p>\n\n\n\n

      Postgres database was eventually replaced with MYSQL database.
      <\/p>\n\n\n\n

      Learnings<\/strong><\/h2>\n\n\n\n

      Team has developed deep functional knowledge on the functioning of the energy meters, the recording of the meter readings and the key health check (voltage, current, frequency) metrics used for monitoring. <\/p>\n\n\n\n

      Team also adopted and learnt project planning techniques such as scope management, schedule estimation, cost estimation.<\/p>\n\n\n\n

      \n
      \n
      \n
      <\/div>\n Academy Pro<\/span>\n <\/div>\n

      \n AWS Cloud Course<\/a>\n <\/p>\n

      Learn AWS Cloud fundamentals and gain hands-on experience in deploying, managing, and scaling applications on AWS.<\/p>\n

      \n
      \n
      <\/div>\n 8.58 Hrs<\/span>\n <\/div>\n
      \n
      <\/div>\n 1 Project<\/span>\n <\/div>\n <\/div>\n \n Enrol Now for AWS Cloud Course\n
      <\/div>\n <\/a>\n <\/div>\n <\/div>\n\n\n\n

      Following were the learnings for AWS perspective:- <\/p>\n\n\n\n

      IoT core service<\/strong><\/p>\n\n\n\n