The Internet of Things (IoT) and Smart Metering

Every month, my husband receives a driver’s report from Chevrolet, providing him with “feedback” on his driving behavior – everything from the intensity of his braking to the number of lane changes without a blinker to the distance and speed he drove in his Silverado truck. The report also provides specific details on the upcoming truck maintenance requirements, including his tire pressure, temperature of transmission fluid and brake fluid, level of windshield wiper fluid, and so much more. And need I mention that Chevy is more knowledgeable about where he drives every day in his truck than I am!

In the last three years in the USA alone, Chevrolet sold more than 1.3M Silverado trucks with hundreds of sensors that power the driver’s report, and hundreds more that allow the truck to run. Imagine the volume of data that this one use case is generating and multiply it by the other industries – utilities, manufacturing, appliances, wearable devices, security and so many more – and it’s very clear that the Internet of Things (IOT) opportunity is real. But the challenge is just as important, and if there was ever a description of data overload, it’s IOT.

Data volume, velocity, and most importantly, differing data formats and incomplete data sets represent an enormous hurdle. The untapped value of sensor data float around in a Hadoop data lake or swim aimlessly in an Amazon S3 bucket. Data is being collected, but for many companies, that’s where the pipeline ends.

Vertica is uniquely designed to save this important data from drowning. Let’s look into this claim a little deeper. In an HPE Labs benchmark research report focused on a smart metering use case, data from 40 million meters was loaded into Vertica. The load step puts the raw data into a temporary “staging area” in Vertica to identify and repair “raw” data quality issues. Vertica can do this with the gap filling and interpolation (GFI) functionality in our time series analytics function. Just like with our in-database machine learning end-to-end functionality workflow, our advanced analytical functions like time series and geospatial and pattern matching (and many more!) all address data preparation and validation.

Now, we get to the analytics, and this is where we redefine the concept of “big data.” The HPE Labs benchmark data represents a “month in the life of a utility” in terms of data collection and analysis. The meter data is collected every 10 minutes for 40 million households. For a 31-day month, that means 178 billion readings. This is 4,464X as much data as a utility that only collects data once a month. But wait … any intelligent utility will also want to perform trend analytics on the monthly data over the course of months and years, and the benchmark study decided to focus on a decade of data that brought the data size to 22.8 trillion readings. Remember that utilities cannot bill their clients until they gather these readings on a daily basis from the thousands to millions of smart meters. Through these benchmarks and real-world use cases, Vertica proves that it can load and analyze smart meter faster and at scale than any of our competitors. The additional value for the utilities, once they account for smart metering tampering or issues with data communication, is to understand the daily, monthly, and annual energy usage and accurately predict demand to both optimize and protect the power grid.

It’s easy to see legacy data warehouses that utilities employ for traditional monthly data management cannot handle this exponential explosion of data. Vertica’s columnar architecture combined with MPP and compression are the foundation of why Vertica is one of the few (I think ONLY!) analytics platforms that can handle the volume, the velocity, Know2Growand especially the variety of data for IOT use cases like this. Then, consider the advanced analytics (time series, pattern matching, geospatial, projections, etc.) and in-database machine learning. If a company is dealing with this volume of data, moving the data to specialized platforms like a Time Series database or a Machine Learning platform is simply not feasible. And let’s not forget valuable and helpful built-in database tools like Database Designer (DBD) that optimizes the data layout to maximize query performance. The industry is buzzing with Data Warehousing as a Service and Autonomous Databases but Vertica is more than buzz. It’s real.

The time is now for us to raise this topic with every customer in every industry. There isn’t a single industry or company that doesn’t have the potential for some form of IOT analytics. It’s our job to protect the data from drowning in the overflowing data lakes and ensure that our customers don’t find themselves in a history book rather than on the Fortune 1000 list because they didn’t derive value from all the sensor data that they had to bring their IOT analytical use cases to market with the peace of mind of performance at massive scale.