Generating value from smart meter data with SMASH


Joshua Thumim,
CSE Project Manager
Bristol, England — (METERING.COM) — April 10, 2013 – A project to harness the large volumes of data that will be produced by smart meters and present it in a form that can be used by the industry and consumers has been completed by the U.K.-based Center for Sustainable Energy, in partnership with the University of Bristol.

The researchers set out to develop a new computational system that would allow the extraction of commercially valuable patterns from smart meter data. They came up with a prototype “big data” platform called “Smart Meter Analytics, Scaled by Hadoop” (SMASH).
SMASH is a highly scalable distributed file system and parallel computing platform which can store, process and retrieve data from very large datasets. It has been tested it with datasets up to 20 Tb in size – equivalent to smart meter data from about 13 million households per year.

In parallel, a data mining team from the University of Bristol applied new, experimental techniques to a sample of real smart electricity meter data to identify interesting subgroups of consumers with statistically different consumption patterns. Scottish and Southern Energy and Western Power Distribution were partners in this project, providing confirmation that the tools and techniques being developed had real business relevance for them. One key application they identified was that the tools would enable energy suppliers and district network operators to generate more accurate profiles of consumption, where before they were forced to generalize.

In addition a web-based user interface has been developed, making it easier for industry clients to use and interpret the results.

“The electricity industry is reliant on balancing electricity generation and demand, and thus being able to predict peak demand periods to avoid distribution network failures,” commented project manager Joshua Thumim. “With ageing infrastructure, ongoing investment in renewables and a need to shift demand so it is spread more evenly over the day, smart meter data could prove invaluable. The challenge for CSE was to extract, analyze and present this data in a way that can be done promptly and cost effectively.”

The project was part funded by the Technology Strategy Board through the ‘Harnessing Large and Diverse Sources of Data’ funding competition.