Smart integration: The key to unlock demand response power from smart metering investments


By Ali Vojdani, Ph.D.

Power utilities today are confronting unprecedented change and opportunity from smart metering programmes. The utility industry’s long established linear approach to change, centred on static design and rigid solutions, is increasingly leading to technological obsolescence and is failing to provide cost effective solutions.

Fast evolving technological innovations, the increasing cost of supply side options, concerns about global climate change and the demand for high quality power in the digital age are weighing heavily on the industry. The choice to migrate away from the traditional approach and confront the dynamic future with an approach that incorporates flexibility, improvisation and agility is critical in order for utilities to withstand the inherent uncertainty of a rapidly evolving global economic environment.

At this critical juncture, utilities in the USA and around the world are taking steps to meet the challenges of this new era by investing heavily to upgrade their antiquated delivery, pricing and service networks and embarking on massive smart metering and smart grid projects. Continuous improvements in technology accompanied by rapidly falling costs make smart grid, smart metering and smart pricing investments attractive and cost justified. Moreover, regulators and policymakers at both the national and regional levels have become receptive since they see these investments as a necessary prerequisite to improve energy efficiency and enable demand response programmes that manage peak demand while reducing overall costs of service delivery.

While the new attention focused on smart grid/metering projects is a welcome development with significant promise, the industry is facing considerable challenges that, if not heeded, may result in potentially massive project cost overruns and possible new stranded costs in under performing or obsolete technology infrastructure.

The most daunting challenge facing utilities during their rapid migration to a smart grid enabled business environment is that they are entering essentially uncharted territory with a number of serious pitfalls – and no one can predict in a traditional planning approach how this fast moving business environment will evolve.

The transition to a smart grid/smart metering enabled business environment will require flexible design, agility and improvisation necessitated by frequent and dramatic changes to business processes that are ill-suited for traditional utility style projects. In this new environment, people, systems, solutions, and business processes must be dynamic and flexible, able to bend, shrink or stretch in response to changes in technology, customer needs, prices, standards, regulations, policies or other requirements.

Managing smart grid and smart metering projects is difficult due to the sheer size and complexity of the number of data points. For example, the demand response programmes often contemplated as a follow on from a smart metering project typically require:

  • Secure and reliable communication and control among a potentially large number of participants
  • Ability for participants to register and interact with one another in an error free environment
  • Ability for various participants to bid, update and interact based on prices and the response of other participants
  • Facility to schedule and implement the transactions that parties have agreed to do
  • Protocols for measurement and verification of the above, and
  • Automated processes for settlement, billing, collection, bookkeeping and dispute resolution.

Likewise, the dynamics of smart metering programmes also require a flexible approach to business process management:

  • Offering different services and tariffs that vary by time-of-use and potentially by type of application
  • Metering and meter data management services
  • New systems for billing and settlement, and
  • New customer service applications capable of supporting the new metering, pricing and billing schemes.

The traditional approach to design for such new systems would be to specify a blueprint that includes a standard architecture for a group of applications that would supposedly provide the needed functionality. Historically, utilities would typically issue RFPs to procure the necessary applications or upgrade existing ones to provide incremental functionality. The vendors or the IT department would design and build static “data bridges” to connect these applications. The main shortcoming of this approach, as already pointed out, is that it represents a static view of a rapidly changing future, namely that systems are built in deterministic ways, satisfying the requirements of the next phase. Interfaces are built to connect these static systems, but nowhere in the specifications of the applications or interfaces is there any explicit requirement for flexibility or adaptability to change1.

How does one build a flexible infrastructure that can be more responsive to a changing business environment and consumer needs? The basic recipe, a “smart business integration” approach, includes the following steps:

  • Define the basic products and services that consumers need at a conceptual level
  • Identify the business processes that can support and deliver those products and services
  • Break down the business processes into a set of services at a higher level of abstractions than is done today
  • Provide the necessary infrastructure for integrating these services in a flexible way according to best practices in service oriented architecture (SOA)
  • Build/buy applications for delivering the desired services in a dynamic and flexible way2
  • Specify and build flexible interfaces to bridge data transfer among different applications, making sure the interfaces are not tightly coupled with the applications3
  • Manage business processes end-to-end with a business process management (BPM) software approach that coordinates among different applications4
  • Simplify the connections between applications and minimise coupling through enterprise application integration (EAI) architecture5, and finally
  • Use smart technologies such as complex event processing (CEP) to monitor and analyse the events as they occur and use the insight obtained to automatically modify business processes dynamically6.
Approach smart integration

Conceptual illustration of the smart integration approach

Demand response (DR) programmes, if appropriately designed and integrated, have the potential to act as effective shock absorbers in electricity markets, dampening price spikes during peak demand periods and reducing price volatility while enhancing grid reliability. When effectively integrated in organised wholesale markets, DR programmes can produce impressive benefits for grid operators, utilities, and consumers alike.

Given such impressive results, why is wide-scale implementation of DR programmes not occurring across the industry? One reason is that DR, like other demand side options including energy efficiency programmes, is still in its infancy relative to the industry’s traditionally strong supply side resources. A second and more important reason is that DR programmes, by their nature, tend to be diffused and distributed among a potentially large number of participating customers. Aggregating and effectively bringing these distributed resources to wholesale markets to compete head on with traditional supply side options is complex.

For DR programmes to work effectively, they must be integrated with the existing computer applications of organised markets, utilities, and service providers, while complying with ever changing market rules and protocols. These applications are among the most complicated and challenging IT systems because of the variety of sub-systems that must be managed and seamlessly operated in unison to perform a variety of complicated tasks and operate a number of interdependent markets in real time.

Anyone with a rudimentary understanding of the operations of an organised electricity market can attest to these complications. Making matters worse, DR programmes must be integrated on top of myriad other computer applications, a maze of market rules and a barrage of changes in rules and procedures coming from both within or outside markets. Since many of the hardware and software systems are essentially hard-wired to perform specific functions, adding DR to an inflexible system is more complicated than it may sound.

Moreover, each organised wholesale market has its own set of rules and protocols and its own members and participants. Naturally, each market uses customised systems to handle its unique requirements, so adding a DR capability to introduce price elasticity on top of existing systems is not trivial.

Experience to date clearly suggests that managers in charge of DR programmes must add flexibility to the long list of requirements for IT systems and supporting applications. In fact, it may be wise to list flexibility ahead of other desirable attributes such as performance, usability, scalability, reliability, and so on.

In this context, managers responsible for procuring IT infrastructure to support DR applications should consider flexibility as a predictor of future change order costs in answering questions including how fast and at what cost they can change existing applications and business processes.

Four key suggestions may be taken away from the preceding discussion. To unlock the power of demand response programmes and the ensuing full potential of smart metering investments, utility managers must consider not only the end systems they choose for deployment, but also a new approach to smart integration that gives significant weight to the flexibility of the IT infrastructure and business processes used to take advantage of smart metering and smart grid investments.

For demand response to work effectively in an organised electricity market it must be fully integrated into the inner working of the market mechanism and it must be flexibly integrated into the system dynamics of utilities participating in the market. Enabled by smart metering and integrated markets, demand response must be treated as a resource on par with supply side resources for energy, capacity and ancillary services. Only in this way can the full potential of DR – adding sorely needed price elasticity into an otherwise inelastic demand – be realised.

Achieving such an objective in the highly complex and constantly evolving computer applications environment of grid operators and utilities is challenging. For DR to work in such a demanding environment, it must have built-in integration flexibility to communicate and support a variety of complicated computer applications.

1 Even if flexibility is mentioned, the industry does not have any convention or methodology for measuring or testing for flexibility

2 Business rules should not be hard coded. This can be accomplished by using Business process management engines. For example, see “California Demand Response Business Network” by by Ali Vojdani, Scott Neumann, and Gaymond Yee, DistribuTech 2006.

3 Interfaces have to be a lot smarter than the dumb bridges of the past. They will be more expensive to build, but will be independent of the applications if they are to be replaced.

4 See “Applying Workflow Technologies to Integrate Utility Business Processes” by Ali Vojdani, DistribuTech 2005.

5 See “Tools for real-time business integration and collaboration”, invited paper, IEEE PES Vol. 18, No. 2, Special issue on Tools for Managing Restructured Energy Systems, pp 555-562, May 2003.

6 The Power of Events: An Introduction to Complex Event Processing in Distributed Enterprise Systems, David Luckham, Addison-Wesley, 2002.