Data changes in an evolving market


A report by Datagenic and The Energy Partnership

Today’s data challenges are not necessarily new but are certainly more complex, and this level of complexity is increasing. The compliance, legacy and IT integration challenges faced by data managers are constantly evolving to address the evolutionary characteristics of the underling energy market as it becomes more of a financial market, while new energy directives and policies at both a European and individual EU member state level are being developed to meet competition, supply security and climate change demands.

These varied challenges provide energy companies with opportunities to address their risk-reward scenarios, particularly with respect to data management. With data becoming ever more critical to efficient and effective business management, companies have to adopt new policies and strategies, and revise existing procedures, to ensure that the risks faced through these challenges are minimised and the rewards maximised. The successful companies will be those that are able to realise the opportunities presented by the various business challenges.

Effective data management relies on data quality, accessibility and availability. Companies need to determine the base thresholds for these factors, which will typically be influenced by the demands of customers and may also be influenced by the systems and processes adopted by competitors.

In managing data, companies can either adopt a centralised company-wide or individual business unit approach. This will be influenced by the uniqueness of data to some business units and how data flows between these units within an organisation. How this data is managed will impact costs and therefore profitability. Companies can also use outsourcing or managed data services for both data acquisition and management, which can provide both management efficiencies and cost savings.

This report takes an overview of the challenges, opportunities and management issues faced by data management.

DataGenic commissioned The Energy Partnership to conduct research into corporate perceptions of, and developments in, energy data management. The brief was to ascertain current data management policy, and perceptions towards outsourcing.

Data management is strongly driven by IT, with 41% of respondents saying this will be a major challenge over the next three years, with only 18% citing businesses responsiveness to market changes as a perceived challenge.

71% of respondents said they had a policy in place that defines standards for the management and governance of data used in critical processes, with most utilising centralised IT infrastructure management in preference to using a centralised data acquisition team.

Two-thirds of respondents apply a portfolio of standard metrics or tests to ensure data quality, with the level of testing reflecting the proprietary nature of the data. Respondents not applying tests explained that such metrics/tests were only omitted for nonproprietary data.

Over half the respondents make data available on an enterprise basis, with those not adopting such an approach saying they had no plans to provide central data stores.

Just over half of all respondents have considered outsourcing non-proprietary data management tasks to a third party, with 43% of these saying an outsource project was either ongoing or under consideration.

Eight in ten respondents said they would be comfortable outsourcing non-proprietary data.

Two-thirds of respondents believe energy data requires a specially designed platform for the management of Energy Data, with an in house solution being favoured due to its business importance and complexity, but with a third saying there is a need to purchase an externally supplied system specifically built to manage that data.

71% of respondents said business units maintained their own staff for data management requirements, with all other respondents using a central IT resource.

Three quarters of respondents said their business units cross-charge for access to data stores, with 77% of these using commercial rates and remainder cross charging at cost.

This section identifies perceived data management challenges over the next three years, how changes will be implemented to address these challenges and whether companies currently have policies in place that define standards for data management and governance.

Respondents were asked what they perceived to be the greatest challenges facing their business over the next three years that will impact their data management systems policy. Respondents were asked to select from compliance (i.e. Basel II, SOX), legacy systems (integration, updating, sunsetting), European directives, IT (SOA, integration), business responsiveness to market changes, and business process re-engineering. As Figure 1 shows, legacy systems and IT dominate business concerns.

The findings that data management is strongly driven by IT is not unexpected, but it is surprising that so few respondents cited businesses responsiveness to market changes as a perceived challenge.

Of potential concern are the suggestions that respondents believe their companies do not face a major challenge from EU directives over the next three years, even though the European Commission is increasingly exercising more central control over energy markets. And with a new Commission taking office in November 2009, the probability for more EU energy directives, and hence new market challenges, is high.

There is a risk that the data market is too blinkered toward IT and legacy system integration challenges and as a consequence is under estimating the potential risk of future EU directives inducing market changes, that in turn may well present new data management challenges.

When asked how the data management changes to meet the challenges would be implemented, over half the respondents (59%, Figure 2) said changes would be implemented company-wide, while a further 29% of respondents said a combined company-wide and individual business unit approach would be adopted.

There is a surprising lack of individual business unit responsibility, even though business units tend to have different data needs and face different risks and challenges. The optimum approach is a combined implementation whereby individual business units address their own systems but the actual final implementation is conducted as a company-wide initiative.

With energy and other management data becoming increasingly valuable to critical business processes in the competitive energy market, respondents were asked whether their company had a policy in place that defines standards for the management and governance of data used in such critical processes (Figure 3).

As expected, the vast majority of companies (71%) have such a policy in operation, with most utilising centralised IT infrastructure management in preference to using a centralised data acquisition team.

But there are still just under a third (29%) of companies that currently have no policies operating for data management and governance standards. There is a suggestion that companies without policies for data management and governance standards do not believe they have any (or have insufficient) critical business processes that require such standards.

It could be that some of these companies are underestimating the nature of some business processes, and as a consequence may be exposed to unnecessary data risks.

Data changes1

As the volume of energy data, and sources of this data increases, leveraging value from this data becomes a business challenge. Companies have to ensure that data being used in management processes is accurate, complete, timely and reliable. There are a number of tests that can be applied to ensure the integrity of data and companies need to apply their own tests to ensure data is fit for purpose.

Companies are typically exposed to two types of data: proprietary data and nonproprietary data. Proprietary data encompasses that data relating to assets that are particular to the company, for example metering data, while non-proprietary data relates to data generally in the public domain, such as oil stock fundamentals. Companies will typically adopt a more rigorous testing of data that is proprietary, although this does not mean that nonproprietary data should not be tested for its quality.

Respondents were asked to list the tests or metrics applied to data before it is used, and were given the choice of timeliness, completeness, z-score, missing values and repeated values tests. An insufficient number of respondents differentiated between the test/metric choices offered to make the type of tests used statistically relevant, preferring instead to take a collective view of the various data tests offered. Of all respondents, 65% said they apply a portfolio of standard metrics or tests to data, with a variety of metrics being used. These respondents said the detail of the testing employed is dependent on the proprietary nature of the data, with the range of metrics and tests applied being dependent on the business sensitivity of the data. Those respondents not applying metrics/tests explained that such metrics/tests were only omitted for non-proprietary data.

Respondents were then asked if data quality standards were managed and applied actively to ensure quality levels meet the needs of the consumers (Figure 5). Two-thirds (65%) of respondents said data quality standards are managed and actively applied, with 64% saying this is provided by a centralised middle office and 36% provided through individual units.

Of those that do not apply these quality standards, 67% said it is not a concern. Interestingly, those respondents saying it is a concern were unsure how far up the corporate ladder this concern was shared, which only served to exacerbate the respondent’s personal concern.

Whether metrics and tests are applied to data depends on whether the data is perceived to be proprietary or not, and the extent of the metrics/tests applied is similarly dependent on how proprietary the data is. Clearly this is a subjective process within an organisation, which could see the same data in different companies treated differently. As such, there appears to be no standardisation in how energy data is treated within the market which in turn may be cause for further concern.

Data changes2

Managing data is a core business function and like other functions has to be conducted both efficiently and economically, with these management imperatives reinforced by the increasing diversity, complexity and inter-relation of energy data across both products and processes.

A number of data management practices can be used, but of most relevance to the energy sector is Enterprise Data Management, which is defined as the set of processes by which all the data in the enterprise (i.e. company) is managed from source to storage to enduse and end of life cycle.

The broad data management processes comprise data acquisition, loading the data onto a system and then normalising the data, matching and cross-referencing of data where required and finally checking and correcting data to ensure it is clean and useable. An efficient and effective data management system should benefit a company by: reducing the costs of data acquisition and management, and thereby increasing the company’s business margins; assisting in reducing business risks as the correct data will be available for management purposes; and improving corporate flexibility in either addressing new opportunities or in meeting new regulations as the requisite data will be readily available.

When asked whether data is available on an enterprise basis, i.e. a centralised store of common commoditised data, over half (59%) the respondents surveyed (Figure 6) said data is available on an enterprise basis. Those respondents that do not make data available on an enterprise basis said they had no plans to provide such data stores.

When those respondents that make data available on an enterprise basis were asked how this data would be managed (Figure 7), only 14% said they would outsource a database management system. The majority of respondents (59%) said they would build their own data system, while just under a third (29%) said they would buy in the necessary system.

It is evident that outsourcing proprietary data is not a popular approach within the energy market, and it should be noted that those respondents who said they are receptive to the principle of outsourcing explained this would only be used for non-proprietary data.

There is a sense that data management is being increasingly seen as an area of competitive advantage as well as a mission critical function with companies seeking to exercise greater internal control over this data. Companies are more likely to build their own data management systems and then buy-in their required data. Yet as this approach is likely to be a more costly process than outsourcing it, suggests that companies are placing the importance of data control over cost.

Aside from the data acquisition process there are similar cost considerations with the internal management of data, and in particular with respect to whether individual business units maintain their own IT staff for managing their data needs.

When asked, nearly threequarters (71%) of respondents said business units maintained their own staff for data management requirements (Figure 8), with all other respondents using a central IT resource to manage their data requirements.

If business units have unique data requirements then maintaining their own IT resources to manage data may be in the best interests of business unit efficiencies. However, if there is some commonality of data between business units then there could be a duplication of corporate IT resources.

When those respondents that had allocated IT resources by individual business units were asked whether this was perceived as being value for money (Figure 9), over half (59%) the respondents said the dedicated IT management of business unit data was a cost effective process with less than one in ten disagreeing. But a third (33%) of respondents conceded they did not know whether this was cost effective since they had not measured this value.

Finally on the issue of costs, respondents were asked whether business units crosscharged for access to their data stores (Figure 10).

Just over three quarters of respondents (76%) said their business units did crosscharge for access to data stores, of which 77% said commercial rates were used for cross-charging, while the remainder cross-charge for data access at cost.

Those respondents who said business units did not crosschange for data access maintained there is no conflict between business units.

While data integrity and the efficient management of this data is a key business requirement there is a sense that some companies could make data management a more economical process. Although respondents were not asked the contribution of IT costs to overall business unit costs, there is a concern that a reasonably large proportion of respondents have not measured the value (or cost) of using dedicated IT management of data for business units. Indeed it is conceivable that some of those respondents that dedicate IT data resources to individual business units may not have conducted a recent audit of individual business unit data management costs.

Data changes3

Outsourcing has tended to have a mixed reception in the energy sector. The perceived value of outsourced services lies in the efficiencies provided by removing, or reducing, the need for companies to invest in internal business functions and instead utilise the expertise of an outsourcing service.

Outsourcing involves the transfer of the management and/or day-to-day execution of an entire business function to an external service provider. The client organisation and the supplier enter into a contractual agreement that defines the transferred services. Under the agreement the supplier acquires the means of production in the form of a transfer of people, assets and other resources from the client. The client agrees to procure the services from the supplier for the term of the contract.

One of the main business segments to use outsourced services is IT, and given the strong involvement of IT in the data management function, there should be strong potential for outsourced data management services. When respondents were asked whether they had considered outsourcing nonproprietary data management tasks to a third party (Figure 11), they were evenly divided on the question of data system outsourcing, with 53% saying they had given this consideration. Of this proportion, 45% said the outsourcing project was either in consideration or ongoing. Among all these respondents it is the IT department that has driven the outsourcing consideration process, with cost reductions and productivity gains cited as the key perceived benefits. All respondents that have not considered outsourcing cited the importance of their data to the business function and the need to control this data as the primary reason.

When respondents were asked which data they would you be comfortable outsourcing (Figure 12), most (82%) said they would only be comfortable outsourcing market (i.e. nonproprietary) data with only 18% prepared to outsource asset (i.e. proprietary) data.

For data that would not be outsourced, two-thirds (65%) of respondents believe data requires an in house solution due to its business importance and complexity, with the balance (35%) saying there is a need to purchase a system specifically built to manage that data.

There is a sense that the economic values and data management benefits of outsourcing have not been properly communicated. Those considering outsourcing cite these benefits, while those opposed to outsourcing believe in total data control and are possibly less cost efficient than those embracing outsourcing.

When outsourcing is dismissed, the expenditure on either an in-house system or a build-to-order system risks outweighing the potential benefits of exercising total control over energy data. Consequently, those companies that dismiss outsourcing data management should only do so after they have fully assessed the risks and rewards of such an approach.

Three main conclusions can be drawn from this survey:

  1. Data management appears to be strongly driven by IT to the extent that business input may be overly subordinated. Yet this very data is the lifeblood of the various businesses. There are multiple challenges facing energy data management and to adequately mitigate these challenges requires looking beyond the IT/legacy challenges, as important as they are.
  2. Data management is increasingly being viewed as proprietary, while its asset value and the competitive advantage from efficient management are also increasing. As such there can be a reluctance to outsource data management, with the consequence that data management costs will increase as companies invest in building and purchasing their required data.
  3. Given these higher data costs, and the associated costs of developing in-house or buying a proprietary data management system, there is a strong persuasive argument in favour of data management system outsourcing, purchasing a proven Energy Data Management platform or subscribing to quality assured managed data services for non proprietary data.