Sunday, 5 November 2017

Averaging Risk is Risking the Average
To assure public safety regulatory agencies like the ACM in the Netherlands or Ofgem in the UK monitor the performance of gas and electric grid operators on area’s like costs, safety and the quality of their networks. The regulator compares the performance of the grid operators and decides on incentives to stimulate improvements in these areas. Difficulty with these comparisons is that grid operators use different definitions and/or methodologies to calculate performance, which complicates a like for like comparison on for example asset health, criticality or risk across the grid operators. In the UK this has led to a new concept for risk calculations, the concept of monetised risk. In calculating monetised risk not only the probability of failure of the asset is used, also the probability of the consequence of a failure and its financial impact are taken into account. The question is if this new method delivers more insightful risk estimations to allow for a better comparison among grid operators. Also, will it support fair risk trading among asset groups or the development of improved risk mitigation strategies?

The cost - risk trade-off that grid operators need to make is complex. Costly risk reducing adjustments to the grid need to be weighed against the rise in cost of operating the network and therefore the rates consumers pay for using the grid. For making the trade-off, an estimate of the probability of failure of an asset is required. In most cases, specific analytical models are developed to estimate these probabilities. Using pipeline characteristics like type of material, age, and data on the environment the pipeline is in (i.e. soil type, temperature and humidity) pipeline specific failure rate models can be created. Results from inspections of the pipeline can be used to further calibrate the model. Due to the increased analytics maturity of grid operators, these models are becoming more common. Grid operators are also starting to incorporate these failure rate models in the creation of their maintenance plans.

Averaging the Risk

As you can probably imagine, there are many ways for constructing failure rate models. This makes it difficult for a regulator to compare reported asset conditions from the grid operators, as these estimates could have been based on different assumptions and modelling techniques.  That is why, in the UK at least, it was agreed between the 4 major gas distribution networks (GDN), to standardise the approach.  In short, the method can be described as follows.
  1. Identify the failure modes of each asset category/sub group in the asset base and estimate the probability of failure for each identified failure mode. 
  2. For each failure mode the consequences of the failure are identified, including the probability of the consequence occurring.
  3.  For each consequence the monetary impact is estimated. 
  4.  By summing up over all failure modes and consequences, a probability weighted estimate of monetised risk for an asset category/sub group is calculated. Summarising over all asset categories/sub groups gives a total level of monetised risk for the grid.

This new standardised way of calculating risks makes the performance evaluation much easier, it also  allows for a more in-depth comparison. See for more details on the method the official documentation.

An interesting part of this new way of reporting risk is the explicit and standardised way of modelling asset failure, consequence of asset failure and cost of the consequence. This is similar to how a consolidated financial statement of a firm is created. Therefore, you could interpret it as a consolidated risk statement. But can risks of individual assets or asset groups be aggregated in the described way and provide a meaningful estimate of the total actual risk? The above described approach sums the estimated (or weighted average) risk for each asset category/sub group, so it’s an estimate of the average risk for the complete asset base. However risk management is not about looking at the average risk, it’s about extreme values. For those who read Sam Savage’s The Flaw of Averages or Nassim Taleb’s Black Swan know what I’m talking about.

Risking the Average

Risks are characterised by extreme outcomes, not averages. To be able to analyse extreme values, a probability distribution of the outcome you’re interested in is required. Averaging reduces the distribution of all possible outcomes to a point estimate, hiding the spread and likelihood of all possible outcomes. Also, averaging risks ignores the dependence between each of the identified modes of failure or consequence. To illustrate let’s assume that we have 5 pipelines, each with a probability of failure of 20%. There is only one consequence (probability =1) with a monetary impact of 1,000,000. The monetised risk per pipeline than becomes 200,000 (=0,20*1,000,000), for the total grid it is equal to 1,000,000. If we take dependence of the failures into account than there will be a 20% probability of all pipes failing when these are fully correlated events. There will be a 0,032% change of all pipes failing if they are fully independent. The estimated financial impact than ranges from 1,000,000 in the fully correlated case to 1,600 in the fully independent case. That’s quite a range which isn’t visible in the monetised risk approach.

Regulators must assess risk in many different areas. Banking has been top of mind in the past years, but industries like Pharma and Utilities also had a lot of attention. How a regulator decides to measure and asses risk is very important. If risks are underestimated, this could impact society (like a banking crisis, deaths due to the admission of unsafe drugs or increase of injuries due to pipeline failures). If risks are overestimated costly mitigation might be imposed, again impacting society with high costs. The above example shows that the monetised risk approach is insufficient as it estimates risk with averages, where in risk mitigation the extreme values are much more important. What than is a better way of aggregating these uncertainties and risks than just averaging them?

Monte Carlo Simulation

The best way to better understand the financial impact of asset failure is to construct a probability density function of all possible outcomes using Monte Carlo simulation and based on that distribution make the trade-off between costs and risk. Monte Carlo Simulation has proven its value in many industries and in this case will provide what we need. Using the free tools of Sam Savage’s the above hypothetical example of 5 pipe lines can be modelled and the distribution of financial impact analysed. In just a few minutes the below cumulative distribution (CDF) of the financial impact for the 5 pipelines case can be created. Remember that the monetised risk calculation resulted in a risk level equal to the average, 1,000,000.

From the graph it immediate follows that P(Financial Impact<=Monetised Risk) = 33%. It implies that the P(Financial Impact>Monetised Risk) = 1-33%=66%. So, a 66% chance that the financial impact of pipe failures will be higher than the calculated monetised risk. Therefore we’re taking a serious risk by using the averaged asset risks. Given the objective of better comparison of grid operator performance and enabling risk trading between asset groups, the monetised risk method is to simple I would say. By averaging the risks, the distribution of financial impact is rolled up into one number leaving you no clue on what the actual distribution looks like (See also Sam Savage’s : The Flaw of Averages) A better way would be to set an acceptable “risk threshold” (say 95%) and use the estimated CDF to determine the corresponding financial impact.

This approach would also allow for better comparison of grid operators by creating a cumulative distribution for all of them and plotting them together into one graph (See example below). In a similar way risk mitigations can be evaluated and comparisons made between different asset groups, allowing for better informed risk trading.

Standardising the way in which asset failures and consequence of failures are estimated and monetised definitely is a good step towards a comparable way to measure risk. But risks should not be averaged in the way the monetised risk approach suggests. There are better ways, which will provide insight on the whole distribution of risk. Given the available tools and computing power, there is no reason not to do so. It will improve our insights on the risks we face and help us find the best mitigation strategies to reducing public risks.
Post a Comment