Óptica financiera sobre la importancia y los beneficios de implementar de manera integral y adecuada un Safety Management System en nuestras organizaciones aéreas.
Victor Manuel del Castillo
AeroSafety World November2012
Calculating Safety ROI
The basic math is easy, but thoughtful effort is required to identify the correct data.
BY WILLIAM B. JOHNSON AND KATRINA AVERS
The math needed to determine the financial return on investment (ROI) for safety interventions is easy (ASW, 10/12, p. 16), but technical expertise is required to calculate the associated benefits and investments.
The key performance indicators discussed by safety executives may differ from those discussed by the corporate finance department. One group may count unstabilized approaches, go-arounds and employee injuries. The other group looks at quarterly financial performance. Safety and profitability are the mutually inclusive, number one priority for most industries, especially transportation.
If you think that safety and finance are not related, then consider how quickly customers flee an airline or a company after a catastrophic event. For example, the 2010 oil spill in the Gulf of Mexico had an extreme impact, not only from clean-up costs but also from the costs associated with public perception. Airline stock prices take a big hit following an accident. Sales are threatened when new model aircraft develop unexpected failures. Offshore helicopter operations suffer the same fate when their safety records are in question.
In most cases, the highly visible catastrophes could have been prevented with safety interventions that seem inexpensive, especially after the fact. The operator could have had more training, the extra safety mechanism should have been installed, the vessel or aircraft could have had one extra safety-oriented design feature, and the company should have tracked the event precursors more closely.
The examples above refer to major events, which seldom occur. This article focuses on the hundreds, if not thousands, of small hazards or errors that add up and ultimately injure employees, impact production and service, and contribute to financial losses. The costs of such errors should not be considered “the cost of doing business” but rather the cost of not doing business as well as possible. These incidents are indicators of organizational safety and potential predictors of aviation accidents.
This article describes an approach to predicting and/or measuring the cost and safety return, or benefit, on safety interventions. It helps technical and safety personnel make a business case for their programs by offering the fundamental vocabulary and procedures for discussing and calculating ROI. It helps finance personnel to see the direct correlation between safety and profit.
The ROI formula is the easy part. Economists who reviewed the approach say that the procedures and math of the simplified calculations are reasonable and correct. It is a matter of addition, subtraction and division. Anyone can calculate ROI.
But the catch comes with the work involved in identifying the benefits and the investments that must be added up, subtracted and divided. Writings and speeches about ROI have not sufficiently emphasized the technical effort of deriving investment and benefit data.
Some technical personnel have not yet adopted the ROI mindset, perhaps because they have not been convinced of the value of their ROI efforts. Typically, they fix problems rather than assign costs and calculate ROI, and they do not always know the entire cost of an error because their priority is production and schedules. Other factors are that financial personnel are the ones who typically perform cost and investment analyses, that executives do not demand ROI calculations for many technical interventions and that corporate culture usually does not expect ROI data from technical personnel.
Now Is the Time
The many recent papers1-7 and speeches discussing the benefits of calculating safety ROI have not changed aviation corporate behavior. However, the increasing worldwide emphasis on safety data may encourage the use of this tool. Safety management systems (SMSs) demand a process and a culture to analyze key performance indicators, to formally identify hazards, to establish management interventions and to measure impact. These activities provide the data and the motivation to increase efforts to calculate ROI. The simplified ROI model has not changed, but the corporate culture has.
You must thoroughly understand your safety challenges before you can calculate ROI, and an SMS can be the foundation for understanding these organizational challenges and determining the procedures and associated costs necessary to manage the risk. An SMS, supported by the right safety culture, can help identify the hazards that contribute to risk. SMS and ROI go hand in hand.
After you conduct a reasonable risk assessment, you know the possible negative outcomes as well as the probability that they could happen. You also know how to address the individual hazards contributing to risk. For example, you know that you have a problem of communication during shift turnover in aircraft maintenance. The afternoon shift has limited overlap with the graveyard shift. As a result, there have been many task handovers when critical information was not conveyed. This communication has resulted in missed steps in maintenance or a repeat of work that has already been completed.
Your SMS data help you know the consequences of that challenge. You can also count the number of times that an issue may have affected airworthiness and/or safety. You can put a value on the cost of the resulting rework, the associated delay of delivery, flight delays and other related costs. Finally, you can determine a remedy — for example, new documentation procedures or increasing the time of shift overlap. In threat and error management terms, you know how to manage the threat to reduce or eliminate the error. You know the costs of the hazard and the costs and timetable of the intervention. Your field experience may help you to assign some level of confidence to your planned solution. This prepares you for an accurate ROI estimation. With the ROI information, you can decide how to proceed. The SMS data can not only identify threats but also help you show, in terms of safety and cost, how your intervention affected the number of subsequent events.
What follows is one example of an ROI calculation that demonstrates the safety and financial payback on a fatigue awareness program implemented by a large maintenance and repair organization (MRO). The six-quarter ROI was more than 3-to-1 on a $200,000 investment.
The ROI calculator, developed in cooperation with Booz Allen Hamilton, is available at the U.S. Federal Aviation Administration’s (FAA’s) website on fatigue management for maintenance personnel — <mxfatigue.com>. The software comprises a sophisticated set of connected Excel worksheets and includes extensive user documentation and guidance. The ROI calculation is based on a straightforward formula that subtracts the total cost from the net return (expected benefit times the probability of success) and divides that number by the total cost (Figure 1). However, the calculation can only be as accurate as the data you input, so you must commit a reasonable amount of effort up front to establish the expected net investment (cost) and the expected net return (benefit).
In this example, a large maintenance organization acknowledged human fatigue as a safety risk. The company began collecting data on the contribution of fatigue to company incidents and accidents. Questions from FAA fatigue management documents were used to identify events in which fatigue was a contributing factor, and the company instituted scheduling limits in 2009. In 2011, the company instituted fatigue countermeasure training as a safety intervention for all maintenance technicians and management. The training was implemented from January 2011 to January 2012.
The training, developed by the FAA–Industry Maintenance Fatigue Workgroup, comprised 90 minutes of interactive training and testing, and viewing of a video titled “Grounded” (available free at <mxfatigue.com>). The computer-based training was delivered at multiple locations throughout the company.
This section demonstrates the ROI calculations, using the FAA’s calculator.
Figure 2 shows the company’s personnel cost estimates for implementing the training. An additional section of the worksheet, not shown in the figure, lists non-labor costs like hardware, facilities, supplies and other expenses. To identify these costs, the company answered about a dozen questions devised to help first-time users collect the necessary data and complete the investment form — for example, “How many personnel were trained?” “Did you have to buy special hardware?” and “Over how many quarters did the training occur?” Other questions may be added as needed.
In the example, the responses to the questions showed that the investment costs were limited to personnel time, and that personnel expenses were limited to the time of the trainees and some of the management and administrative support. The employees completed the training via the FAA safety website <faasafety.gov>. Company training personnel logged completions for corporate tracking. Forty percent of the employees completed the training away from the worksite, so there were no lost production costs. The others trained instead of working, so the cost was associated with their unavailability. As previously mentioned, there was no cost to the company to develop the training.
Data on investments and returns do not show the quarterly cash flow, or the timeline for financial and safety returns. The next steps required the company to assign estimated spending and return rates by quarter. These data are not presented here.
To estimate the return on the training, the company answered a series of questions regarding financial and safety returns, such as “How many safety incidents do you expect the intervention will resolve?” “What key performance indicators will be influenced by this intervention?” “What are the metrics you will use to measure these changes (e.g., aircraft damage, rework delivery delay, employee injuries, lost time job injuries)?” and “What are the costs associated with each metric you selected?”
The company expected to see a reduction in aircraft damage and injuries8 compared with 2010 performance (Figure 3). The company believed the training could target 10 percent of the predicted aircraft damage events (10 percent of 89 events in 2011, at an average cost of $105,000) and 10 percent of the predicted on-the-job injuries (10 percent of 189 injuries in 2011, at an average cost of $6,307).
Most ROI is calculated based on predictions of expected costs and returns, derived from estimates that likely are not completely accurate. Therefore, the probability of success is part of the calculation. It is used in the formula to compute net return and is a function of prior experience, the level of corporate support, the availability of resources and the amount of planning that is committed to the development of the safety intervention.
Figure 4 shows 20 questions, rated by the company using a 5-point scale to assign a probability of success. The software automatically assigns a plus or minus 10 percent confidence level around the probability in the output. In this example, the probability of the training intervention successfully resolving the target safety and investment returns was 80 percent.
Figure 5 p. 42 shows the ROI output chart in the project analysis summary. In this example, the ROI over six quarters is 312 percent. The original investment of personnel time is paid back within the first quarter. The extraordinarily high ROI is partially attributable to the extremely low training costs. Even if the company had made a large investment in training materials, however, there still would have been a high payback.
The company estimated, conservatively, that adherence to the fatigue training could improve worker efficiency by an additional 1 percent. In 2011, 1 percent of all hours worked would have meant a benefit of $900,000 in efficiency (an amount not included in ROI calculation). When the investment is low and the benefits are high, the ROI can be impressive.
ROI calculations can inform decisions about safety interventions. Following implementation of safety interventions, a straightforward comparison of performance can be made from one year to the next. In this company, the cost of aircraft damages was reduced by nearly 30 percent in 2011, compared with 2010. That is $3.04 million in savings. The number of injuries was unchanged in 2011, but the average cost of an incident was reduced by nearly 15 percent, resulting in savings of $183,534. These performance improvements were achieved by a variety of programs, including the fatigue countermeasure training.
More ROI Examples
During 2012, the authors worked with airlines, manufacturers and MROs to implement the ROI procedure, as outlined in the FAA website. It became obvious that every safety intervention was not conducive to a reasonable ROI. For example, one airline reported a series of incidents in which a company procedure resulted in a certain part of the landing gear not being properly torqued when the task was transferred from one shop to another. An employee noticed the procedural error and reported it through a corporate voluntary reporting system. Neither the airline nor the manufacturer saw a safety issue. The company adjusted the procedure to correct the hazard. There was a fine imposed by the regulator because of a lengthy non-compliance period. Obviously, the authors did not use avoidance of a regulatory penalty as an exemplary numerator for an ROI calculation.
You must be careful to be accurate in your estimates and measures and to remember that conservative, relatively low estimates are often best. Also, it is important to be aware of relevant developments when you attribute savings and safety improvements solely to your intervention. For example, a few years ago, a researcher claimed that his intervention reduced personal injury by nearly 90 percent at an airline maintenance facility. He was unaware that, when he made the final measures, the facility had reduced staff by nearly 75 percent.
Sometimes your safety intervention may have unexpected positive or negative results. For example, in one instance, an airline did not plan to calculate the additional benefit of improved employee safety but then determined that numerous incidents were being prevented because of the intervention.
Should the investment or benefits change, the worksheet makes it easy to alter the values and immediately recompute the ROI value.
Some say that calculating ROI focuses too much on money and not enough on safety. Nevertheless, money and safety are inseparably linked. While ROI is a financial concept, the monetary returns are largely driven by the safety returns. Safety interventions make a difference. It will take executive attention and ROI calculations to make these interventions a priority. Safety interventions like the ones shown above can be the gateway to a competitive advantage, instead of being the first thing cut when budgets are tightened.
Although the FAA ROI Calculator provides step-by-step instructions and guidance, the software cannot check the quality of your input. The hard work is up to you.
William B. Johnson, Ph.D., is the FAA chief scientist and technical adviser for human factors in aircraft maintenance systems. Katrina Avers, Ph.D., is a research scientist at the FAA Civil Aerospace Medical Institute.
This work was supported by the Civil Aerospace Medical Institute, the Human Factors Research and Engineering Group, the FAA Flight Standards Directorate, Office of Aviation Safety Chief Scientific and Technical Advisor Program and industry partners that provided critical data to test the ROI process. This article was developed from a presentation by Johnson and Avers to the Shell Aviation Safety Seminar in October in The Hague, Netherlands.
- Johnson, W.B.; Sian, I.B.; Watson, J. (2000). “Measuring the Impact of Human Factors Interventions.” SAE Meeting on Advances in Aviation Safety. Daytona Beach, Florida, U.S. April 11–13, 2000.
- Hastings, P.A.; Merriken, M.; Johnson, W.B. (2000). “An Analysis of the Costs and Benefits of a System for FAA Safety Inspectors.” International Journal of Industrial Ergonomics, 26, 231–248.
- Johnson, W.B.; Avers, K. (2012). “Calculating Payback for Safety and Training Programs.” The Journal for Civil Aviation Training. Issue 2/2012.
- Johnson, W.B. (2006). “Return on Investment in Human Factors.” The Journal for Civil Aviation Training. Issue 4/2006.
- Krois, P.; Farrow, D.; Johnson, W.; Blair, D. (2007). “Advancing the Human Factors Business Case.” InProceedings of the 14th Annual Symposium on Aviation Psychology. Dayton, Ohio, U.S.: Wright State University. April 25, 2007.
- Johnson, W.B. (2012). “Looking for the Big ASAP Success Story.” Aircraft Maintenance Technology Magazine. July 2012, 24–28.
- FAA (2005). The Operator’s Manual for Human Factors in Aviation Maintenance.<www.hf.faa.gov/opsmanual>.
- In this instance, “injuries” are those that must be reported under U.S. Occupational Safety and Health Administration (OSHA) guidelines, including injuries that result in fatalities, lost workdays, job transfers or termination of employment, or that require medical treatment.