First published in Water e-Journal Vol 3 No 2 2018.
We are often faced with uncertainty when making decisions – from trivial decisions such as whether to take an umbrella, or major decisions such as whether to buy that house. Appreciating the uncertainty in future conditions (‘Will it rain today?’; ‘Will house prices continue to go up?’) is crucial to making good decisions. This is no different for water resource managers, who need to make decisions on flood prevention, climate adaptation or coal resource developments. As scientists, we strive to inform decision-makers about uncertainties in a comprehensive, unbiased and transparent manner.
In this paper, we discuss some of the challenges and approaches used to communicate uncertainty during our contributions to the Bioregional Assessments Program, a federally-funded research project to assess the potential impacts of coal resource development on water resources and water-dependent assets in central and eastern Australia.
A first step in analysing potential impacts is to identify the causal pathways that detail how development activities can possibly affect the groundwater and surface water systems, and how these changes might affect the economic, social and ecological functioning of a region. This conceptual model provides the framework for detailed geological, hydrogeological, hydrological and ecological modelling.
Predictions have traditionally been made using a single deterministic model, a calibrated model that best fits the available observations. However, to assess the likelihood of potential impacts, we used a stochastic approach to create an ensemble of possible predictions (hundreds and thousands of possible answers) that are all consistent with the available observations. This results in a range or distribution of predictions. However, it is very challenging to communicate the range of model results, as well as all of the complexities and underlying assumptions, in a way that is relevant and accessible to decision-makers.
For bioregional assessments, we have worked with decision-makers to improve communication of uncertainty. This resulted in a consistent, calibrated language, tables, plots of the range of predictions and maps designed to convey probabilistic information in an intuitive manner. Further, model details and assumptions are documented in technical reports, and the data, models and predictions are made publicly available to increase transparency and reproducibility.
The amount and technical detail of that information can be challenging for decision-makers to identify what is important and what is not. To support decision-makers, we use a qualitative uncertainty analysis to summarise the rationale for, and effect on, prediction of each major assumption. This table, in combination with a plain English discussion, allows readers to rapidly appreciate the limitations, as well as opportunities for further data collection or modelling.
Bioregional assessments have highlighted the importance of early consultation with target audiences, which has enabled us to tailor the uncertainty communication products for decision-makers, as well as to avoid the potential for biased interpretation of results, where decision-makers are drawn to the extremes.
Like any other decision-maker, water resource managers are often faced with uncertainty when making decisions – such as how high to build a flood prevention levy bank, or whether to approve a proposed development near an important wetland or bird breeding area. Appreciating the uncertainty in future conditions – predictive uncertainty – is in making good decisions. As scientists, we strive to inform decision-makers on potential future conditions and their uncertainties in a comprehensive, unbiased and transparent manner. This is not only a first step in managing those uncertainties, it also provides decision-makers with a better understanding of the consequences of their decisions.
There is little value in using complex models or elaborating uncertainty analysis if the modelling results cannot be interpreted by the target audience. Probabilities of model predictions, a typical outcome of an uncertainty analysis, are, for instance, notoriously difficult to communicate effectively (Spiegelhalter et al., 2011). In the case of cumulative impact assessments, numerical models predict changes through space and time for a number of hydrological metrics with a range of possible input parameters. The resulting range of possible model predictions is a high-dimensional dataset, which is challenging to visualise and summarise. The challenge in visualisation and communication is to simplify high-dimensional datasets, through careful selection of variables or aggregation of spatio-temporal dimensions, in such a way that the main results can readily and intuitively be absorbed, while still retaining the relevant details and nuance of the underpinning data and models.
Methods to represent quantitative uncertainty of data spatially have been extensively reviewed. Spatial uncertainty can be represented using simple approaches to display different types of data separately (e.g. maps of data and uncertainty), more complicated approaches (e.g. intrinsic approaches that change the appearance of an object – colour hue, saturation, intensity, whiteness, transparency, blur) or extrinsic approaches that use additional symbols (e.g. hatching, glyphs). More than one approach is recommended when communicating uncertainty, because no one method will suit all audience.
Communicating deeper uncertainties resulting from incomplete or disputed knowledge – or from essential indeterminacy about the future – poses another challenge. In particular, decision-makers need to understand and interpret the effect that parameter and conceptual uncertainties have on model predictions. Uncertainty analysis needs to consider different sources of uncertainty, from input and parameter uncertainty in a model, over-acknowledged and unknown inadequacies in the modelling process to possible disagreements about the framing of the problem (Spiegelhalter and Riesch, 2011). Being mindful of these is important as some sources of uncertainty cannot be assigned probabilities, such as the choice of future development pathway. This is especially important as future development scenarios can only be projected (i.e. 'What would happen if...'), but not predicted ('What will happen').
To build trust and confidence, uncertainty analysis needs to be communicated in a comprehensive, unbiased and transparent manner that translates the results to support decision making. Open and transparent communication enables public scrutiny of the study, where each aspect of the study can be examined in detail. As scientists we strive to provide objective and independent advice. It is however unavoidable that conscious and unconscious biases affect our analysis, for instance when we adopt a conservative approach where we overestimate rather than underestimate impacts. To engender confidence in our model predictions it is paramount that we make decision-makers aware of known biases through clear statements.
While techniques to quantify the effect of potential sources of uncertainty are well documented, few studies offer practical guidance on communication of predictive uncertainty. Examples from fields such as climate change, nuclear waste disposal and weather forecasting, provide an opportunity to improve communication of uncertainty for cumulative impact assessment. This paper describes the challenges and solutions used in the communication of uncertainty analysis developed for the Bioregional Assessment Program. In this paper, we discuss some of the challenges and approaches used to turn uncertainty into confidence.
There is a growing number of proposals to develop coal seam gas and coal resources in eastern Australia. The proposed coal resource developments considered include the extraction of approximately 41,000 MT of coal from 58 new mines or expansions to existing mines, as well as approximately 16,000 PJ of natural gas from eight new coal seam gas developments (Figure 1). Their combined worth is estimated to potentially contribute almost $3 trillion dollars to the Australian economy.
Such developments may have an impact on water resources and the water-dependent assets that rely on this water. The Australian Commonwealth Scientific and Industrial Research Organisation (CSIRO), in partnership with the Australian Government’s Department of Environment and Energy, the Bureau of Meteorology and Geoscience Australia, has undertaken a series of cumulative impact assessments known as ‘bioregional assessments’ to assess the potential impacts of coal resource development on water resources and water-dependent assets such as wetlands and groundwater bores. The Program investigated six bioregions, subdivided into thirteen subregions that each have different geology, hydrology, ecology and coal resource developments.
The methodology for impact analysis was therefore adjusted and tailored to the particularities of each subregion. The high-level approach however, captured in a set of methodology reports, set out a series of overarching goals. The main aim of the work is to assess the potential cumulative, direct and indirect impact of new developments on hydrology and ecology in addition to the impacts from already approved developments. We adopted a probabilistic approach to systematically capture and document the uncertainties in this analysis. To enable open and transparent communication, all data, models and reports were made publicly available through bioregionalassessments.gov.au. In addition, the lineage of datasets, including the processing and modelling steps used to develop and interpret the modelling results were documented. The following sections describe the high-level methodology of uncertainty analysis and communication, starting with how to explore sources of uncertainties during the scoping and conceptualisation of the study.
A crucial step in any impact assessment study is framing the problem; defining what it is exactly that you are predicting the impact of, and on what. This will determine the scope of the study, which sources of uncertainty to include and which to exclude. Take for instance dust created during open-pit coal mining. This can have a detrimental effect on the ecology close to a mine, but, since it is not a water-related impact, it was not in scope of the bioregional assessments. Sourcing water for dust suppression however can cause hydrological and ecological impacts, which means it was in scope for the bioregional assessments.
Untangling the myriad ways that coal resource development can cause water related impacts necessitates developing a conceptual model of causal pathways. It describes the logical connections between coal resource developments and potential impacts on water-dependent assets.
The first step is to identify all possible hazards associated with the life cycle of the coal seam gas (CSG) developments and coal mines. Bioregional assessments used a modified version of Failure Modes and Effects Analysis (FMEA) to systematically identify and rank all hazards that could potentially link planned or unplanned activities to impacts on water resources or water-related assets. Examples of such causal pathways include depressurising an aquifer to extract CSG that could potentially decrease the groundwater flow rate to a spring or dewatering an aquifer to enable coal to be mined, which could potentially decrease baseflow to a stream.
Whether a mine or CSG development will proceed is one of the largest sources of uncertainty. Rather than embarking on elaborate socioeconomic analysis to account for this source of uncertainty, bioregional assessments used a single coal resource development pathway to describe existing baseline developments (as of December 2012) and the most likely additional future developments at the time of analysis. The reporting emphasised that all results are predicated on this assumption.
The conceptual model of causal pathways distils our understanding of the system in a simplified diagram that highlights the main causal pathways from development to potential impacts (Figure 2). The conceptual model shows the flow paths of water through the system (e.g. from recharge zones, flow through aquifers to discharge zones at springs, wells and streams) and how the coal resource development may potentially impact on water-dependent assets.
The causal pathways also make explicit which hydrological and ecological variables can be used to quantify potential impacts. For this purpose, assets were aggregated into landscape classes (e.g. terrestrial groundwater-dependent ecosystems, lowland riverine). For each landscape class a set of relevant set of receptor impact variables, such as leaf area index or number of macroinvertebrates, are defined through a structured discussion with local experts. The experts also indicated which hydrological response variables, such as groundwater level variations or the number of zero flow days, affect the receptor impact variables. The numerical groundwater and surface water models in bioregional assessments were designed to provide the change in hydrological response variables at the landscape scale. Receptor impact models, which are statistical models capturing local expert knowledge, are then used to translate that change in hydrology into a change in ecology.
Knowledge gaps identified during the conceptual modelling phase provide a first systematic assessment of the sources of uncertainty. For instance, recurring knowledge gaps in local hydrogeology include the hydraulic behaviour of aquitards and faults. Whether coal seams are in direct hydraulic connection with aquifers or separated by intact aquitards is often poorly known. Likewise, information on the location of faults and whether they are conduits or barriers to flow is usually limited. Where possible, such structural uncertainty in the conceptual models is propagated through to the numerical modelling and the quantitative uncertainty analysis. The qualitative uncertainty analysis records when technical or logistical limitations prevent us incorporating these sources of uncertainties and discusses the potential impacts on predictions. With respect to such conceptual or structural uncertainty, the BA program has been conscious of not under-estimating the uncertainties, rather than actively over-estimating them so as to be conservative.
For a decision-maker, it is essential to know which sources of uncertainty are incorporated in the predictive uncertainty analysis. While land use, climate and the coal resource development pathway are all highly uncertain, within bioregional assessments they are assumed to be fixed. The primary comparison of interest was between a future with and a future without the additional coal resource development. These additional factors could be incorporated into the assessment (e.g. by considering the distribution of possible future land-use trajectories) but the substantial increase in complexity needed to be weighed up against the potential dilution in focus.
There is a plethora of techniques available to quantify predictive uncertainty, but they are often perceived as very technical, difficult to implement in practice and most importantly, opaque and hard to communicate (Leung et al., 2015). In essence, uncertainty analysis explores the range of model predictions that are consistent with our knowledge of the modelled system. The design of the model and uncertainty analysis should explicitly address three main questions:
Causal pathway analysis establishes the objective of the model (e.g. what is the impact of development X on groundwater bore Y?) and the corresponding model predictions that are spatially and temporally explicit (e.g. the maximum drawdown over the next 100 years at bore Y, in the aquifer where bore Y is screened).
Model predictions are what the numerical model calculates, whereas the model objective addresses the management question. While the objective of the model is a high-level statement, model predictions are detailed descriptions of how the model will be used to address the objective. Clearly articulating model predictions is a critical step in the design of the model and uncertainty analysis to ensure that the objective of the model is addressed.
An often-neglected step in model conceptualisation is the systematic description of multiple possible conceptual models that are supported by our knowledge of the system. This step identifies additional potential sources of uncertainty, such as the role of faults or hydraulic functioning of springs. However, there are limitations to the number of sources of uncertainty that can be represented in a numerical model. To address and acknowledge these limitations, bioregional assessments combined a quantitative uncertainty analysis with a systematic qualitative uncertainty analysis.
The quantitative uncertainty analysis seeks to find the range of parameter combinations and corresponding model predictions that are consistent with the knowledge of the system and the qualitative uncertainty analysis systematically outlines the rationale behind model assumptions and potential effects on model predictions (Peeters, 2017).
Model predictions should be consistent with both historical observations (i.e. real-world data) and our knowledge of the system (expert knowledge). Independent knowledge, such as field measurements, literature values or expert opinion, is needed to ensure that the range of model parameters is plausible. This ensures that historical observations are reproduced using plausible model parameters, rather than using calibrated, but implausible model parameters. Further, using uncertainty analysis means that we do not just need the one best-fitting model, instead we need a range of possible models that agree with the range of observed data. However, this means that we need to explicitly specify what is an acceptable mismatch between observed and simulated values. For example, streamflow observations need to consider the detection limit of the gauge, while very precise groundwater level measurements need to consider the three-dimensional accuracy of the surveying and grid resolution of the groundwater model.
Sensitivity analysis identifies which parameters contribute to predictive uncertainty and which are most constrained by the data. Figure 3 shows the sensitivity of model predictions (drawdown) and observations (river fluxes and groundwater levels) to model parameters.
Predictions of maximum drawdown due to coal resource development are sensitive to the hydraulic properties of the coal seams, whereas historical observations of river flux and groundwater levels are sensitive to recharge, riverbed conductance and shallow aquifer hydraulic conductivity.
The sensitivity analysis indicates that while the model is consistent with the historical observations, these observations do not constrain the model to reduce predictive uncertainty.
Quantifying and acknowledging uncertainty adds several dimensions to a decision-making process. Deterministic results, single ‘best-estimate’ model predictions, give the illusion of certainty and allow for unambiguous application of decision rules. An example of such a rule is ‘A development cannot cause more than 20cm drawdown at a groundwater dependent ecosystem’.
If a groundwater model of this development produces a drawdown of 16cm, a decision-maker can quite confidently state that this development meets the drawdown impact criterion as it will not cause more than 20cm drawdown.
An uncertainty analysis of the same model may, however, inform that the model setup is conservative and overestimates impacts, that the most likely drawdown prediction is indeed 16cm but that there is a 23% probability that drawdown is greater than 20cm. In light of such findings, the confidence in the earlier statement evaporates rapidly. Rather than an unambiguous decision-making process that relies on deterministic model predictions, new questions need to be answered. Is a 23% probability of exceeding a threshold acceptable? How conservative is the model setup?
This example shows that while a deterministic prediction may appear unambiguous in relation to a threshold, the reality is that they can be misleading what may happen faithfully and can be misleading. A probabilistic representation brings some additional complexity to the interpretation but allows for better risk-based decision making. The role of science is to lay it out as objectively as possible, and let those who make decisions consider the burden of proof and incorporate their desire for precaution or conservatism into the decision. Different users (e.g. government regulators versus industry proponents) may focus on different parts of the predictive distribution. There is often a tendency to focus on the extremes of predictive distributions (95th or 99th percentile). As this can be very misleading, especially when dealing with skewed distributions, it is essential that the entire distribution is visualised and communicated.
How can we manage this apparent juxtaposition between transparently documenting model predictions and their uncertainty and providing unambiguous, clear information for science-based decision making? In bioregional assessments, we achieved this by turning uncertainty into confidence through the ‘zone of potential hydrological change’. This is a construct that rules out parts of the landscape where further assessment is not deemed necessary. By choosing a threshold beyond which a small change is very unlikely (e.g. less than a 5% chance of exceeding 20cm of additional groundwater drawdown, or a less than 5% chance of a 1% change in a surface water metric) there is high confidence that water-dependent assets or ecosystems beyond this zone are very unlikely to be impacted.
The predictive uncertainty may also be used to identify parts of the landscape where exceeding a more meaningful level of hydrological change is very likely (e.g. a greater 95% chance of exceeding 2 metres of additional groundwater drawdown).
Such a screening exercise provides a strong scientific basis to confidently rule out zones that are very unlikely to be affected. It also allows future assessment to move from a regional scale analysis, such as bioregional assessment, to a local, more focussed analysis. Anything within the zone of potential change warrants closer inspection, possibly by using more detailed local observations or modelling.
Bioregional assessors worked with decision-makers to improve communication of uncertainty by structuring the information clearly (Schmidt et al., 2014) and by using a consistent, calibrated language, and maps, plots and tables.
Bioregional assessments used two approaches to summarise predictive uncertainty: (1) probability of exceeding a threshold (e.g. 0.2 m groundwater drawdown) and (2) range of model predictions for defined percentiles of probabilities (e.g. 5th, 50th and 95th). The probability of exceeding a threshold enables the reader to select their own level of uncertainty (e.g. 20% chance of exceeding 0.2m), which is suitable when assessing potential impacts for assets protected by regulatory thresholds, such as those prescribed by Queensland’s Water Act 2000. Defined probabilities enable the reader to select their own level of impact (e.g. chance of experiencing 3.5m drawdown) and one of three levels of uncertainty (5th, 50th or 95th percentile estimates).
Communicating this complex scientific information about risk and uncertainty is challenging, particularly in politically contentious spaces where stakeholders are sensitive about nuances of language. This probabilistic modelling has not been used to date by regulators of coal resource development, nor by proponents, therefore the method of reporting results needed to be developed, so that they were understandable and useful to both audiences.
Across all 13 bioregional assessments, a common language for expressing probabilities was used as per the approach of the Intergovernmental Panel on Climate Change: the term ‘very likely’ was used to describe where there is a greater than 95% chance that the model results exceed thresholds, and ‘very unlikely’ was used where there is a less than 5% chance. Agreement on these terms (and many others) was facilitated across disciplines and geographic areas, and published in an online glossary.
However, not all results were quantitative: in some cases there was insufficient evidence to report anything but qualitative results based on logic. As qualitative data often requires expression using natural language instead of numbers, authors used terms such as ‘negligible’, ‘minimal’ or ‘large’. The varying stakeholders were sensitive to nuances in these terms, and sometimes had conflicting requirements. Ultimately the Program needed to balance the users’ needs for formal definitions of terms that describe risk and uncertainty, with the authors’ need for a range of words that communicate complex scientific information in an unbiased way without overstating the confidence in results.
In addition to textual descriptions, bioregional assessments used a combination of maps, plots and tables of the range of predictions to convey probabilistic information in an intuitive manner. The range of model predictions can be communicated in four ways: spatial distribution, cumulative plots, summary tables and calibrated language. Spatial distributions simplify high-dimensional datasets by aggregating spatio-temporal dimensions (i.e. maximum drawdown over 100 years) to enable the results to be readily and intuitively understood, while retaining the relevant details and nuance of the underpinning data and models. Cumulative plots and summary tables enable the reader to identify the number or extent of assets that are potentially impacted.
Figure 4 summarises predictions of groundwater drawdown near a proposed open-cut coal mine. This enables the reader to visualise the spatial distribution (map), number of groundwater bores exceeding the 0.2m or 5m drawdown thresholds (cumulative plots and tables), which is summarised in the key messages. Figure 5 shows the spatial distribution of groundwater drawdown for three defined probabilities, which can be used to visualise the extent potentially impacted. These techniques enable the uncertainty analysis to be communicated in a comprehensive, unbiased and transparent manner to build trust and confidence and ultimately translate the results to inform decision making.
Accounting for and acknowledging uncertainty in model predictions is essential to enable robust decision making. Scientists not only need to make sure their analysis is rigorous, they need to be open and transparent in documenting their work and they also need to communicate and visualise their results in such a way that the essence can easily be digested.
While this remains a daunting task, we show through examples from the bioregional assessments that such an approach can be applied in complex impact assessments and, moreover, that careful framing of uncertainty results can increase confidence in model predictions.
This research is part of the Bioregional Assessment Program, which is funded by the Australian Government Department of the Environment and Energy. The Bioregional Assessment Program is a transparent and accessible program of baseline assessments that increase the available science for decision making associated with the impacts of coal seam gas and coal mining development on water resources. Bioregional assessments are a collaboration between the Department of Environment and Energy, the Bureau of Meteorology, CSIRO and Geoscience Australia. For more information visit www.bioregionalassessments.gov.au.
Dr Luk Peeters | CSIRO Mineral Resources
Luk’s research focuses on modelling groundwater dynamics at regional to continental scales with a focus on the uncertainty of model predictions both quantitatively and qualitatively. Luk led the development and application of the uncertainty analysis, including propagation of uncertainty between models, for the Bioregional Assessments Program. He is passionate about improving the communication of uncertainty analysis to a range of audiences.
Dr Russell Crosbie | CSIRO Land and Water
Russell’s research spans water resources, climate change, salinity and water in the resources sector, including high profile projects such as Sustainable Yields Projects (MDB, Northern Australia and Tasmania), the Water Information Research and Development Alliance (WIRADA) and the Bioregional Assessments Program (BA). He is best known for his expertise in estimating groundwater recharge and is the author (or co-author) of over 100 scientific publications.
Dr Brent Henderson | CSIRO Data61
Brent works at the interface of statistical science and risk analysis to address the environmental and agricultural challenges facing Australia. He has broad interests in spatio-temporal modelling, monitoring and survey design, reconciling data with modelled observations, multivariate analysis, risk analysis and cumulative impact analysis. Brent led the development and delivery of science-based assessments for the Bioregional Assessments. Current projects include using digital data to inform farmer-led decision-making in the Digiscape future science platform within CSIRO and a risk assessment for potential release of a carp virus in Australian waters to suppress the carp population.
Dr Kate Holland | CSIRO Land and Water
Kate’s research addresses significant natural resource management challenges in Australia, including floodplain salinity, environmental flows, catchment hydrology and regional-scale groundwater resources. She recently led a multi-disciplinary team (geology, hydrogeology, hydrology, ecology, risk assessment and uncertainty analysis) assessing the cumulative impacts of coal resource development in the sub-region in the Bioregional Assessments Program.
Dr Steven Lewis | Geoscience Australia
Steve is a mineral exploration geologist with extensive experience in diamonds and hydrothermal ore systems. His recent focus at Geoscience Australia is in hydrogeology and impacts of coal seam gas developments and coal mining.
Dr David Post | CSIRO Land and Water
David’s research interests focus broadly on the impacts of land use and climate change on water resources, as well as on the regionalisation of hydrologic response to ungauged areas. He is currently the Projects Director of the Bioregional Assessment Program and has led other high-profile projects such as South Eastern Australian Climate Initiative (SEACI) and the Sustainable Yields Project in Tasmania. He has a PhD in Resource Management from the Australian National University and is President of the Modelling and Simulation Society of Australia and New Zealand (MSSANZ).
Dr Rebecca Schmidt | CSIRO Land and Water
Becky is a scientific integrator with expertise in developing, integrating and delivering information that can be effectively used to bridge the gap between science and policy. She has lead the improvement in communication of science, particularly by improving language and publication standards for large-scale interdisciplinary projects addressing politically sensitive issues, including the Bioregional Assessment Program, Sustainable Yields Projects and the Flinders and Gilbert Agricultural Resource Assessment.