From the INSTITUTE OF ATMOSPHERIC PHYSICS, CHINESE ACADEMY OF SCIENCES and the “all models are wrong, some might be useful” department.

Researchers work toward systematic assessment of climate models

A research team based at the Pacific Northwest National Laboratory in Richland, Wash., has published the results of an international survey designed to assess the relative importance climate scientists assign to variables when analyzing a climate model’s ability to simulate real-world climate.

The results, which have serious implications for studies using the models, were published as a cover article in Advances in Atmospheric Sciences on June 22, 2018.

In assessing climate models, experts typically evaluate across a range of criteria to arrive at an overall evaluation of the model’s fidelity. They use their knowledge of the physical system and scientific goals to assess the relative importance of different aspects of models in the presence of trade-offs. Burrows et al. (2018) show that climate scientists adjust the importance they assign to different aspects of a simulation depending on the science question the model will be used to address. Their research also shows that expert consensus on importance differs across model variables. CREDIT
Advances in Atmospheric Sciences

“Climate modelers spend a lot of effort on calibrating certain model parameters to find a model version that does a credible job of simulating the Earth’s observed climate,” said Susannah Burrows, first author on the paper and a scientist at the Pacific Northwest National Laboratory who specializes in Earth systems analysis and modeling.

However, Burrows noted, there is little systematic study on how experts prioritize such variables as cloud cover or sea ice when judging the performance of climate models.

“Different people might come to slightly different assessments of how ‘good’ a particular model is, depending to large extent on which aspects they assign the most importance to,” Burrows said.

One model, for example, may better simulate sea ice while another model excels in cloud simulation. Each scientist must strike a balance between their competing priorities and goals–a difficult thing to capture systematically in data analysis tools.

“In other words, there isn’t a single, completely objective definition of what makes a ‘good’ climate model, and this fact is an obstacle to developing more systematic approaches and tools to assist in model evaluations and comparisons,” Burrows said.

The researchers found, from a survey of 96 participants representing the climate modelling community, that experts took specific scientific objectives into consideration when rating variable importance. They found a high degree of consensus that certain variables are important in certain studies, such as rainfall and evaporation in the assessment of the Amazonian water cycle. That agreement falters on other variables, such as how important it is to accurately simulate surface winds when studying the water cycle in Asia.

Understanding these discrepancies and developing more systematic approaches to model assessment is important, according to Burrows, since each new version of a climate model must undergo significant evaluation, and calibration by multiple developers and users. The labor-intensive process can take more than a year.

The tuning, while designed to maintain a rigorous standard, requires experts to make trade-offs between competing priorities. A model may be calibrated at the expense of one scientific objective in order to achieve another.

Burrows is a member of an interdisciplinary research team at PNNL working to develop a more systematic solution to this assessment problem. The team includes Aritra Dasgupta, Lisa Bramer, and Sarah Reehl, experts in data science and visualization, and Yun Qian, Po-Lun Ma, and Phil Rasch, climate science experts.

To help climate modelers understand these trade-offs more clearly and efficiently, the visualization researchers are building interactive, intuitive visual interfaces that allow modelers to summarize and explore complex information about different aspects of model performance.

The data scientists are working to characterize expert climate model assessment in greater detail, building on the findings from the initial survey. Eventually, the researchers aim to blend a combination of metrics with human expertise to assess how well-suited climate models are for specific science objectives, as well as to predict how frequently experts will agree or disagree with that assessment.

“[We plan] to combine the best of both worlds, using computing to reduce manual effort and allowing scientists to more efficiently apply their human insight and judgment where it is most needed,” Burrows said.

###

Here is the paper: https://link.springer.com/article/10.1007%2Fs00376-018-7300-x

Characterizing the Relative Importance Assigned to Physical Variables by Climate Scientists when Assessing Atmospheric Climate Model Fidelity

Abstract

Evaluating a climate model’s fidelity (ability to simulate observed climate) is a critical step in establishing confidence in the model’s suitability for future climate projections, and in tuning climate model parameters. Model developers use their judgement in determining which trade-offs between different aspects of model fidelity are acceptable. However, little is known about the degree of consensus in these evaluations, and whether experts use the same criteria when different scientific objectives are defined. Here, we report on results from a broad community survey studying expert assessments of the relative importance of different output variables when evaluating a global atmospheric model’s mean climate. We find that experts adjust their ratings of variable importance in response to the scientific objective, for instance, scientists rate surface wind stress as significantly more important for Southern Ocean climate than for the water cycle in the Asian watershed. There is greater consensus on the importance of certain variables (e.g., shortwave cloud forcing) than others (e.g., aerosol optical depth). We find few differences in expert consensus between respondents with greater or less climate modeling experience, and no statistically significant differences between the responses of climate model developers and users. The concise variable lists and community ratings reported here provide baseline descriptive data on current expert understanding of certain aspects of model evaluation, and can serve as a starting point for further investigation, as well as developing more sophisticated evaluation and scoring criteria with respect to specific scientific objectives.