Characterising performance of environmental models

Akaike, 1974, A new look at the statistical model identification, IEEE Transactions on Automatic Control, 19, 716, 10.1109/TAC.1974.1100705

Alexandrov, 2011, Technical assessment and evaluation of environmental models and software: letter to the Editor, Environmental Modelling and Software, 26, 328, 10.1016/j.envsoft.2010.08.004

Andréassian, 2009, Crash tests for a standardized evaluation of hydrological models, Hydrology and Earth System Sciences, 13, 1757, 10.5194/hess-13-1757-2009

Arhonditsis, 2004, Evaluation of the current state of mechanistic aquatic biogeochemical modeling, Marine Ecology-Progress Series, 271, 13, 10.3354/meps271013

Bates, 1988

Beck, 2006, Model Evaluation and Performance, 10.1002/9780470057339.vam028

Bellman, 1970, On structural identifiability, Mathematical Biosciences, 7, 329, 10.1016/0025-5564(70)90132-X

Bellochi, 2009, Validation of biophysical models: issues and methodologies. A Review, Agronomy for Sustainable Development

Bennett, 2010, Performance evaluation of environmental models

Berthet, 2010, How significant are quadratic criteria? Part 2. On the relative contribution of large flood events to the value of a quadratic criterion, Hydrological Sciences Journal, 55, 1063, 10.1080/02626667.2010.505891

Beven, 2008

Blocken, 2012, Ten iterative steps for model development and evaluation applied to Computational Fluid Dynamics for Environmental Fluid Mechanics, Environmental Modelling and Software, 33, 1, 10.1016/j.envsoft.2012.02.001

Boucher, 2009, Tools for the assessment of hydrological ensemble forecasts obtained by neural networks, Journal of Hydroinformatics, 11, 297, 10.2166/hydro.2009.037

Boyle, 2000, Toward improved calibration of hydrologic models: combining the strengths of manual and automatic methods, Water Resources Research, 36, 3663, 10.1029/2000WR900207

Campo, 2010, Modelling with stakeholders within a development project, Environmental Modelling and Software, 25, 1302, 10.1016/j.envsoft.2010.01.005

Carpentieri, 2012, Evaluation of a neighbourhood scale, street network dispersion model through comparison with wind tunnel data, Environmental Modelling and Software, 37, 110, 10.1016/j.envsoft.2012.03.009

Chapman, 1996, Nonlinear compartmental model indistinguishability, Automatica, 32, 419, 10.1016/0005-1098(95)00152-2

Checchi, 2005, Reliability of parameter estimation in respirometric models, Water Research, 39, 3686, 10.1016/j.watres.2005.06.021

Chiew, 1993, Assessing the adequacy of catchment streamflow yield estimates, Australian Journal of Soil Research, 31, 665, 10.1071/SR9930665

Choi, 2007, Multi-period and multi-criteria model conditioning to reduce prediction uncertainty in an application of topmodel within the GLUE framework, Journal of Hydrology, 332, 316, 10.1016/j.jhydrol.2006.07.012

Chou, 2007, Applying multi-resolution analysis to differential hydrological grey models with dual series, Journal of Hydrology, 332, 174, 10.1016/j.jhydrol.2006.06.031

Christoffersen, 1998, Evaluating intervals forecasts, International Economic Review, 39, 841, 10.2307/2527341

Cochran, 1977

Congalton, 1991, A review of assessing the accuracy of classifications of remotely sensed data, Remote Sensing of Environment, 37, 35, 10.1016/0034-4257(91)90048-B

Coron, 2012, Crash testing hydrological models in contrasted climate conditions: an experiment on 216 Australian catchments, Water Resources Research, 48, W05552, 10.1029/2011WR011721

Costanza, 1989, Model goodness of fit: a multiple resolution procedure, Ecological Modelling, 47, 199, 10.1016/0304-3800(89)90001-X

Crochemore, L., 2011. Evaluation of Hydrological Models: expert Judgement vs Numerical Criteria. Master thesis, Université Pierre et Marie Curie, PolyTech Paris, France, 54 pp.

Dawson, 2007, HydroTest: a web-based toolbox of evaluation metrics for the standardised assessment of hydrological forecasts, Environmental Modelling and Software, 22, 1034, 10.1016/j.envsoft.2006.06.008

Dawson, 2010, HydroTest: further development of a web resource for the standardised assessment of hydrological models, Environmental Modelling and Software, 25, 1481, 10.1016/j.envsoft.2009.01.001

De Pauw, D.J.W., 2005. Optimal Experimental Design for Calibration of Bioprocess Models: a Validated Software Toolbox. PhD thesis in Applied Biological Sciences, BIOMATH, University of Gent. URL: http://biomath.ugent.be/publications/download/.

Devisscher, 2006, Estimating costs and benefits of advanced control for wastewater treatment plants – the MAgIC methodology, Water Science and Technology, 53, 215, 10.2166/wst.2006.126

Dochain, 2001

Doney, 2007, Mechanisms governing interannual variability of upper ocean temperature in a global ocean hindcast simulation, Journal of Physical Oceanography, 37, 1918, 10.1175/JPO3089.1

Ebisuzaki, 1997, A method to estimate the statistical significance of a correlation when the data are serially correlated, Journal of Climate, 10, 2147, 10.1175/1520-0442(1997)010<2147:AMTETS>2.0.CO;2

Ehret, 2011, Series distance – an intuitive metric to quantify hydrograph similarity in terms of occurrence, amplitude and timing of hydrological events, Hydrology and Earth System Sciences, 15, 877, 10.5194/hess-15-877-2011

Evans, 2002, Identifiability of uncontrolled nonlinear rational systems, Automatica, 38, 1799, 10.1016/S0005-1098(02)00094-8

Ewen, 2011, Hydrograph matching method for measuring model performance, Journal of Hydrology, 408, 178, 10.1016/j.jhydrol.2011.07.038

FAIRMODE, 2010

Fox, 1981, Judging air quality model performance, Bulletin of the American Meteorological Society, 62, 599, 10.1175/1520-0477(1981)062<0599:JAQMP>2.0.CO;2

Frey, 2002, Identification and review of sensitivity analysis methods, Risk Analysis, 22, 553, 10.1111/0272-4332.00039

Gernaey, 2004, Activated sludge wastewater treatment plant modelling and simulation: state of the art, Environmental Modelling and Software, 19, 763, 10.1016/j.envsoft.2003.03.005

Ghelli, 2008, Meteorological applications, Special Issue: Forecast Verification, 15

Gilliam, 2005, The atmospheric model evaluation tool: meteorology module

Giordano, 2012, A fuzzy GIS-based system to integrate local and technical knowledge in soil salinity monitoring, Environmental Modelling and Software, 36, 49, 10.1016/j.envsoft.2011.09.004

Gneiting, 2007, Strictly proper scoring rules, prediction, and estimation, Journal of the American Statistical Association, 102, 359, 10.1198/016214506000001437

Grimm, 2006, A standard protocol for describing individual-based and agent-based models, Ecological Modelling, 198, 115, 10.1016/j.ecolmodel.2006.04.023

Grimm, 2010, The ODD protocol: a review and first update, Ecological Modelling, 221, 2760, 10.1016/j.ecolmodel.2010.08.019

Gupta, 2009, Decomposition of the mean squared error and NSE performance criteria: implications for improving hydrological modelling, Journal of Hydrology, 377, 80, 10.1016/j.jhydrol.2009.08.003

Gupta, 2012, Towards a comprehensive assessment of model structural adequacy, Water Resources Research W08301, 10.1029/2011WR011044

Gy, 1998

Haefner, 2005

Hannachi, 2007, Empirical orthogonal functions and related techniques in atmospheric science: a review, International Journal of Climatology, 27, 1119, 10.1002/joc.1499

Hejazi, 2008, The effect of climate and land use change on flow duration in the Maryland Piedmont region, Hydrological Processes, 22, 4710, 10.1002/hyp.7080

Henriksen, 2009, Harmonised principles for public participation in quality assurance of integrated water resources modelling, Water Resources Management, 23, 2539, 10.1007/s11269-008-9395-9

Holmberg, 1982, On the practical identifiability of microbial models incorporating Michaelis–Menten-type nonlinearities, Mathematical Biosciences, 62, 23, 10.1016/0025-5564(82)90061-X

Houghton-Carr, 1999, Assessment criteria for simple conceptual daily rainfall–runoff models, Hydrological Sciences Journal, 44, 237, 10.1080/02626669909492220

Hutchins, 2006, The model evaluation tool: guidance for applying benchmark criteria for models to be used in river basin management, Large Rivers, 17, 23

IPCC, 2001, 944

Jakeman, 1993, How much complexity is warranted in a rainfall–runoff model?, Water Resources Research, 29, 2637, 10.1029/93WR00877

Jakeman, 1990, Computation of the instantaneous unit hydrograph and identifiable component flows with application to two small upland catchments, Journal of Hydrology, 117, 275, 10.1016/0022-1694(90)90097-H

Jakeman, 2006, Ten iterative steps in development and evaluation of environmental models, Environmental Modelling and Software, 21, 602, 10.1016/j.envsoft.2006.01.004

Kämäri, 2006, Process for benchmarking models: dialogue between water managers and modellers, Large Rivers, 17, 3

Kara, 2012, Time-scale dependence in numerical simulations: assessment of physical, chemical, and biological predictions in a stratified lake at temporal scales of hours to months, Environmental Modelling and Software, 35, 104, 10.1016/j.envsoft.2012.02.014

Keesman, 2011, Uncertainty modelling and analysis of environmental systems: a river sediment yield example

Kelleher, 2011, Ten guidelines for effective data visualization in scientific publications, Environmental Modelling and Software, 26, 822, 10.1016/j.envsoft.2010.12.006

Kitanidis, 1980, Real-time forecasting with a conceptual hydrologic model. 2. Application and results, Water Resources Research, 16, 1034, 10.1029/WR016i006p01034

Kleijnen, 1998, Validation of trace-driven simulation models: a novel regression test, Management Science, 44, 812, 10.1287/mnsc.44.6.812

Kleijnen, 1999, Validation of models: statistical techniques and data availability, 647

Knol, 2010, The use of expert elicitation in environmental health impact assessment: a seven step procedure, Environmental Health, 9, 19, 10.1186/1476-069X-9-19

Kohavi, 1995, A study of cross-validation and bootstrap for accuracy estimation and model selection, vol. 2, 1137

Krause, 2005, Comparison of different efficiency criteria for hydrological model assessment, Advances in Geosciences, 5, 89, 10.5194/adgeo-5-89-2005

Krueger, 2012, The role of expert opinion in environmental modelling, Environmental Modelling and Software, 36, 4, 10.1016/j.envsoft.2012.01.011

Kuhnert, 2006, Comparing raster map comparison algorithms for spatial modeling and analysis, Photogrammetric Engineering and Remote Sensing, 71, 975, 10.14358/PERS.71.8.975

Laio, 2007, Verification tools for probabilistic forecasts of continuous hydrological variables, Hydrology and Earth System Sciences, 11, 1267, 10.5194/hess-11-1267-2007

Lane, 2007, Assessment of rainfall–runoff models based upon wavelet analysis, Hydrological Processes, 21, 586, 10.1002/hyp.6249

Littlewood, 2008, Data time-step dependency of conceptual rainfall streamflow model parameters: an empirical study with implications for regionalisation, Hydrological Sciences Journal, 53, 685, 10.1623/hysj.53.4.685

Liu, 2005, A validation methodology for AI simulation models, vol. 7, 4083

Ljung, 1999

Makler-Pick, 2011, Sensitivity analysis for complex ecological models – a new approach, Environmental Modelling and Software, 26, 124, 10.1016/j.envsoft.2010.06.010

Marsili-Libelli, 2004, Circadian patterns recognition in ecosystems by wavelet filtering and fuzzy clustering

Marsili-Libelli, 2005, Identification of dynamic models for horizontal subsurface constructed wetlands, Ecological Modelling, 187, 201, 10.1016/j.ecolmodel.2005.01.043

Marsili-Libelli, 2008, Water quality modelling for small river basins, Environmental Modelling and Software, 23, 451, 10.1016/j.envsoft.2007.06.008

Marsili-Libelli, 2003, Confidence regions of estimated parameters for ecological systems, Ecological Modelling, 165, 127, 10.1016/S0304-3800(03)00068-1

Marsili-Libelli, 1992, Parameter estimation of ecological models, Ecological Modelling, 62, 233, 10.1016/0304-3800(92)90001-U

Marsili-Libelli, 2006, Control of SBR switching by fuzzy pattern recognition, Water Research, 40, 1095, 10.1016/j.watres.2006.01.011

Matott, 2009, Evaluating uncertainty in integrated environmental models: a review of concepts and tools, Water Resources Research, 45, 10.1029/2008WR007301

Matott, 2012, A benchmarking framework for simulation-based optimization of environmental models, Environmental Modelling and Software, 35, 19, 10.1016/j.envsoft.2012.02.002

Matthews, 2011, Raising the bar? – the challenges of evaluating the outcomes of environmental modelling and software, Environmental Modelling and Software, 26, 247, 10.1016/j.envsoft.2010.03.031

2011, vol. 26 (3), 245

Miles, 2000, End user evaluation of engineering knowledge based systems, Civil Engineering and Environmental Systems, 17, 293, 10.1080/02630250008970287

Moriasi, 2007, Model evaluation guidelines for systematic quantification of accuracy in watershed simulations, Transactions of the ASABE, 50, 885, 10.13031/2013.23153

Moussa, 2010, When monstrosity can be beautiful while normality can be ugly: assessing the performance of event-based flood models, Hydrological Sciences Journal, 55, 1074, 10.1080/02626667.2010.505893

Murphy, 1987, A general framework for forecast verification, Monthly Weather Review, 115, 1330, 10.1175/1520-0493(1987)115<1330:AGFFFV>2.0.CO;2

Murray–Darling Basin Commission, 2000

Nash, 1970, River flow forecasting through conceptual models part i – a discussion of principles, Journal of Hydrology, 10, 282, 10.1016/0022-1694(70)90255-6

Nguyen, 2007, A new approach to testing an integrated water systems model using qualitative scenarios, Environmental Modelling and Software, 22, 1557, 10.1016/j.envsoft.2006.08.005

Nocerino, 2005, Role of sampling devices and laboratory subsampling methods in representative sampling strategies, Environmental Forensics, 6, 35, 10.1080/15275920590913903

Norton, 2008, Algebraic sensitivity analysis of environmental models, Environmental Modelling and Software, 23, 963, 10.1016/j.envsoft.2007.11.007

Norton, 2009

Nossent, 2011, Sobol' sensitivity analysis of a complex environmental model, Environmental Modelling and Software, 26, 1515, 10.1016/j.envsoft.2011.08.010

Olesen, 2005

Oreskes, 1994, Verification, validation, and confirmation of numerical models in the earth sciences, Science, 263, 641, 10.1126/science.263.5147.641

Page, 2012, Eliciting fuzzy distributions from experts for ranking conceptual risk model components, Environmental Modelling and Software, 36, 19, 10.1016/j.envsoft.2011.03.001

Parker, 2002, Progress in integrated assessment and modelling, Environmental Modelling and Software, 17, 209, 10.1016/S1364-8152(01)00059-7

Perrin, 2006, Simple benchmark models as a basis for criteria of model efficiency, Archiv für Hydrobiologie Supplement 161/1–2, Large Rivers, 17, 221

Perrin, 2006, Use of simple rainfall–runoff models as a baseline for the benchmarking of the hydrological component of complex catchment models, Archiv für Hydrobiologie Supplement 161/1–2, Large Rivers, 17, 75

Petersen, 2003, A simplified method to assess structurally identifiable parameters in Monod-based activated sludge models, Water Research, 3, 2893, 10.1016/S0043-1354(03)00114-3

Petersen, B., 2000. Calibration, Identifiability and Optimal Experimental Design of Activated Sludge Models. PhD thesis in Applied Biological Sciences, BIOMATH, University of Gent. URL: http://biomath.ugent.be/publications/download/.

Piuleac, 2010, Ten steps of electrolysis processes by using neural networks, Environmental Modelling and Software, 25, 74, 10.1016/j.envsoft.2009.07.012

Pohjanpalo, 1978, System identifiability based on the power series expansion of the solution, Mathematical Biosciences, 41, 21, 10.1016/0025-5564(78)90063-9

Pontius, 2010, Death to kappa: birth of quantity disagreement and allocation disagreement for accuracy assessment, International Journal of Remote Sensing, 32, 4407, 10.1080/01431161.2011.552923

Pontius, 2004, Useful techniques of validation for spatially explicit land-change models, Ecological Modelling, 179, 445, 10.1016/j.ecolmodel.2004.05.010

Poole, 2000, Inference for deterministic simulation models: the Bayesian melding approach, Journal of the American Statistical Association, 95, 1244, 10.1080/01621459.2000.10474324

Pushpalatha, 2012, A review of efficiency criteria suitable for evaluating low-flow simulations, Journal of Hydrology, 420–421, 171, 10.1016/j.jhydrol.2011.11.055

Raick, 2006, Model complexity and performance: how far can we simplify?, Progress in Oceanography, 70, 27, 10.1016/j.pocean.2006.03.001

Ratto, 2012, Emulation techniques for the reduction and sensitivity analysis of complex environmental models, Environmental Modelling and Software, 34, 1, 10.1016/j.envsoft.2011.11.003

Ravalico, 2010, Management Option Rank Equivalence (MORE) – a new method of sensitivity analysis for decision-making, Environmental Modelling and Software, 26, 171, 10.1016/j.envsoft.2009.06.012

Refsgaard, 2004, Modelling guidelines – terminology and guiding principles, Advances in Water Resources, 27, 71, 10.1016/j.advwatres.2003.08.006

Refsgaard, 2005, Quality assurance in model based water management – review of existing practice and outline of new approaches, Environmental Modelling and Software, 20, 1201, 10.1016/j.envsoft.2004.07.006

Reusser, 2009, Analysing the temporal dynamics of model performance for hydrological models, Hydrology and Earth System Sciences, 13, 999, 10.5194/hess-13-999-2009

Risbey, 1996, Assessing integrated assessments, Climatic Change, 34, 369, 10.1007/BF00139298

Robson, 2004, Three-dimensional modelling of a Microcystis bloom event in the Swan River estuary, Western Australia, Ecological Modelling, 174, 203, 10.1016/j.ecolmodel.2004.01.006

Robson, 2008, Ten steps applied to development and evaluation of process-based biogeochemical models of estuaries, Environmental Modelling and Software, 23, 367, 10.1016/j.envsoft.2007.05.019

Robson, 2010, Using satellite-derived optical data to improve simulation of the 3D light field in a biogeochemical model

Rowan, 2012, Development of a classification and decision-support tool for assessing lake hydromorphology, Environmental Modelling and Software, 36, 86, 10.1016/j.envsoft.2011.09.006

Rykiel, 1996, Testing ecological models: the meaning of validation, Ecological Modelling, 90, 229, 10.1016/0304-3800(95)00152-2

Saltelli, 2010, How to avoid a perfunctory sensitivity analysis, Environmental Modelling and Software, 25, 1508, 10.1016/j.envsoft.2010.04.012

Saltelli, 2000, Sensitivity Analysis

Schoups, 2010, A formal likelihood function for parameter and predictive inference of hydrologic models with correlated, heteroscedastic, and non-Gaussian errors, Water Resources Research, 46, W10531, 10.1029/2009WR008933

Schwarz, 1978, Estimating the dimension of a model, Annals of Statistics, 6, 461, 10.1214/aos/1176344136

Seber, 1989

Seibert, 2001, On the need for benchmarks in hydrological modelling, Hydrological Processes, 15, 1063, 10.1002/hyp.446

Seiller, 2012, Multimodel evaluation of twenty lumped hydrological models under contrasted climate conditions, Hydrology and Earth System Sciences, 16, 1171, 10.5194/hess-16-1171-2012

Seppelt, 2005, “It was an artefact not the result”: a note on systems dynamic model development tools, Environmental Modelling and Software, 20, 1543, 10.1016/j.envsoft.2004.12.004

Seppelt, 2006, Corrigendum to “It was an artefact not the result: a note on systems dynamic, Environmental Modelling and Software, 21, 756, 10.1016/j.envsoft.2006.01.003

Seppelt, 2003, Optimization methodology for land use patterns—evaluation based on multiscale habitat pattern comparison, Ecological Modelling, 168, 217, 10.1016/S0304-3800(03)00138-8

Seppelt, 2009, Challenges of simulating complex environmental systems at the landscape scale: a controversial dialogue between two cups of espresso, Ecological Modelling, 220, 3481, 10.1016/j.ecolmodel.2009.09.009

Shahsavani, 2011, Variance-based sensitivity analysis of model outputs using surrogate models, Environmental Modelling and Software, 26, 723, 10.1016/j.envsoft.2011.01.002

Smiatek, 2012, Implementation and performance analysis of a high resolution coupled numerical weather and river runoff prediction model system for an Alpine catchment, Environmental Modelling and Software, 38, 231, 10.1016/j.envsoft.2012.06.001

Sorooshian, 1980, Stochastic parameter estimation procedures for hydrologic rainfall–runoff models: correlated and heteroscedastic error cases, Water Resources Research, 16, 430, 10.1029/WR016i002p00430

Stow, 2009, Skill assessment for coupled biological/physical models of marine systems, Journal of Marine Systems, 76, 4, 10.1016/j.jmarsys.2008.03.011

STOWA/RIZA, 1999

Strang, 1996

Thunis, 2012, A tool to evaluate air quality model performances in regulatory applications, Environmental Modelling and Software, 38, 220, 10.1016/j.envsoft.2012.06.005

Tompa, 2000, Perceptually based image comparison, 489

Torrence, 1998, A practical guide to wavelet analysis, Bulletin of the American Meteorological Society, 79, 61, 10.1175/1520-0477(1998)079<0061:APGTWA>2.0.CO;2

USEPA, 2002

USEPA, 2009

van der Sluijs, 2004

van der Sluijs, 2005, Combining quantitative and qualitative measures of uncertainty in model-based environmental assessment: the NUSAP system, Risk Analysis, 25, 481, 10.1111/j.1539-6924.2005.00604.x

Vanrolleghem, 1996, Identification of biodegradation models under model and data uncertainty, Water Science and Technology, 33, 91, 10.1016/0273-1223(96)00192-8

Voinov, 2010, Modelling with stakeholders, Environmental Modelling and Software, 25, 1268, 10.1016/j.envsoft.2010.03.007

Vrana, 2012, A group agreement-based approach for decision making in environmental issues, Environmental Modelling and Software, 36, 99, 10.1016/j.envsoft.2011.12.007

Vrugt, 2009, Equifinality of formal (DREAM) and informal (GLUE) Bayesian approaches in hydrologic modeling?, Stochastic Environmental Research and Risk Assessment, 23, 1011, 10.1007/s00477-008-0274-y

Wagener, 2007, Numerical and visual evaluation of hydrological and environmental models using the Monte Carlo analysis toolbox, Environmental Modelling and Software, 22, 1021, 10.1016/j.envsoft.2006.06.017

Wagener, 2001, A framework for development and application of hydrological models, Hydrology and Earth System Sciences, 5, 13, 10.5194/hess-5-13-2001

Wagener, 2003, Towards reduced uncertainty in conceptual rainfall–runoff modelling: dynamic identifiability analysis, Hydrological Processes, 17, 455, 10.1002/hyp.1135

Walter, 1997, Identification of Parametric Models from Experimental Data

Walter, 1982

Wealands, 2005, Quantitative comparison of spatial fields for hydrological model assessment – some promising approaches, Advances in Water Resources, 28, 15, 10.1016/j.advwatres.2004.10.001

Weijs, 2010, Why hydrological predictions should be evaluated using information theory, Hydrology and Earth System Sciences, 14, 2545, 10.5194/hess-14-2545-2010

Willmott, 1981, On the validation of models, Physical Geography, 2, 184, 10.1080/02723646.1981.10642213

Yang, 2011, Convergence and uncertainty analyses in Monte-Carlo based sensitivity analysis, Environmental Modelling and Software, 26, 444, 10.1016/j.envsoft.2010.10.007

Yilmaz, 2008, A process-based diagnostic approach to model evaluation: application to the NWS distributed hydrologic model, Water Resources Research, 44, W09417, 10.1029/2007WR006716

Young, 1980, An instrumental variable method for model order identification, Automatica, 16, 281, 10.1016/0005-1098(80)90037-0

Young, 2011