A penny for your thoughts: a survey of methods for eliciting beliefs

Allen, F. (1987). Discovering personal probabilities when utility functions are unknown. Management Science, 33(4), 542–544. Andersen, S., Fountain, J., Harrison, G. W. and Rutström, E. E. (2014). Estimating Subjective Probabilities. Journal of Risk and Uncertainty, 48(3), 207–220. Armantier, O., & Treich, N. (2009). Subjective probabilities in games: A solution to the overbidding puzzle. International Economic Review, 50(4), 1079–1102. Armantier, O., & Treich, N. (2013). Eliciting beliefs: Proper scoring rules, incentives, stakes and hedging. European Economic Review, 62, 17–40. Artinger, F., Exadaktylos, F., Koppel, H., & Sääksvuori, L. (2010). Applying Quadratic Scoring Rule transparently in multiple choice settings: a note. Technical report, Jena Economic Research Paper. Beach, L., & Phillips, L. (1967). Subjective probabilities inferred from estimates and bets. Journal of Experimental Psychology, 75(3), 354–359. Beach, L. R., & Wise, J. A. (1969). Subjective probability revision and subsequent decisions. Journal of Experimental Psychology, 81(3), 561–565. Becker, G., DeGroot, M., & Marschak, J. (1964). Measuring utility by a single-response sequential method. Behavioral Science, 9(3), 226. Berg, J. E., Rietz, T. A., & Dickhaut, J. W. (2008). On the performance of the lottery procedure for controlling risk preferences. Handbook of Experimental Economics Results, 1, 1087–1097. Blanco, M., Engelmann, D., Koch, A. K., & Normann, H.-T. (2010). Belief elicitation in experiments: Is there a hedging problem? Experimental Economics, 13(4), 412–438. Brier, G. W. (1950). Verification of forecasts expressed in terms of probability. Monthly Weather Review, 78(1), 1–3. Cervera, J. L., & Muñoz, J. (1996). Proper scoring rules for fractiles. In J. O. Berger, A. P. Dawid, & A. F. M. Smith (Eds.), Bayesian statistics 5 (pp. 513–519). Oxford: Oxford University Press. Costa-Gomes, M. A., Huck, S. and Weizsacker, G. (2012). Beliefs and actions in the trust game: creating instrumental variables to estimate the causal effect. WZB Discussion Paper, 2012–302. Croson, R. T. A. (2000). Thinking like a game theorist: Factors affecting the frequency of equilibrium play. Journal of Economic Behavior & Organization, 41(3), 299–314. Dawes, R. M., McTavish, J., & Shaklee, H. (1977). Behavior, communication, and assumptions about other people’s behavior in a commons dilemma situation. Journal of Personality and Social Psychology, 35(1), 1. De Finetti, B. (1965). Methods for discriminating levels of partial knowledge concerning a test item. British Journal of Mathematical and Statistical Psychology, 18(1), 87–123. De Finetti, B. (1970). Logical foundations and measurement of subjective probability. Acta Psychologica, 34, 129–145. De Finetti, B. (1974). Theory of probability (Vol. 1). New York: Wiley. Delavande, A., Giné, X., & McKenzie, D. (2011). Measuring subjective expectations in developing countries: A critical review and new evidence. Journal of Development Economics, 94(2), 151–163. Ducharme, W., & Donnell, M. (1973). Intrasubject comparison of four response modes for “subjective probability” assessment. Organizational Behavior and Human Performance, 10, 108–117. Engelmann, D., & Strobel, M. (2000). The false consensus effect disappears if representative information and monetary incentives are given. Experimental Economics, 260(2000), 241–260. Erev, I., Bornstein, G., & Wallsten, T. (1993). The negative effect of probability assessments on decision quality. Organizational Behavior and Human Decision Processes, 55, 78–94. Fischer, G. W. (1982). Scoring-rule feedback and the overconfidence syndrome in subjective probability forecasting. Organizational Behavior and Human Performance, 29(3), 352–369. Fountain, J., & Harrison, G. W. (2011). What do prediction markets predict? Applied Economics Letters, 18(3), 267–272. Friedman, D. (1983). Effective scoring rules for probabilistic forecasts. Management Science, 29(4), 447–454. Gächter, S., & Renner, E. (2010). The effects of (incentivized) belief elicitation in public goods experiments. Experimental Economics, 13(3), 364–377. Garthwaite, P. H., Kadane, J. B., & O’Hagan, A. (2005). Statistical methods for eliciting probability distributions. Journal of the American Statistical Association, 100(470), 680–701. Gigerenzer, G., & Hoffrage, U. (1995). How to improve bayesian reasoning without instruction: Frequency formats. Psychological Review, 102(4), 684–704. Gneiting, T., & Raftery, A. E. (2007). Strictly proper scoring rules, prediction, and estimation. Journal of the American Statistical Association, 102(477), 359–378. Good, I. (1952). Rational decisions. Journal of the Royal Statistical Society. Series B, 14(1), 107–114. Grether, D. (1981). Financial incentive effects and individual decision-making. California Institute of Technology, Working Paper 401. Guerra, G., & Zizzo, D. J. (2004). Trust responsiveness and beliefs. Journal of Economic Behavior & Organization, 55(1), 25–30. Hao, L., & Houser, D. (2012). Belief elicitation in the presence of novice participants: An experimental study. Journal of Risk and Uncertainty, 2, 161–180. Harrison, G. W., Martínez-Correa, J., & Swarthout, J. (2014). Eliciting subjective probabilities with binary lotteries. Journal of Economic Behavior and Organization, 101, 128–140. Harrison, G. W., Martínez-correa, J., Swarthout, J. T., & Ulm, E. R. (2013a). Scoring rules for subjective probability distributions. Manuscript, Georgia State University. Harrison, G. W., Martinez-Correa, J., & Swarthout, T. (2013b). Inducing risk neutral preferences with binary lotteries: A reconsideration. Journal of Economic Behavior and Organization, 94, 145–159. Harrison, G. W., & Rutström, E. E. (2009). Expected utility theory and prospect theory: One wedding and a decent funeral. Experimental Economics, 12(2), 133–158. Heinemann, F., Nagel, R., & Ockenfels, P. (2009). Measuring strategic uncertainty in coordination games. Review of Economic Studies, 76, 181–221. Hoffmann, T. (2013). The Effect of Belief Elicitation on Game Play. Manuscript, Mannheim University. Hollard, G., Massoni, S. and Vergnaud, J. (2010). Subjective beliefs formation and elicitation rules : Experimental evidence. Centre d’Economie de la Sorbonne Working Paper, 2010.88. Holt, C. (2006). Markets, games and strategic behavior. Boston: Pearson/Addison-Wesley. Holt, C. A., & Laury, S. (2002). Risk aversion and incentive effects. The American Economic Review, 92(5), 1644. Hossain, T., & Okui, R. (2013). The binarized scoring rule. The Review of Economic Studies, 80(3), 984–1001. Huck, S., & Weizsäcker, G. (2002). Do players correctly estimate what others do?: Evidence of conservatism in beliefs. Journal of Economic Behavior & Organization, 47(1), 71–85. Hurley, T., & Shogren, J. (2005). An experimental comparison of induced and elicited beliefs. Journal of Risk and Uncertainty, 30(2), 169–188. Hurley, T. T. M., Peterson, N., & Shogren, J. J. F. (2007). Belief elicitation: An experimental comparison of scoring rule and prediction methods. Manuscript, University of Minnesota. Jaffray, J., & Karni, E. (1999). Elicitation of subjective probabilities when the initial endowment is unobservable. Journal of Risk and Uncertainty, 8, 5–20. Jenkinson, D. (2005). The elicitation of probabilities—A review of the statistical literature. Manuscript, University of Sheffield. Jensen, F. A., & Peterson, C. R. (1973). Psychological effects of proper scoring rules. Organizational Behavior and Human Performance, 9(2), 307–317. Jose, V. R. R., & Winkler, R. L. (2009). Evaluating quantile assessments. Operations Research, 57(5), 1287–1297. Kadane, J., & Winkler, R. (1988). Separating probability elicitation from utilities. Journal of the American Statistical Association, 83(402), 357–363. Karni, E. (2009). A mechanism for eliciting probabilities. Econometrica, 77(2), 603–606. Karni, E., & Safra, Z. (1995). The impossibility of experimental elicitation of subjective probabilities. Theory and Decision, 38, 313–320. Koessler, F., Noussair, C., & Ziegelmeyer, A. (2012). Information aggregation and belief elicitation in experimental parimutuel betting markets. Journal of Economic Behavior & Organization, 83(2), 195–208. Kothiyal, A., Spinu, V., & Wakker, P. (2011). Comonotonic proper scoring rules to measure ambiguity and subjective beliefs. Journal of Multi-Criteria Decision Analysis, 101, 101–113. Krawczyk, M. (2011). Overconfident for real? Proper scoring for confidence intervals. Manuscript, University of Warsaw. Lambert, N., Pennock, D., & Shoham, Y. (2008). Eliciting properties of probability distributions: The highlights. ACM SIGecom Exchanges, 7(3), 1–5. Lindley, D. V., Tversky, A., & Brown, R. V. (1979). On the reconciliation of probability assessments. Journal of the Royal Statistical Society. Series A (General), 142(2), 146. Lipkus, I., Samsa, G., & Rimer, B. (2001). General performance on a numeracy scale among highly educated samples. Medical Decision Making, 21, 37–44. Machina, M. J., & Schmeidler, D. (1992). A more robust definition of subjective probability. Econometrica, 60(4), 745–780. Manski, C. (2002). Identification of decision rules in experiments on simple games of proposal and response. European Economic Review, 46, 880–891. Manski, C. (2004). Measuring expectations. Econometrica, 72(5), 1329–1376. Manski, C. F. (2006). Interpreting the predictions of prediction markets. Economics Letters, 91(3), 425–429. Matheson, J., & Winkler, R. (1976). Scoring rules for continuous probability distributions. Management Science, 22(10), 1087–1096. McCarthy, J. (1956). Measures of the value of information. Proceedings of the National Academy of Sciences of the United States of America, 42(9), 654–655. McKelvey, R., & Page, T. (1990). Public and private information: An experimental study of information pooling. Econometrica, 58(6), 1321–1339. Miller, N., Resnick, P., & Zeckhauser, R. (2005). Eliciting informative feedback: The peer-prediction method. Management Science, 51(9), 1359–1373. Nyarko, Y., & Schotter, A. (2002). An experimental study of belief learning using elicited beliefs. Econometrica, 70(3), 971–1005. Offerman, T., & Palley, A. B. (2013). Lossed in Translation : An Off-the-Shelf Method to Recover Probabilistic Beliefs from Loss-Averse Agents. Manuscript, University of Amsterdam. Offerman, T., Sonnemans, J., & Schram, A. (1996). Value orientations, expectations and voluntary contributions in public goods. The Economic Journal, 106(437), 817–845. Offerman, T., Sonnemans, J., Van de Kuilen, G., & Wakker, P. P. (2009). A truth serum for non-bayesians. Review of Economic Studies, 76(4), 1461–1489. Palfrey, T. R., & Wang, S. W. (2009). On eliciting beliefs in strategic games. Journal of Economic Behavior & Organization, 71(2), 98–109. Phillips, L. D., & Edwards, W. (1966). Conservatism in a simple probability inference task. Journal of Experimental Psychology, 72(3), 346–354. Prelec, D. (2004). A bayesian truth serum for subjective data. Science, 306, 462–466. Price, P. (1998). Effects of a relative-frequency elicitation question on likelihood judgment accuracy: The case of external correspondence. Organizational Behavior and Human Decision Processes, 76(3), 277–297. Ramsey, F. (1926). Truth and Probability. In R. B. Braithwaite (ed.), The foundations of mathematics and other logical essays (pp. 156–198). New York (1931): Harcourt. Roby, T. B. (1964). Belief states: A preliminary empirical study. Technical Documentary Report, Decision Sciences Laboratory. Rutström, E. E., & Wilcox, N. T. (2009). Stated beliefs versus inferred beliefs: A methodological inquiry and experimental test. Games and Economic Behavior, 67(2), 616–632. Savage, L. J. (1954). The foundation of statistics. New York: Wiley. Savage, L. J. (1971). Elicitation of personal probabilities and expectations. Journal of the American Statistical Association, 66(336), 783–801. Schervish, M. (1989). A general method for comparing probability assessors. The Annals of Statistics, 17(4), 1856–1879. Schlag, K. H. and Tremewan, J. (2014). Simple belief elicitation. SSRN Working paper 2449224. Schlag, K. H., & van der Weele, J. J. (2012). Incentives for interval elicitation. Manuscript, Vienna University. Schlag, K. H., & van der Weele, J. J. (2013). Eliciting probabilities, means, medians, variances and covariances without assuming risk neutrality. Theoretical Economics Letters, 03(1), 38–42. Schmalensee, R. (1976). An experimental study of expectation formation. Econometrica, 44(1), 17–41. Schotter, A., & Trevino, I. (2014). Belief elicitation in the laboratory. Annual Review of Economics, 6, 103–128. Schum, D. A., Goldstein, I. L., Howell, W. C., & Southard, J. F. (1967). Subjective probability revisions under several cost-payoff arrangements. Organizational Behavior and Human Performance, 2, 84–104. Seghers, R. C., Fryback, D. G., & Goodman, B. C. (1973). Relative variance preferences in a choice-among-bets paradigm. Technical report, DTIC Document. Selten, R. (1998). Axiomatic characterization of the quadratic scoring rule. Experimental Economics, 62, 43–62. Selten, R., Sadrieh, A., & Abbink, K. (1999). Money does not induce risk neutral behavior, but binary lotteries do even worse. Theory and Decision, 46, 211–249. Shuford, E., Albert, A., & Massengill, H. E. (1966). Admissible probability measurement procedures. Psychometrika, 31(2), 125–145. Smith, A. (2013). Estimating the causal effect of beliefs on contributions in repeated public good games. Experimental Economics, 16(3), 414–425. Smith, C. (1961). Consistency in statistical inference and decision. Journal of the Royal Statistical Society. Series B, 23(1), 1–37. Sonnemans, J., & Offerman, T. T. T. (2001). Is the quadratic scoring rule behaviorally incentive compatible?. Manuscript, University of Amsterdam. Staël von Holstein, C.-A. S. (1970). Measurement of subjective probability. Acta Psychologica, 34, 146–159. Toda, M. (1951). Measurement of intuitive probability by a method of game. Japanese Journal of Psychology, 22, 29–40. Toda, M. (1963). Measurement of subjective probability distribution. Report No. 3(3), State College, Pennsylvania, Institute for Research, Division of Mathematical Psychology. Trautmann, S. T. and van de Kuilen, G. (2014). Belief elicitation: A horse race among truth serums. The Economic Journal. doi:10.1111/ecoj.12160. Tversky, A., & Koehler, D. J. (1994). Support theory: A nonextensional representation of subjective probability. Psychological Review, 101(4), 547. Vlek, C. (1973a). The fair betting game as an admissible procedure for assessment of subjective probabilities. British Journal of Mathematical and Statistical Psychology, 26(1), 18–30. Vlek, C. A. J. C. (1973b). Coherence of human judgment in a limited probabilistic environment. Organizational Behavior and Human Performance, 9(460–481), 460–481. Wang, H., Dash, D., & Druzdzel, M. J. (2002). A method for evaluating elicitation schemes for probabilistic models. IEEE Transactions on Systems, Man, and Cybernetics, Part B, 32(1), 38–43. Wang, S. W. (2011). Incentive effects: The case of belief elicitation from individuals in groups. Economics Letters, 111(1), 30–33. Whitcomb, K. M., Önkal, D., Benson, P. G., & Curley, S. P. (1993). An evaluation of the reliability of probability judgments across response modes and over time. Journal of Behavioral Decision Making, 6(4), 283–296. Wilcox, N. T. and Feltovich, N. (2000). Thinking like a game theorist: Comment. University of Houston Department of Economics working paper. Winkler, R. (1996). Scoring rules and the evaluation of probabilities. Test, 5(1), 1–60. Winkler, R., & Murphy, A. (1968). ”Good” probability assessors. Journal of Applied Meteorology, 7, 751. Winkler, R., & Murphy, A. (1970). Nonlinear utility and the probability score. Journal of Applied Meteorology, 9, 143–148. Winkler, R., & Murphy, A. (1979). The use of probabilities in forecasts of maximum and minimum temperatures. The Meteorological Magazine, 108(1288), 317–329. Wolfers, J. and Zitzewitz, E. (2006). Interpreting prediction market prices as probabilities. NBER Working Paper 12200. Yaniv, I., & Foster, D. (1995). Graininess of judgement under uncertainty: An accuracy-informativeness trade-off. Journal of Experimental Psychology: General, 124(4), 424–432. Yaniv, I., & Foster, D. (1997). Precision and accuracy of judgmental estimation. Journal of Behavioral Decision Making, 10, 21–32. Zizzo, D. J. (2009). Experimenter demand effects in economic experiments. Experimental Economics, 13(1), 75–98.