In Spring 2021 Dr. Worthy will be teaching a graduate seminar on decision-making on Fridays (PSYC 689). Syllabus
Below is the reading list for the course:
1-January 29 Probability Learning 1
Shanks, D.R., Tunney, R.J., & McCarthy, J.D. (2002). A re-examination of probability matching and rational choice. Journal of Behavioral Decision-Making, 15, 233-250. .pdf
Bower, G.H. (1994). A turning point in mathematical learning theory. Psychological Review, 101(2), 290-300. .pdf
Vulkan, N. (2000). An economist’s perspective on probability matching. Journal of Economic Surveys, 14(1), 101-118. .pdf
Optional:
Estes, W.K. (1950). Toward a statistical theory of learning. Psychological Review, 57(2), 94-107..pdf
2-February 5 Probability Learning 2
Kahneman, D., & Tversky, A. (1972). Subjective probability: A judgment of representativeness. Cognitive Psychology, 3, 430-454. .pdf
Estes, W.K. (1976). The cognitive side of probability learning. Psychological Review, 83(1), 37-64. .pdf
Bar-Hillel, M. (1980). The base-rate fallacy in probability judgments. Acta Psychologica, 44, 211-233. .pdf
Optional:
Don, H.J., Otto, A.R., Cornwall, A.C., Davis, T., & Worthy, D.A. (2019). Learning reward frequency over reward probability: A tale of two learning rules. Cognition, 193, 104042. .pdf
Estes, W.K. (1976). Some functions of memory in probability learning and choice behavior. Psychology of Learning and Motivation. .pdf
3-February 12 Framing
Tversky, A., & Kahneman, D. (1981). The framing of decisions and the psychology of choice. Science, 211, 453-458. .pdf
De Martino, B., Kumaran, D., Seymour, B., & Dolan, R.J. (2006). Frames, biases, and rational decisión-making in the human brain. Science, 313, 684-687. .pdf
Levin, I.P., Schneider, S.L., & Gaeth, G.J. (1998). All frames are not created equal: A typology and critical analysis of framing effects. Organizational Behavior and Human Decision Processes, 76, 149-188. .pdf
4-February 19 Judgment and Confidence
Einhorn, H.J., & Hogarth, R.M. (1978). Confidence in Judgment: Persistence of the illusion of validity. Psychological Review, 85 (5), 395-416. .pdf
Dawes, R.M., Faust, D., & Meehl, P.E. (1989). Clinical versus actuarial judgment. Science, 243, 1668-1674. .pdf
Dawes, R.M. (1979). The robust beauty of improper linear models in decision-making. American Psychologist, 34, 571-582. .pdf
Optional:
Grove, W.M., Zald, D.H., Lebow, B.S., Snitz, B.E., & Nelson, C. (2000). Clinical versus mechanical prediction: A meta-analysis. Psychological Assessment, 12(1), 19-30. .pdf
5-February 26 Intertemporal Choice
McClure, S.M., Laibson, D.I., Loewenstein, G., & Cohen, J.D. (2004). Separate neural systems value immediate and delayed monetary rewards. Science, 306, 503-507. .pdf
Otto, A.R., Markman, A.B., && Love, B.C. (2012). Taking more now: The optimality of impulsive choice hinges on environment structure. Social Psychological and Personality Sciences 3(2), 131-138. .pdf
Peters, J., & Buchel, C. (2010). Episodic future thinking reduces reward delay discounting through and enhancement of prefrontal-mediotemporal interactions. Neuron, 66(1), 138-148. .pdf
Optional:
Figner, B., Knoch, D., Johnson, E.J., Krosch, A.R., Lisanby, S.H., Feher, E., et al. (2010). Lateral prefrontal cortex and self-control in intertemporal choice. Nature Neuroscience, 13 (5), 538-539. .pdf
6-March 5 Memory
Gershman, S.J., & Daw, N.D. (2017). Reinforcement learning and episodic memory in humans and animals: An integrative framework. Annual Review of Psychology, 68, 101-128. .pdf
Mattar, M., & Daw, N.D. (2018). Prioritized memory access explains planning and hippocampal replay. Nature Neuroscience. .pdf
Shadlen, M.N., & Shohamy, D. (2016). Decision making and sequential sampling from memory. Neuron, 90, 927-939. .pdf
7-March 12 Habitual versus Goal-Directed Choice
Rangel, A., Camerer, C., & Montague, P.R. (2008). A framework for studying the neurobiology of value-based decision making. Nature Reviews Neuroscience, 9, 545-556. .pdf
Linnebank, F.E., Kindt, M., & de Wit, S. (2018). Investigating the balance between goal-directed and habitual control in experimental and real-life settings. Learning & Behavior, 46(3), 306-319. .pdf
Chen, Z., Veling, H., Diksterhuis, A. & Holland, R.W. (2016). How does not responding to appetitive stimuli cause devaluation: Evaluative conditioning or response inhibition? Journal of Experimental Psychology: General, 145(12), 1687-1701. .pdf
Optional:
Sloman, S.A. (1996). The empirical case for two systems of reasoning. Psychological Bulletin, 119, 3-22. .pdf
Daw, N.D., Niv, Y., & Dayan, P. (2005). Uncertainty-based competition between prefrontal and dorsolateral striatal systems for behavioral control. Nature Neurosciences, 8, 1704-1711. .pdf
March 19 – Spring Break
8-March 26 Exploration/Exploitation
Daw, N.D., O’Doherty, J.P., Dayan, P., Seymour, B., & Dolan, R.J. (2006). Cortical substrates for exploratory decisions in humans. Nature, 441, 876-879. .pdf
Otto, A.R., Knox, W.B., Markman, A.B., & Love, B.C. (2014). Physiological and behavioral signatures of reflective exploratory choice. Cognitive, Affective, & Behavioral Neuroscience, 14(4), 1167-1183. .pdf
Wilson, R.C., Geana, A., White, J.M., Ludvig, E.A., & Cohen, J.D. (2014). Humans use directed and random exploration to solve the explore-exploit dilemma. Journal of Experimental Psychology: General, 143(6). 2074-2081. .pdf
Optional:
Cohen, J.D., McClure, S.M., & Yu, A.J. (2007). Should I stay or should I go? How the human brain manages the trade-off between exploitation and exploration. Philosophical Transactions of the Royal Society B, 362, 933-942. .pdf
Frank, M.J., Doll, B.B., Oas-Terpstra, J., & Moreno, F. (2009). Prefrontal and striatal dopaminergic genes predict individual differences in exploration and exploitation. Nature Neuroscience, 12, 1062-1068. .pdf
April 2 Reading Day, no classes
9-April 9 “Real-world” Decisions
Schulz, E., Bhui, R., Love, B.C., Brier, B., Todd, M.T., & Gershman, S.J. (2019). Structured, uncertainty-driven exploration in real world consumer choice. Proceedings of the National Academy of Sciences, 116 (28), 13903-13908. .pdf
Thorstad, R., & Wolff, P. (2018). A big data analysis of the relationship between future thinking and decision-making. Proceedings of the National Academy of Sciences, 115(8), 1740-1748. .pdf
Neiman, T., & Loewenstein, Y. (2011). Reinforcement learning in professional basketball players. Nature Communications, 2(1), 569. .pdf
Optional:
Otto, A.R., Fleming, S.M., & Glimcher, P.W. (2016). Unexpected but incidental positive outcomes predict real-world gambling. Psychological Science, 277(3), 299-311. .pdf
10-April 16 Emotion
Bechara, A. (2004). The role of emotion in decision-making: Evidence from neurological patients with orbitofrontal damage. Brain & Cognition, 55, 30-40. .pdf
Seymour, B., & Dolan, R. (2008). Emotion, Decision Making, and the Amygdala. Neuron, 58, 662-671. .pdf
Wu, Y., van Dijk, E., & Clark, L. (2015). Near-wins and near-losses in gambling: A behavioral and facial EMG study. Psychophysiology, 52(3), 359-366. .pdf
Optional:
Loewenstein, G.F., Weber, E.U., Hsee, C.K., & Welch, N. (2001). Risk as feelings. Psychological Bulletin, 127, 267-286. .pdf
11-April 23 Effort
Westbrook, A., Bosch, R., van den Maata, J.I., Hofmans, L., Papadopetraki, D., Cools, R., & Frank, M.J. (2020). Dopamine promotes cognitive effort by biasing the benefits versus costs of cognitive work. Science, 367(6484), 1362-1366. .pdf
Otto, A.R., & Vassena, E. (2020). It’s all relative: Reward-induced cognitive control modulation depends on context. Journal of Experimental Psychology: General. .pdf
Chong, T.T.-J., Apps, M., Giehl, K., Sillence, A., Grima, L.L., & Husain, M. (2017). Neurocomputational mechanisms underlying subjective valuation of effort costs. PLOS Biology, 15(2), e1002598. .pdf
Older readings from our journal club from 2017-2019
Upcoming
Estes, W.K. (1976). Some functions of memory in probability learning and choice behavior. Psychology of Learning and Motivation. .pdf
Wilson, R.C., Shenhav, A., Straccia, M., & Cohen, J.D. (2019). The eighty five percent rule for optimal learning. Nature Communications. .pdf
Lohse, K.R., Miller, M.W., Daou, M., Valerius, W., & Jones, M. (2020). Dissociating the contributions of reward-prediction errors to trial-level adaptation and long-term learning. Biological Psychology. .pdf
Radulescu, A., Niv, Y., & Ballard, I. (2019). Holistic reinforcement learning: The role of structure and attention. Trends in Cognitive Sciences, 23 (4), 278-292. .pdf
Schedule for Fall 2019:
12/3/19 Chapter 5 of Computational Modeling book
11/19/19 Chapter 4 of Computational Modeling book – sections 4.3 – 4.5
11/5/19 Leong, Y.C., Radulescu, A., Daniel, R., DeWoskin, V., & Niv, Y. (2017). Dynamic Interaction between Reinforcement Learning and Attention in Multidimensional Environment. Neuron, 93, 451-463. .pdf
The Leong et al., article above is the target article. One thing we will discuss is how (or whether) the model presented in the above article differs from Kruschke’s Alcove model. They may be very similar, but come from different perspectives – computational neuroscience (Leong) versus cognitive psychology (Kruschke), and focus on slightly different paradigms – decision-making versus category learning.
Kruschke, J.K. (1992). ALCOVE: An exemplar-based connectionist model of category learning. Psychological Review, 99(1), 22-44. .pdf
11/4/19 Journal club presentation in Cognoscenti on Compensation, Maintenance, and Reserve.
Cabeza et al. (2018). Maintenance, reserve, and compensation: the cognitive neuroscience of healthy ageing. Nature Reviews Neuroscience. .pdf
Stern et. al (2018). Whitepaper: Defining and investigating cognitive reserve, brain reserve, and brain maintenance. Alzheimer’s & Dementia. .pdf
Note: Why does the Stern et al. paper above have no references? Why would a journal publish a “Review Article” without referencing any of the papers being reviewed?
10/29/19
Chapter 4 of Computational Modeling book – sections 4.3 – 4.5
10/22/19
Lee, S., Gold, J.I., & Kable, J.W. (2019). The Human as Delta-Rule Learner. Decision. .pdf
10/15/19
Chapter 4 of Computational Modeling book – sections 4.1 – 4.2
10/8/19
Inkster, A.B., Mitchell, C.J., Schlegelmilch, R., & Wills, A.J. (2020). Effect of a context shift on the inverse base-rate effect. Open Journal of Experimental Psychology and Neuroscience. .pdf
*note we will read the above journal article instead of the thesis below, because it is not as long and we only have so much time
Inkster, A. (2019). Attention, context, and the inverse base rate effect. Doctoral Dissertation. University of Plymouth. .pdf
Don, H.J., Beesley, T., & Livesey, E.J. (2019). Learned predictiveness models predict opposite attention biases in the inverse Base-Rate Effect. .pdf
Possibly model IBRE data.
10/1/19
Chapter 3 of Computational Modeling book
9/24/19
Hartley, C.A., & Phelps, E.A. (2012). Anxiety and decision-making. Biological Psychiatry, 72, 113-118. .pdf
Will possibly model some anxiety-decision-making data
9/17/19
Chapter 2 of Computational Modeling
Iowa Gambling Task or other RL task simulation
9/10/19
Bornstein, A..M., Khaw, M.W., Shohamy, D.., & Daw, N.D. (2017). Reminders of past choices bias decisions for reward in humans. Nature Communications, 8:15958. .pdf
Notes on Gershman & Daw, 2017, and Bornstein et al., 2017
Gershman, S.J., & Daw, N.D. (2017). Reinforcement learning and episodic memory in humans and animals: An Integrative Framework. Annual Review of Psychology, 68, 101-28. .pdf
9/3/19
Chapter 1 of Computational Modeling of Cognition and Behavior
Iowa Gambling Task Modeling
8/27/19
Hotaling, J.M., Jarvstad, A., Donkin, C., & Newell, B.R. (in press). How to change the weight of rare events in decisions from experience. Psychological Science. .pdf
9/3/19
Chapter 1 of Computational Modeling of Cognition and Behavior
Iowa Gambling Task Modeling
Other articles to read…
Newell, A. (1973). You can’t play 20 questions with nature and win. In W.G. Chase (ed.) Visual Information Processing, New York: Academic Press. .pdf
Kosslyn, S.M. (2006). You can play 20 questions with nature and win: Categorical versus coordinate spatial relations as a case study. Neuropsychologia, 44, 1519-1523. .pdf
Katahira, K. (2018). The statistical structures of reinforcement learning with asymmetric value updates. Journal of Mathematical Psychology, 87, 31-45. .pdf
Lefebvre, G., Lebreton, M., Meyniel, F., Bourgeois-Gironde, S., & Palmintieri, S. (2017). Behavioral and neural characterization of optimistic reinforcement learning. Nature Human Behavior. .pdf Supp
Notes on Lefebvre 2017 and Katahira 2018, which present conflicting explanations for a “positivity bias” for positive versus negative prediction errors.
Estes, W.K. (1964). All-or-none processes in learning and retention. American Psychologist, 19(1), 16-25. Notes .pdf
Platt, J.R. (1964). Strong inference. Science 146, 347-353. .pdf – Nice philosophy of science paper on how not all scientific methods are equal. “What I am saying is that, in numerous areas that we call science, we have come to like our habitual ways, and our studies that can be continued indefinitely. We measure, we define, we compute, we analyze, but we do not exclude. And this is not the way to use our minds most effectively or to make the fastest progress in solving scientific questions.”
I highly recommend Simon Farrell and Stephan Lewandowsky’s book, Computational Modeling of Cognition and Behavior.
R code for the book is available here.
A folder with the code from the book and .pdf files for each chapter will be distributed via email.
2017-2018 Readings (reverse ordered):
5/11/18 Gonzalez, C., & Dutt, V. (2011). Instance-based learning: Integrating sampling and repeated decision from experience. Psychological Review. .pdf
5/4/18 Stewart, N., Chater, N., & Brown, G.D.A. (2006). Decision by sampling. Cognitive Psychology, 53, 1-26. .pdf
4/27/18 Kool, W., Gershman, S.J., & Cushman, F.A. (2017). Cost-benefit arbitration between multiple reinforcement-learning systems. Psychological Science, 28, 1321-1333. .pdf
4/13/18 Suppes, P. Estes’ Statistical Learning Theory: Past, Present, and Future. A.F. Healy, S.M. Kosslyn, R.M. Shiffrin (Eds.), From Learning Theory to Connectionist Theory: Essays in Honor of William K. Estes, Vol.1. Hillsdale, NJ: Lawrence Erlbaum, 1992, pp. 1-20. .pdf
4/6/18 Rigoli, F., Mathys, C., Friston, K.J., Dolan, R.J. A unifying Bayesian account of contextual effects in value-based choice. PLOS Computational Biology. .pdf
3/23/18 Miller, K.J., Botvinick, M.M., & Brody, C.D. (2018). Value representations in orbitofrontal cortex drive learning, but not choice. bioRxiv. .pdf
Roesch, M.R., Esber, G. R., Li, J., Daw, N.D., & Schoenbaum, G. (2012). Suprise! Neural correlates of Pearce-Hall and Rescorla-Wagner coexist within the brain. European Journal of Neuroscience, 35, 1190-1200. .pdf
3/9/18 Sutton, R.S., & Barto, A.G. (1981). Toward a modern theory of adaptive networks: Expectation and prediction. Psychological Review, 88, 135-170. .pdf
3/2/18 Pearce, J.M., & Hall, G. (1980). A model for Pavlovian learning: Variations in the effectiveness of conditioned but not of unconditioned stimuli. Psychological Review, 87, 532-552. .pdf
Diederen, K.M.J. , Spencer, T., Vestergaard, M.D., Fletcher, P.C., & Schultz, S. (2016). Adaptive prediction error coding in the human midbrain and striatum facilitates behavioral adaptation and learning efficiency. Neuron, 90, 1127-1138. .pdf
2/23/18 Gershman, S.J. (2018). Deconstructing the human algorithms for exploration. Cognition, 173, 34-42. .pdf
Shiffrin, R.M. (2010). Perspectives on modeling in cognitive science. Topics in Cognitive Science, 2, 736-750. .pdf
2/16/18 Gershman, S.J. (2018). Deconstructing the human algorithms for exploration. Cognition, 173, 34-42. .pdf
Shiffrin, R.M. (2010). Perspectives on modeling in cognitive science. Topics in Cognitive Science, 2, 736-750. .pdf
(Discussed models and simulations; never got to the readings)
2/9/18 Erev, I., Ert, E., Plonsky, O., Cohen, D., & Cohen, O. (2017). From anomalies to forecasts: Toward a descriptive model of decisions under risk, under ambiguity, and from experience. Psychological Review, 124 (4), 369-409. .pdf
2/2/18 Nosofsky, R.M. (1991). Tests of an exemplar model for relating perceptual classification and recognition memory. Journal of Experimental Psychology: Human Perception and Performance, 17 (1), 3-27. .pdf
Shepard, R.N. (1987). Toward a universal law of generalization in psychological science. Science, 237 (4820), 1317-1323. .pdf
1/26/18 Peckham, A.D., & Johnson, S.L. (2015). Spontaneous eye-blink rate as an index of reward responsivity: Validation and links to bipolar disorder. Clinical Psychological Science. .pdf
Anderson, B.A. et. al. (2017). Linking dopaminergic reward signals to the development of attentional bias: A positron emission study. Neuroimage, 157, 27-33. .pdf
12/21/17 Yechiam, E. & Ert, E. (2007). Evaluating the reliance on past choices in adaptive learning models. .pdf
12/8/17 Estes, W.K. (1976). The cognitive side of probability learning. Psychological Review, 83, (1), 37-64. .pdf
12/1/17 Turner et al. (2017). Approaches to analysis in model-based cognitive neuroscience. Journal of Mathematical Psychology, 76 Part B, 65-79. .pdf
Love, B.C. (2017). Cognitive models as bridge between brain and behavior. Trends in Cognitive Sciences, 1545. .pdf
11/17/17 Lee et al. (Psycharxiv). Deep neural networks as Gaussian processes. .pdf
Bush, R.R., & Mosteller, F. (1951). A mathematical model for simple learning. Psychological Review. .pdf
Aitken, M.R.F., & Dickinson, A. (2005). Simulations of a modified SOP model applied to retrospective revaluation of human causal learning. Learning & Behavior, 33(2), 147-159. .pdf
11/10/17 Psychonomics – no meeting but we’ll read extra for 11/17
11/3/17 Estes, W.K. (1950). Toward a Statistical Theory of Learning. Psychological Review. .pdf
10/27/17 McClaren & Mackintosh (2000). An elemental model of associative learning: I. Latent inhibition and perceptual learning. Animal Learning & Behavior, 28 (3), 211-246. .pdf
10/20/17 McClaren & Mackintosh (2002). Associative learning and elemental representation: II Generalization and discrimination. Animal Learning and Behavior. .pdf
10/13/17 No meeting because of Armadillo conference
10/6/17 Esber & Haselgrove (2011). Reconciling the influence of predictiveness and uncertainty on stimulus salience: A model of attention in associative learning. Proceedings of the Royal Society B. SuppMs .pdf
Mondragon, Alonso, & Kokkola (2017). Associative learning should go deep. Trends in Cognitive Sciences. .pdf
9/29/17 Nasser, Calu, Schoenbaum, & Sharpe (2017). The dopamine prediction error: Contributions to associative models of reward learning. Frontiers in Psychology. .pdf
9/22/17 Navarro, Newell, & Schultze (2016). Learning and choosing in an uncertain world: An investigation of the explore-exploit dilemma in static and dynamic environments. Cognitive Psychology. .pdf
