The authors were supported by a 5R01EY017921 Grant GDC-0199 manufacturer to R.D., by the European Community’s Seventh Framework Programme (Grant PIRG05-GA-2009-246761),
the General Secretariat for Research and Technology (Grant 9FR27), and the Special Account of Research Funds, University of Crete (Grant 3004) to G.G.G. S.J.G. was supported initially by MH64445 from the National Institutes of Health (USA) and later by the National Institute of Mental Health, Division of Intramural Research. “
“Learning to make choices in a complex world is a difficult problem. The uncertainty attending such decisions requires a trade-off between two contradictory courses of action: (1) to choose from among known options those that are believed to yield the best outcomes, or (2) to explore new, unknown alternatives in hope of an even better result (e.g., when at your favorite restaurant, do you try the chef’s
new special or your “usual” choice?). This well-known exploration-exploitation dilemma (Sutton and Barto, 1998) deeply complicates decision making, with optimal solutions for even simple environments often being unknown or computationally intractable (Cohen et al., 2007). Abundant evidence now supports striatal dopaminergic mechanisms in learning to exploit (see Doll Akt inhibitor and Frank, 2009 and Maia, 2009 for review). By contrast, considerably less is known about the neural mechanisms driving exploration (Aston-Jones and Cohen, 2005, Daw et al., 2006 and Frank et al., 2009). In the reinforcement learning literature, exploration is often modeled using stochastic choice rules. Such rules permit agents to exploit the best known actions for reward while also discovering better actions over time by periodically choosing at random or by increasing stochasticity of choice when options have similar expected values (Sutton and Barto, 1998). A more efficient strategy is to direct exploratory choices
to those actions about which one is most uncertain (Dayan and Sejnowski, 1996 and Gittins and Jones, 1974). Put another way, the drive to explore may vary in proportion to the differential uncertainty about ADP ribosylation factor the outcomes from alternative courses of action. Thus, from this perspective, the brain should track changes in relative uncertainty among options, at least in those individuals who rely on this strategy for exploratory choices. Neurons in prefrontal cortex (PFC) may track relative uncertainty during decision making. Using fMRI, Daw et al., (2006) observed activation in rostrolateral prefrontal cortex (RLPFC; approximately Brodmann area [BA] 10/46) during a “multiarmed bandit task” when participants selected slot machines that did not have the highest expected value. Daw et al. tested whether participants guide exploration toward uncertain options, but did not find evidence for an “uncertainty bonus.