Generalization from past experience is an important feature of intelligent systems. When faced with a new task, one efficient computational approach is to evaluate solutions to earlier tasks as candidates for reuse. Consistent with this idea, we found that human participants (n = 38) learned optimal solutions to a set of training tasks and generalized them to novel test tasks in a reward-selective manner. This behavior was consistent with a computational process based on the successor representation known as successor features and generalized policy improvement (SF&GPI). Neither model-free perseveration or model-based control using a complete model of the environment could explain choice behavior. Decoding from functional magnetic resonance imaging data revealed that solutions from the SF&GPI algorithm were activated on test tasks in visual and prefrontal cortex. This activation had a functional connection to behavior in that stronger activation of SF&GPI solutions in visual areas was associated with increased behavioral reuse. These findings point to a possible neural implementation of an adaptive algorithm for generalization across tasks.

Fig 1. Experimental design.

Participants competed a gem collector game while their brain activity was measured with functional magnetic resonance imaging. On each trial in the experiment, gems with distinct shapes could be resold for either a gain or a loss. Participants made a choice between four cities from around the world, each leading to a distinct collection of gems. To maximize profit overall, participants needed to choose the city best suited to the selling prices shown on each trial. Each block consisted of 32 training trials that included feedback (shown in A), followed by a mixture of 16 training trials with feedback and 20 test trials without feedback (shown in B). The gem collection associated with each city changed from block to block. A: An example training task. Following the presentation of the market values, participants selected a city and saw a feedback screen. The feedback screen revealed the gem numbers in the selected city and the profit earned on the current trial. Four training tasks were used in the experiment and were designed so that two of the four cities were optimal across training tasks. B: An example test task. On test tasks, participants saw a set of market values that had not appeared during training and selected a city, but did not see the outcome of their decision. Four test tasks were used in the experiment and were designed so that the two cities that were previously suboptimal now offered the highest returns. A model-based agent with a complete model of the environment is expected to be sensitive to this change. A “memory-based” SF&GPI agent that evaluates earlier task solutions as candidates for generalization is expected to choose among the cities that were optimal during training.

https://doi.org/10.1371/journal.pbio.3003174.g001

Fig 2. Behavioral results.

Columns are grouped into theoretical predictions and human performance. Rows are grouped into training and test tasks. A, B, F, G: Theoretical predictions are shown for a model-based algorithm on training (A) and test tasks (F), as well as an SF&GPI algorithm on training (B) and test tasks (G). Each theoretical plot shows the predicted proportion of choices leading to each feature (gem) triplet. Theoretical plots show expected choice profiles after model convergence on training tasks and do not include choices during learning. C, H: Human choice profiles for training (C) and test tasks (H). The axis structure matches the theoretical plots. D, I: Performance histograms. The x-axis shows the average reward earned per trial within 10-point bins, and the y-axis shows the number of participants with this average. The dashed red line indicates the performance expected from random choices. The dashed gray line indicates the maximum performance possible. E–J: Use of the optimal training solutions (policies) on training tasks (E) and their reuse on test tasks (J). Both plots show the proportion of choices in which either optimal training policy was used (left bars) and the proportion of choices in which the more rewarding one was used (right bars). Dashed red lines indicate chance. A, B, C, F, G, H: π* denotes feature triplets associated with optimal training and test policies. Optimal policies for training tasks lead to feature triplet ɸ(1) or ɸ(4), and optimal policies for test tasks lead to ɸ(2) or ɸ(3). C, H, E, J: Dots show data points from individual participants. A–J: Materials to reproduce these panels are available at https://gin.g-node.org/sam.hall-mcmaster/sfgpi-behavioural-analysis.

https://doi.org/10.1371/journal.pbio.3003174.g002

Fig 3. Decoding results.

A: Brain regions of interest. The four regions include occipitotemporal cortex (OTC), the medial temporal lobe (MTL), orbitofrontal cortex (OFC) and dorsolateral prefrontal cortex (DLPFC). Regions were defined using FreeSurfer [27]. B: Decoding evidence for the optimal training policies on test tasks (y-axis) shown for each brain region (x-axis). The dashed line indicates chance. C: Neural prioritization to the optimal training policies. This panel shows the difference in decoding evidence on test tasks between the optimal training policies and the objective best policy. Higher values indicate stronger evidence for the optimal training policies within the neural signal. D, E: Time resolved decoding evidence for the optimal training policies on test trials. Panels begin at the onset of the test cue. The initial period shows negative decoding evidence due to a control procedure. To avoid activity from the previous trial biasing our assessment of the current trial, we excluded trials in which the current policy category was selected on the previous trial. B–E: The effects with consistent numerical and statistical evidence across panels B–E are the activation and prioritization of the optimal training policies in OTC, as well as the activation and prioritization of the more rewarding training policy in DLPFC. The remaining effects in B–E did not show consistent evidence across analyses and are not interpreted further. F, G: Time-resolved decoding evidence locked to the response phase onset. Evidence is presented for both the optimal policy on training trials and the optimal training policies on test trials. D–G: Colored bars below each line indicate significant decoding clusters. Shaded error bars show the standard error of mean. H, I: Relationships between decoding evidence for the more rewarding training policy in a specific ROI (y-axis) and the proportion of test trials in which participants reused that policy (x-axis). Black lines indicate linear fits to the data and gray lines indicate 95% confidence intervals of the fits. B–I: Throughout the figure, decoding evidence for the more rewarding among the two optimal training policies is shown in orange, and the less rewarding among the two optimal training policies is shown in blue. The policy within this set that is more or less rewarding varies across test trials depending on the specific cue. B–G: Materials to reproduce these panels are available at https://gin.g-node.org/sam.hall-mcmaster/sfgpi-neural-analysis. H, I: Materials to reproduce these panels are available at https://gin.g-node.org/sam.hall-mcmaster/sfgpi-behavioural-analysis.

https://doi.org/10.1371/journal.pbio.3003174.g003

Fig 4. Reward outcomes for each combination of task cue and policy in the experiment.

Rewards for training tasks are shown in the left-hand matrix. Rewards for test tasks are shown in the right-hand matrix. The order of feature triplets here is the same as other figures in the main text and supplement. [120,50,110] (or a permutation of these numbers) corresponds to ‘Feature Triplet 1′, [90,80,190] (or a permutation of these numbers) corresponds to ‘Feature Triplet 2′ and so on. Materials to reproduce this figure are available at https://gin.g-node.org/sam.hall-mcmaster/sfgpi-behavioural-analysis.

https://doi.org/10.1371/journal.pbio.3003174.g004

Participants learned optimal policies for training tasks

We first examined performance on the training tasks (Fig 2A–2E). This included training trials from the initial learning phase and training trials that were interspersed between test trials. The average reward earned per choice was higher than what would be expected if participants were choosing at random (M = 49.77, SD = 10.61, M_random = −54.38, SD_random = 6.16, t(74) = 52.34, corrected p < 0.001) suggesting successful learning. To understand training performance in more detail, we examined which choice options were selected. Training tasks were designed so that the most rewarding choice on each trial would lead to gem triplet ɸ(1) or gem triplet ɸ(4). Gem triplets are called feature triplets hereafter. Consistent with the training structure, cities associated with ɸ(1) and ɸ(4) were chosen significantly more often than the other cities (Fig 2C). ɸ(1) was reached more often than ɸ(2) or ɸ(3) (M_ɸ(1) = 41.66% of training trials, SD_ɸ(1) = 3.44, M_ɸ(2) = 6.52%, SD_ɸ(2) = 2.70, t(37) = 37.78, corrected p < 0.001; M_ɸ(3) = 6.38%, SD_ɸ(3) = 2.71, t(37) = 37.95, corrected p < 0.001). ɸ(4) was similarly reached more often than ɸ(2) or ɸ(3) (M_ɸ(4) = 45.28% of training trials, SD_ɸ(4) = 2.02, t_{ɸ(4) vs. ɸ(2)}(37) = 45.98, corrected p < 0.001; t_{ɸ(4) vs. ɸ(3)}(37) = 45.42, corrected p < 0.001). Choices leading to ɸ(1) and ɸ(4) were more comparable in number. However, ɸ(4) was reached more often than ɸ(1) (t(37) = 5.34, corrected p < 0.001). To understand the optimality of these decision patterns, we examined how often participants made the optimal choice on training trials. The percentage of optimal choices was significantly above chance (M = 82.94%, SD = 6.36, chance = 25%, t(37) = 56.16, corrected p < 0.001, Fig 2E) indicating that participants acquired the optimal training policies. Together, these results indicate that participants acquired robust and effective decision strategies to maximize reward on the training tasks.

Participants transferred optimal training policies to test tasks

Having shown that participants learned the optimal training policies, we turned our attention to the test tasks (Fig 2F–2J). The test tasks were designed so that choices leading to feature triplets ɸ(2) or ɸ(3) were the most rewarding. A model-based agent was expected to compute anticipated rewards under all available policies and therefore make different choices on test tasks compared to the training tasks. An SF&GPI agent that stores information about the best cities from training was expected to compute anticipated rewards only under the optimal training policies. This would result in continued choices to reach ɸ(1) and ɸ(4), rather than switching to ɸ(2) and ɸ(3). Although decisions for ɸ(1) and ɸ(4) were suboptimal, we expected that participants would still select the option that generated a higher reward among this set.

Consistent with an SF&GPI algorithm, participants continued using optimal policies from training on most test trials (M = 68.81%, SD = 15.58, chance = 50%, t(37) = 7.44, p < 0.001, Fig 2J). The choice profile on test trials was also similar to the profile seen on training trials (Fig 2H). The city associated with feature triplet ɸ(1) was selected significantly more often than cities that led to ɸ(2) and ɸ(3) (M_ɸ(1) = 42.52% of test trials, SD_ɸ(1) = 8.27, M_ɸ(2) = 15.23%, SD_ɸ(2) = 7.43, t(37) = 11.48, corrected p < 0.001; M_ɸ(3) = 15.96%, SD_ɸ(3) = 10.23, t(37) = 9.78, corrected p < 0.001). The city associated with feature triplet ɸ(4) was similarly selected significantly more often than those associated with ɸ(2) and ɸ(3) (M_ɸ(4) = 26.29% of test trials, SD_ɸ(4) = 10.64, t_{ɸ(4) vs. ɸ(2)}(37) = 4.14, corrected p < 0.001; t_{ɸ(4) vs. ɸ(3)}(37) = 3.17, corrected p = 0.006). ɸ(1) was reached more often than ɸ(4) during test tasks (t(37) = 9.12, corrected p < 0.001). While most choices were technically suboptimal, participants still performed well on test tasks overall. The average reward per choice was significantly higher than the reward expected from random choice (M = 147.00, SD = 13.94, M_random = 76.89, SD_random = 7.84, t(74) = 27.02, p < 0.001, Fig 2I). This was due to participants selecting the more rewarding solution among the optimal training policies on test tasks significantly more often than chance (M = 63.71%, SD = 14.22, chance = 0.25, t(37)=16.78, corrected p < 0.001, Fig 2J). On trials where participants used either of the two optimal training policies, the more rewarding one was indeed selected in most cases (M = 92.86%, SD = 5.21, chance = 0.5, t(37) = 50.74, corrected p < 0.001). The proportion of SF&GPI-predicted choices on test tasks was positively related to the proportion of optimal choices during training but not to a significant degree (Spearman’s Rho = 0.294, corrected p = 0.073).

Auxiliary behavioral results are presented in the supplementary information. Training performance reached an asymptote before test tasks were introduced (S1 Fig), and participants could be separated into subgroups based on how closely their behavior matched the SF&GPI predictions (S2 Fig). The SF&GPI-predicted choice was made significantly more often than the alternatives on 3 of the 4 individual test tasks (S3 Fig), with the remaining test task showing an even split between SF&GPI and MB choice, an interesting behavioral signature of Universal Value Function Approximation (S4 Fig). Generalization performance could not be explained by a simple (model-free) perseveration process (S5 Fig). The experiment also included an assessment of participants’ explicit knowledge. Following scanning, participants estimated the number of gems that were present in each city during the final block of the experiment (S6 Fig). While we did not detect a difference in the mean gem estimate error across policies (corrected p-values > 0.09, S6A Fig), we observed a significant positive correlation between the estimate error for cities that were suboptimal during training and SF&GPI-consistent choices on test trials (Spearman’s Rho = 0.493, corrected p = 0.016, S6B Fig). This suggests a possible link between noisy memory for feature triplets associated with the suboptimal training policies and the reuse of successful past solutions.

Taken together, the results in this section indicate that participants were choosing among the optimal training policies on test trials in a reward-sensitive manner, a behavioral profile more consistent with the predictions of an SF&GPI algorithm than an MB algorithm operating on a full model of the environment.

Optimal training policies could be decoded during test tasks

Having established that behavior was consistent with the predictions of an SF&GPI model, we tested its neural predictions on test tasks: (1) that optimal training policies would be activated as decision candidates; (2) that their activation strength would be higher than alternative policies; and (3) that features associated with the optimal training policies would also be represented. We tested these predictions in four brain regions. Predictions 1–2 were expected in DLPFC due to its proposed role in policy encoding [11,12] and context-dependent action [13–17]. Prediction 3 was expected in MTL and OFC due to research implicating these regions in coding predictive information about future states [18–22]. OTC was examined as a final region due to its central role in early fMRI decoding studies and its continued inclusion in recent ones [20,23–25]. We first focus on policy activation (predictions 1–2).

To examine policy activation, decoders based on logistic regression were trained to distinguish the four cities seen during feedback on the training tasks (i.e., the cities served as training labels). One measurement volume (TR) per eligible trial was used as input for decoder training, taken 4–6 s after feedback onset to account for the hemodynamic delay. Decoders were trained separately for each region of interest (ROI). The specific time lag used for each ROI was determined in validation analyses of the training tasks, independent from our predictions about test task activity (see S7 Fig). Validation analyses further showed that decoder training locked to feedback onset on training tasks was more effective than decoder training locked to task cue onset (see S8 Fig). The decoders were applied to each TR on held-out test trials in a leave-one-run-out cross-validation procedure. This resulted in a decoding probability time course for each city on each test trial, which reflected the evidence that a particular city stimulus was encoded in the fMRI signal. To account for class imbalances in the decoder training set, we repeated the analysis 100 times using random subsampling that matched the number of trials per category (M = 53 trials per training class, SD = 6.48, minimum = 37, maximum = 65). Decoding probabilities based on the cities were then coded into the following categories: (1) the more rewarding policy (among the optimal training policies); (2) the less rewarding policy (among the optimal training policies); (3) the objective best policy; and (4) the remaining policy.

We first tested the prediction that optimal training policies would be activated on test tasks, by comparing decoding evidence for the optimal training policies against chance in each ROI (Fig 3B). Test trials lasted 10 s on average (Fig 1B). Decoding evidence was therefore averaged from +2.5 s to +12.5 s following test trial onset to account for the hemodynamic delay. This revealed significant average decoding evidence for the more rewarding training policy during test tasks in OTC and DLPFC (OTC: M = 28.07%, SD = 2.17, t(37) = 8.57, corrected p < 0.001; DLPFC: M = 25.85%, SD = 1.51, t(37) = 3.40, corrected p = 0.011), but not in MTL or OFC (MTL: M = 25.17%, SD = 1.43, t(37) = 0.72, corrected p = 0.744; OFC: M = 25.40%, SD = 1.37, t(37) = 1.79, corrected p = 0.405). Numerical evidence for the less rewarding training policy was also detected in OTC, but this did not survive correction (M = 25.97%, SD = 2.34, t(37) = 2.53, uncorrected p = 0.016, Bonferroni–Holm corrected p = 0.094, see the section entitled Reactivation and stimulus processing contributed to the decoding effects for evidence from cluster-based permutation tests). Average decoding evidence for the less rewarding training policy was not detected in the remaining ROIs (MTL: M = 25.32%, SD = 1.76, t(37) = 1.12, corrected p = 0.744; OFC: M = 25.31%, SD = 1.40, t(37) = 1.38, corrected p = 0.708; DLPFC: M = 25.30%, SD = 1.53, t(37) = 1.17, corrected p = 0.744). Decoding evidence on individual test tasks is included in the supplement (S9 Fig). These results indicate that the more rewarding among the optimal training policies was activated in OTC and DLPFC during test tasks.

Optimal training policies were prioritized on test tasks

Having established that the optimal training policies were activated on test tasks (prediction 1), we next tested whether their activation strength was higher than the other policies (prediction 2, Fig 3C). Average decoding evidence during the test tasks was significantly higher for the more rewarding training policy than the objective best policy in OTC, OFC, and DLPFC (OTC: M_diff = 4.09%, SD_diff = 3.69, t(37) = 6.75, corrected p < 0.001; MTL: M_diff = 0.48%, SD_diff = 2.46, t(38) = 1.17, corrected p = 0.743; OFC: M_diff = 0.97%, SD_diff = 1.87, t(38) = 3.14, corrected p = 0.030; DLPFC: M_diff = 1.61%, SD_diff = 2.59, t(38) = 3.77, corrected p = 0.006). Similar results were observed for the less rewarding training policy. Average decoding evidence for the less rewarding training policy was significantly higher than the objective best policy in OTC and DLPFC (OTC: M_diff = 2.00%, SD_diff = 3.95, t(37) = 3.07, corrected p = 0.032; DLPFC: M_diff = 1.05%, SD_diff = 2.15, t(37) = 2.99, corrected p = 0.035; MTL: M_diff = 0.63%, SD_diff = 2.80, t(37) = 1.36, corrected p = 0.722; OFC: M_diff = 0.88%, SD_diff = 1.97, t(37) = 2.71, corrected p = 0.060). Relative decoding scores above take the form: (evidence for training policy X) − (evidence for the objective best policy) and measure the extent to which optimal training policies were prioritized in neural activity. We found that relative decoding scores were significantly higher for the more rewarding training policy than the less rewarding one in OTC (M_diff = 2.09%, SD_diff = 2.37, t(37) = 5.38, corrected p < 0.001) but comparable in other ROIs (MTL: M_diff = −0.15%, SD_diff = 1.61, t(37) = −0.58, corrected p > 0.99; OFC: M_diff = 0.09%, SD_diff = 1.93, t(37) = 0.28, corrected p > 0.99; DLPFC: M_diff = 0.55%, SD_diff = 1.94, t(37) = 1.73, corrected p = 0.092). The results from testing predictions 1 and 2 suggest that successful training policies were activated and prioritized on test tasks in OTC. The more rewarding training policy was activated and prioritized in DLPFC (Fig 3B–3E). Remaining effects did not show consistent evidence when testing both predictions and are not considered further.

Reactivation and stimulus processing contributed to the decoding effects

We next used cluster-based permutation testing [26] to estimate when the optimal training policies were activated in the decoding time courses (window tested = 2.5–12.5 s post cue; 10,000 permutations). These tests revealed significant decoding evidence for the more rewarding training policy from 5 to 12.5 s on test trials in OTC (corrected p < 0.001) and from 8.75 to 12.5 s in DLPFC (corrected p < 0.001). Cluster-based permutation testing is potentially more sensitive than testing average evidence because activation that occurs in smaller subsets of time points can be identified. Consistent with this idea, cluster tests identified significant decoding evidence for the less rewarding training policy in OTC from 5 to 7.5 s (corrected p < 0.001) and 11.25 to 12.5 s (corrected p = 0.016), as well as a candidate cluster at 3.75 s in DLPFC (corrected p = 0.072). No signfiicant decoding clusters were detected for OFC or MTL (all corrected p-values > 0.301). These results suggest that the more rewarding training policy was activated in the MRI signal from OTC about 5 s after trial onset and in the signal from DLPFC about 8.75 s after trial onset. The OTC signal also had transient information about the less rewarding past policy from about 5 s (Fig 3D, 3E).

To assess whether these effects were impacted by the choice made on each test trial, we re-ran the cluster tests above but excluded evidence from trials where the policy category matched the choice participants made. A significant decoding cluster for the more rewarding training policy was detected in OTC from 5 to 7.5 s (corrected p = 0.003) but no significant clusters were detected in DLPFC (candidate cluster at 5 s, corrected p = 0.254). The same results were seen for the less rewarding training policy. Significant decoding clusters were observed in OTC from 5 to 7.5 s (corrected p = 0.003) and from 11.25 to 12.5 s (corrected p = 0.021), but not in DLPFC (no candidate clusters). These results suggest that information about successful past policies could still be decoded from OTC when controlling for the choice made. This was not the case for DLPFC.

So far we have seen that OTC activated the optimal training policies and that DLPFC activated the more rewarding of the optimal training policies. These effects could be due to the neural reactivation of the policies from memory following the test cue, selective attention to specific policies when the response screen was shown, or both. To arbitrate between these possibilities, we examined decoding evidence locked to the response phase, and asked whether activation of the optimal training policies arose earlier than could be expected from the response phase alone (Fig 3F, 3G). The procedure for training the decoder remained the same as the previous analyses. However, the evaluation was conducted at each time point from 0 to 7.5 s relative to response phase onset rather than the cue onset. To establish the expected lag of policy decoding after the response phase onset, we examined decoding evidence on training trials using cluster-based permutation tests (window tested = 0–7.5 s, 10,000 permutations). Using these trials as a baseline revealed a significant decoding cluster for the optimal policy in OTC that was first detected 3.75 s after the start of the response phase during training trials (significant cluster window = 3.75–7.5 s, corrected p < 0.001, Fig 3F). In contrast to this expected lag, the cluster onset was shifted earlier on test trials, with significant information about the optimal training policies present in OTC already 0 s from the response phase onset (more rewarding training policy: significant cluster window = 0–7.5 s, corrected p < 0.001; less rewarding past policy: first significant cluster window = 0–2.5 s, corrected p =  0.001, second significant cluster window = 6.25–7.5 s, corrected p = 0.014). In line with these results, decoding evidence in OTC at 0 s from the response phase onset was significantly higher on test trials compared to training trials (more rewarding training policy: M_diff = 2.94%, SD_diff = 3.38, t(37) = 5.29, corrected p < 0.001; less rewarding training policy: M_diff = 2.81%, SD_diff = 3.97, t(37) = 4.31, corrected p < 0.001). No differences at 0 s were detected for DLPFC (corrected p-values > 0.623, Fig 3G). To summarize, information about the optimal training policies was present from response phase onset in OTC during test trials. Due to the hemodynamic delay, this implies some reactivation of the optimal training policies in OTC before the response screen was displayed. Attention to specific stimuli shown on the response screen would have then plausibly contributed to the decoding effects in OTC from around 2.5 to 3.75 s after its onset (Fig 3F).

Policy activation in OTC was associated with test choices

The neural results presented above suggest that OTC and DLPFC represented information predicted under an SF&GPI algorithm. To examine whether neural coding in OTC and DLPFC had a functional connection to participant choices, we correlated average decoding strength for the more rewarding training policy (the orange dots in Fig 3B) with the proportion of test trials in which participants generalized the more rewarding training policy. This revealed a significant positive correlation between neural activation of the more rewarding training policy in OTC and the implementation of that policy at test (Spearman’s Rho = 0.447, corrected p = 0.010, Fig 3H). The equivalent correlation was not detected for DLPFC (Spearman’s Rho = 0.115, corrected p = 0.492, Fig 3I). The correlation results held when using neural prioritization towards the more rewarding training policy (the orange dots in Fig 3C) as the neural variable (OTC: Spearman’s Rho = 0.433, corrected p = 0.013; DLPFC: Spearman’s Rho = 0.161, corrected p = 0.335). Neural coding in OTC was also directionally specific. Increased neural evidence for the more rewarding training policy in OTC showed a significant negative relation to the proportion of optimal MB choices at test (OTC: Spearman’s Rho = −0.375, corrected p = 0.041; DLPFC: Spearman’s Rho = −0.098, corrected p = 0.560). These results suggest that policy coding within the OTC was associated with test choices.

Features could not be decoded on test tasks

Having seen that optimal training policies were activated on test tasks (prediction 1) and that the evidence for them was higher than the objective best policy (prediction 2), we tested whether features associated with the optimal policies were also represented (prediction 3). This required a different decoding approach for two reasons. The first was that feature triplets ɸ(1) and ɸ(4) were consistently associated with the optimal training policies and were thus selected on most trials. Training a feature decoder on these data directly would result in large imbalances in the number of trials per class. The second reason was that the feature triplets were correlated with the reward. Feature triplets ɸ(1) and ɸ(4) often resulted in a profit and feature triplets ɸ(2) and ɸ(3) often resulted in a loss.

To circumvent these issues, we trained feature decoders on fMRI data from a separate associative memory paradigm (see Session One in the Methods section and S10 Fig). Participants were first pre-trained on associations between visual cues and target stimuli. The target stimuli were feature numbers or cities that would later be used in the gem collector game. On each scanning trial, participants were shown a visual cue and needed to select its associated target from a choice array. Trials in which the correct number target was selected were used to train logistic decoders that could distinguish neural responses for the 12 number targets. The training process was similar to the process used for policy decoding. One TR per eligible trial was used as training input, taken 4–6 s after the response screen onset. The time shift and smoothing used for each ROI were the same as those used to decode policies. This approach could successfully recover the 12 number targets when tested in held-out data from the associative memory paradigm in all ROIs except MTL (S11 Fig). The trained decoders were then shown neural data from the test trials in the gem collector paradigm. This returned a cross-validated decoding probability for each feature number at each TR on each test trial. To ensure the decoding probabilities were stable, we repeated the procedure 100 times using random subsampling to match trial numbers (M = 36 trials per training class, SD = 2.44, minimum = 29, maximum = 39). We then identified the three feature numbers associated with each of the optimal training policies and averaged the decoding probabilities for those number labels on each test trial.

Using this approach, we examined whether average decoding evidence for the three features anticipated under each of the optimal training policies was higher than chance (8.33% based on the 12 possible feature numbers). Like the earlier section on policies, decoding evidence was averaged from +2.5 s to +12.5 s following test trial onset. Feature information associated with the more rewarding training policy was not detected on test tasks (OTC: M = 8.71%, SD = 1.12, t(37) = 2.04, corrected p = 0.392; MTL: M = 8.37%, SD = 0.40, t(37) = 0.51, corrected p > 0.99; OFC: M = 8.28%, SD = 0.52, t(37) = −0.60, corrected p > 0.99; DLPFC: M = 8.34%, SD = 0.72, t(37) = 0.05, corrected p > 0.99). Equivalent results were found for features associated with the less rewarding training policy (OTC: M = 8.61%, SD = 0.91, t(37) = 1.87, corrected p = 0.486; MTL: M = 8.38%, SD = 0.59, t(37) = 0.47, corrected p > 0.99; OFC: M = 8.37%, SD = 0.52, t(37) = 0.48, corrected p > 0.99; DLPFC: M = 8.38%, SD = 0.57, t(37) = 0.50, corrected p > 0.99, S12 Fig).

Computational accounts of generalization behavior

Consistent with previous behavioral research [7], a computational process based on SF&GPI could explain human generalization performance. However, it did not capture participant choices perfectly. This was evident in data showing that participants made fewer choices leading to feature triplet ɸ(4) on test trials than the model predicted (Fig 2G, 2H). Exploratory tests revealed that this was due to the presence of two distinct subgroups (S2 Fig). Half of the participants showed a full recapitulation of the SF&GPI predictions on test tasks. The other half showed a partial recapitulation. This suggests that some participants used different strategies on specific test tasks. When examining individual test tasks (S3 Fig), we further observed that the SF&GPI algorithm predicted the dominant choice in most cases. However, there was one test task on which choices were evenly split between the SF&GPI and MB predictions. Whereas most test tasks contained an anti-correlated structure in the feature weights that was similar to one or more training tasks, this outlier test task did not. This raises the possibility that at least some participants exploited structural similarities between the training and test tasks to determine their choices, similar to a Universal Value Function Approximator (UVFA) process in which similar task cues lead to similar rewards for a given action [28]. Tomov and colleagues observed evidence for UVFAs in a minority of subjects [7], but also had less structural overlap between training cues and the test cue. Future research should systematically manipulate the similarity between training and test scenarios to understand how this affects the computational process used for generalization.

Post hoc behavioral predictions of a UVFA process are included in S4 Fig. These predictions show consistencies with human choice, suggesting that UVFA could offer an alternative account of behavior. However, the neural findings from this experiment appear to be inconsistent with several UVFA predictions. Vanilla UVFAs predict equal activation of all available actions to compute and compare Q-values. In contrast, OTC showed activation of the optimal training actions/policies at choice onset, followed by sustained activation of the more rewarding action/policy (Fig 3D), whereas DLPFC showed activation of the more rewarding action/policy (Fig 3E). While this could be explained by a partial Markov decision process, in which people only evaluate actions that were reinforced during training, it would be inconsistent with the UVFA-like empirical choice profile on test task w = [1,1,−1] (S4B Fig), in which a non-reinforced action is selected with high frequency. A UVFA explanation of the empirical choice profile on test task w = [1,1,−1] (S4B Fig) would also predict approximately equal neural activation of both the objective best policy and the more rewarding training policy. Instead, we find that the more rewarding training policy is activated significantly more than the objective best policy, for which decoding evidence is at chance (S9B Fig). Despite these neural inconsistencies, the behavioral similarities to UVFAs suggest that humans may utilize some kind of UVFA-like generalization, perhaps in addition to SF&GPI-like policy reuse, in which the evaluated actions are not explicitly represented in the brain regions considered here. Exploring potential hybrid strategies that combine UVFAs with SF&GPI could be a promising area of future investigation.

Recent studies have also shown behavioral and neural evidence for reinforcement learning (RL) based on configurations of cues, over and above their individual elements [29,30]. Applied to our setup, this would correspond to learning unique values for each (city, task) configuration. This account would not immediately capture the generalization effects reported here. While an agent learning to respond to each configuration would be expected to learn optimal solutions to the training tasks, it would not have an obvious mechanism to map this knowledge to new tasks (i.e., new configurations of cues), the same issue facing a model free-agent (S5 Fig). Nonetheless, configural accounts could in principle be extended to support generalization to novel configurations using function approximation. This would allow an agent to smoothly interpolate between the values for different (city, task) configurations experienced during training, to compute approximate values for the new (city, task) configurations on test trials. Interestingly, this is precisely what UVFAs do. While our experiment was designed to disambiguate between SF&GPI and MB control using a complete model, we do see signatures of UVFA-like response patterns (S4B Fig), even though our neural results appear to be inconsistent with UVFA predictions (S9B Fig). Exploring UVFAs as a generalization of configural RL in the multi-task domain could be an interesting direction for future investigation.

A final account of generalization behavior could be a heuristic strategy, in which participants focus exclusively on the feature with the highest positive reward weight. Participants could then retrieve a policy from the training tasks that shared this top ranked feature. Such a heuristic would lead to SF&GPI-like behavior in three test cases. However, it would not predict neural activation of the less rewarding training policy, as there would be no need to consider other features or policies once the top-ranked gem had been identified. Therefore, this heuristic strategy would not explain the neural effects in OTC, where both the more and less rewarding training policies were activated on test tasks (Fig 3C, 3D).

Neural predictions of SF&GPI

Consistent with SF&GPI predictions, we observed neural prioritization of the optimal training policies during test tasks. This was most prominent in OTC. Evidence from our data suggests that two distinct sources contributed to decoding in this region. One was a reactivation of optimal training policies after seeing the test cue, as optimal training policies could be decoded earlier than what would be expected based on training trials. This could reflect expectations about upcoming stimuli in visual cortex [31,32], with added modulation based on behavioral relevance. The other source was prioritized processing of the optimal training policies shown on the response screen. This enhanced processing could reflect value-driven attentional capture, in which stimuli previously associated with reward are preferentially attended to in new contexts [33]. While OTC is not a usual candidate for experiments on RL and decision-making, we note recent reports that neural replay decoded from OTC relates to the formation of successor representations [25], which are one of the theoretical foundations of SF&GPI. Neural prioritization was also observed DLPFC, an area proposed to encode policies [11,12]. This finding aligns with DLPFC’s role in context-dependent behavior [13–17], as the task cue on each trial can be seen as a context cue that determines the current response mapping. One interpretation of our findings is that DLPFC can generalize this role outside a set of training cues, retrieving relevant response mappings for novel context cues with similar but non-identical structure to earlier contexts. We also note that although the prioritization seen in our data is consistent with an SF&GPI-like process, it could also support hybrid models that use cached option values to identify useful candidates for particular decisions, and then perform model-based planning on that subset of options [34,35].

Based on proposals that the MTL and OFC serve as predictive maps that encode information about future states [18–22], we predicted that features expected under the optimal training policies would be detected in these regions on test trials. We did not find evidence that this was the case. One possibility is that the features were not central to participants’ decision process, which could occur if choices were primarily based on structural similarities between the training and test cues. It is also possible that the features were used but that we were unable detect them. This could occur if feature numbers were represented with different neural patterns in the localizer task used to train the neural decoders than the main task. Cross-validated tests with policy decoders imply some support for this possibility, suggesting that decoders trained and tested on the main task tended to perform better, compared to those trained on the localizer and then tested on the main task (S13 Fig). Future experiments should therefore develop designs in which feature information can be remapped between the optimal and suboptimal policies within each block of a single paradigm to train more sensitive feature decoders. A second reason for not detecting features could be because each test task was repeated five times per block without feedback. This could have meant that feature triplets were used to compute expected rewards during initial test trials but that a policy was cached and used for remaining test trial repeats. Decoding predictive feature representations remains a critical target in future neural tests of SF&GPI.

It is also possible that OFC and MTL were engaged during the experiment but encoded other variables. For instance, one influential view of OFC function is that the OFC represents the prospect of reward for different options during decision-making [36,37]. The present experiment was not optimized to decode value signals due to the wide variation and largely unsystematic distribution of the rewards across training and test tasks (Fig 4), a compromise that arose from the other factors we needed to balance in the experiment to test our central predictions. Some studies also suggest that hippocampal signaling may modulate the reinstatement strength of neural patterns in visual areas [38,39]. Based on these factors, one speculative possibility could be that value information in OFC and hippocampal signaling modulate OTC and DLPFC activity on new tasks, in favor of successful past solutions.

Limitations

The present study had two main limitations. One was that each trial involved a single decision step. This meant that multi-step successor features were equivalent to single-step state features in the present design. This simplification was made to meet the practical constraints of fMRI scanning but future studies will need to devise practical ways to retain a multi-step element that can distinguish between computational processes using successor features and state features for generalization. This would also help to distinguish reuse based on SF&GPI from more subtle forms of MB control that operate on a partial model of the environment. The most direct data we have on whether participants had a partial model in the current experiment were feature estimates collected after fMRI scanning. Specifically, participants were asked to estimate the features (gem numbers) for each city in the final experimental block (S6 Fig). If participants had a partial model of the environment, one prediction is that participants should have more accurate knowledge about features associated with the optimal training policies. We did not detect robust evidence for this prediction (corrected p-values > 0.09, S6A Fig). However, we did observe a significant correlation between having less accurate knowledge about features associated with the suboptimal training policies and the reuse of successful past solutions during test tasks (Spearman’s Rho = 0.493, p = 0.016, S6B Fig). One possibility based on this result is that participants with higher policy reuse had a partial model of the environment or a noisier memory for certain features. Unfortunately, the present design can only arbitrate between a computational process based on SF&GPI and a MB process that uses a complete model of the environment. It is unable to arbitrate between SF&GPI and forms of MB processing that use a partial model of the environment or have noisy memory. Multi-step designs could better address the separation between SF&GPI and partial MB control in future experiments. An MB agent with a partial model should be able to report features for each individual state that comprises the optimal training policies. A memory-based SF&GPI agent is expected to fail on such queries, as it stores a compact summary of this information that does not have individual state resolution. A second limitation was that the reward for the objective best policy was only 10–20 points higher than the more rewarding training policy (from among the optimal training policies). While we observed a high degree of reuse as predicted under an SF&GPI algorithm, it could be that the computational process would differ—with more equal evaluation across all options—if the reward prospect for previously unsuccessful policies had been higher at test. This would align with evidence that more computationally intensive forms of decision-making, such as MB planning, are adopted when the reward for accurate performance is heightened [40]. The present experiment was not optimized to address meta-control between computational processes for different reward-effort trade-offs. Future research could therefore manipulate the difference in reward between successful and unsuccessful training policies during test tasks, to better understand the conditions under which an SF&GPI-like transfer process is (or is not) used.

Conclusions

Overall, the present study provides behavioral and neural evidence that generalization to new tasks was more consistent with an SF&GPI-based algorithm than an MB algorithm using a full model of the environment. These results do not rule out an MB algorithm operating on a partial model of the environment, or with noisy memory for features associated with suboptimal training choices. SF&GPI and UVFAs should also be systematically compared in the presence and absence of structural similarity between training and test cues in future studies. Ultimately, this experiment found that successful past solutions were prioritized as candidates for decision-making on tasks outside the training distribution. This prioritization provides flexibility when faced with new decisions problems and has lower computational cost than considering all available options. These findings take a step towards illuminating the flexible yet efficient nature of human intelligence.

Ethics statement

The experimental protocol was approved by the ethics commission at the German Psychological Society (reference: SchuckNicolas2022-10-24AM) and conducted in accordance with the Declaration of Helsinki. Participants signed informed consent before each test session.

Participants

Forty people participated in the experiment. One participant was excluded from the final sample due to low behavioral performance. The number of points earned in each scanning session was more than 2.5 standard deviations below the sample mean. Another participant was excluded due to excessive head motion. This was based on framewise displacement (FD), which measures the change in head position between adjacent data points [41]. All functional runs in the second scanning session for the participant were more than 2.5 standard deviations above the sample mean for FD. The resulting 38 participants were between 18 and 35 years of age (mean = 26 years, 23 female). All individuals had normal or corrected-to-normal vision and did not report an on-going neurological or psychiatric illness. €75 was paid for completing the experiment and €25 extra could be earned as a performance dependent bonus (€10 in session 1 and €15 in session two).

Materials

Psychopy3 (RRID: SCR_006571, version 2021.2.3, [42]) and Pavlovia (RRID:SCR_023320, https://pavlovia.org/) were used to prescreen prospective participants. Stimulus presentation during the scan sessions was controlled using Psychophysics Toolbox-3 (RRID:SCR_002881, version 3.0.17) in MATLAB (RRID:SCR_001622). The sessions used stimuli from [43] and city images from Rijan Hamidovic, Aleksander Pesaric, Pixabay and Burst that were retrieved from https://pexels.com. Scan preparation was completed on a computer outside the scanner with a spatial resolution of 2048 × 1,152 and refresh rate of 60 Hz (MATLAB version R2021b). The main experimental tasks were run on a stimulus PC with a spatial resolution of 1,024 × 768 and a refresh rate of 60 Hz, which projected to a magnetically safe screen for participants inside the scanner (MATLAB version R2017b). Responses were recorded using two fiber optic button boxes (Current Designs, Philadelphia, PA, USA). Participants used two buttons on the left box (middle finger and index finger) and three buttons on the right box (index finger, middle finger and ring finger). Output files from the scanner were converted to the BIDs naming convention (RRID:SCR_016124) using ReproIn (RRID:SCR_017184, HeuDiConv version 0.9.0). Preprocessing and quality control were conducted using fMRIprep (RRID:SCR_016216, version 20.2.4, [44]) and MRIQC (RRID:SCR_022942, version 0.16.1, [45]). Behavioral analyses were performed in MATLAB (version 2021b). Neural analyses were performed in Python (RRID:SCR_008394, version 3.8.13), primarily using SciPy (RRID:SCR_008058, version 1.7.3), Pandas (RRID:SCR_018214, version 1.4.4), NumPy (RRID:SCR_008633, version 1.21.5), Matplotlib (RRID:SCR_008624, version 3.5.2), MNE (RRID:SCR_005972, version 1.4.2), Sklearn (RRID:SCR_019053, version 1.3.0), and Nilearn (RRID:SCR_001362, version 0.10.1). Data and code management was performed using DataLad (RRID:SCR_003931, version 0.17.9) and GIN (RRID:SCR_015864).

Data and code

Data and code for this project are openly available in the following repositories (Table 1).

Online prescreening

Participants completed an online prescreening task prior to study admission. This was conceptually similar to the main task in MRI session two. However, it had different stimuli, different state features, and a different theme to avoid biasing participants during scanning. The market values used for the prescreening were: w_train = {[1, −1, 0], [−1, 1, 0], [1, −2, 0], [−2, 1, 0]}. The state features were: ɸ = {[100, 0, 0], [30, 30, 160], [100, 100, 0], [0, 100, 70]}. The vector elements in each triplet were shuffled with a randomly selected permutation rule for each participant. The theme used for prescreening was from [7]. Participants completed 80 trials with equal numbers of each training task (w_train) in a random order. No test tasks (w_test) were presented. Trials had the same structure as Fig 1A but with different event timing. Market values were shown for 2.5 s, no response limit was imposed, and feedback was shown for 3 s. The cue-target interval was set to 0 s and the inter-trial interval (ITI) to 1 s. Prospective participants needed to achieve an average reward above zero points to pass the prescreening and participate in the experiment. Up to three attempts could be made. Ninety-five percent of people who passed the prescreening did so on their first attempt. Four prospective participants did not pass and were not admitted to the experiment.

Session one

During the first experimental session, participants completed an associative retrieval task inside the MRI scanner. Session one was designed to produce a localizer dataset that could be used as training data for neural decoders. Data from this session were used to train feature decoders but not policy decoders in the present paper. For the procedure behind the gem collector game, see session two.

Session two

During the second experimental session, participants completed the gem collector game described in the main text. Sessions were held on consecutive days, with session two occurring the day after the session one.

MRI parameters

MRI acquisition parameters were based on [24]. Data were acquired using a 32-channel head coil on a 3-Tesla Siemens Magnetom TrioTim MRI scanner. Functional measurements were whole brain T2* weighted echo-planar imaging (EPI) with multi-band acceleration (voxel size = 2 × 2 × 2 × mm; TR = 1250 ms; echo time (TE)=26 ms; flip angle (FA)=71°; 64 slices; matrix = 96 × 96; FOV = 192 × 192 mm; anterior-posterior phase encoding direction; distance factor = 0%; multi-band acceleration = 4). Slices were tilted +15 relative to the corpus callosum, with the aim of balancing signal quality from MTL and OFC [54]. Functional measurements began after five TRs (6.25 s) to allow the scanner to reach equilibrium and help avoid partial saturation effects. Up to 401 volumes (session one) or 333 volumes (session two) were acquired during each task run; acquisition was ended earlier if participants had completed all trials. Fieldmaps were measured using the same scan parameters, except that two short runs of 20 volumes were collected with opposite phase encoding directions. Fieldmaps were later used for distortion correction in fMRIPrep (RRID: SCR_016216, version 23.0.2, [44]). Anatomical measurements were acquired using T1 weighted Magnetization Prepared Rapid Gradient Echo (MPRAGE) sequences (voxel size = 1 × 1 ×1 × mm; TR = 1900 ms; TE = 2.52 ms; FA = 9°; inversion time (TI)=900 ms; 256 slices; matrix = 192 × 256; FOV = 192 × 256 mm).

MRI preprocessing

Results included in this manuscript come from preprocessing performed using fMRIPrep 23.0.2 (RRID:SCR_016216, [44,55]), which is based on Nipype 1.8.6 (RRID:SCR_002502, [56,57]).

Computational models

Theoretical predictions for choices on test tasks were generated using code from [7]. The code was adapted to the current task design, which included changing the training tasks, test tasks, state features, and the state space. Model training procedures remained the same.

The models were situated in a standard RL formalism called a Markov Decision Process , where is a set of states, is a set of actions, is the probability of transitioning to a subsequent state (s′) from an initial state (s) after taking action a, is the reward received when entering state and is a discount factor between 0 and 1 that down-weights future rewards. The environment also contained state features ɸ(s) and tasks w. Policies defining the action an agent takes in each state are denoted .

Behavioral analyses

Seventeen statistical tests are reported in the main text for behavioral data. The corresponding p-values reported were therefore corrected for 17 tests using the Bonferroni-Holm correction [76]. Auxiliary tests presented in the supplement but mentioned in the main text were separately corrected using Bonferroni-Holm. Supplementary figure legends provide more details about the correction applied in these cases.

Neural analyses

Decoding was performed separately for each ROI. The voxel activity on each trial was extracted from the current ROI and smoothed 2–4 mm. The specific smoothing was determined in validation analyses that did not use the trials needed to test our hypothesis. The resulting data were detrended, z-scored, and motion confounds (identified by fMRIPrep) were removed. Data cleaning was performed separately for each functional run using Nilearn signal clean. Trials without a response were then removed. The decoding procedure next entered a cross-validation loop. One block was held out to use as test data. The remaining blocks were used to train the decoder. A random subsample was taken to match the number of trials for each training class. It is important to note that only trials with feedback were used for decoder training (see Fig 1A). Voxel activity 4–6 s following feedback onset was extracted, and the city shown on the feedback screen was used as the training label. The specific time lag was determined in validation analyses (see S7 Fig). The resulting voxel patterns and labels were used to train a logistic regression model. This was implemented in Sklearn with an L2 penalty term and a one-vs-the-rest multiclass strategy. For more details, please see the Sklearn documentation. The trained decoder was then tested on each trial in the held-out run. Testing was performed at multiple TRs on each trial, from cue onset (TR₀) through to 15 s after it (TR₊₁₂). This resulted in one probability score per city stimulus at each TR in the held-out test trial. Each score was the probability—according to the model—that a particular city was the correct label for the neural data provided. The class probabilities at a specific data point did not need to sum to 1 (due to the one-vs-the-rest approach). The cross-validation procedure was repeated until all functional runs had been used as held-out test data. For robustness, the analysis procedure was repeated 100 times with random subsampling on each iteration. The results were based on model probabilities averaged across iterations. These probabilities are interpreted as decoding evidence throughout the paper.

S1 Fig. Learning dynamics on training tasks.

This plot shows the proportion of optimal choices for the 48 training trials per block (i.e., as a function of time). The black line shows the mean across participants and shading indicates the standard error of mean. The red line indicates the point in each block when test tasks are introduced. The earliest this could occur was trial 34. Performance begins around chance (M_{trial 1} = 0.24, SD_{trial 1} = 0.16) and reaches an asymptote by the time test tasks are introduced (M_{trial 33} = 0.93, SD_{trial 33} = 0.10). Materials to reproduce this figure are available at https://gin.g-node.org/sam.hall-mcmaster/sfgpi-behavioural-analysis.

https://doi.org/10.1371/journal.pbio.3003174.s001

(TIF)

S2 Fig. Subgroup analysis.

Exploratory analysis of participant choices on test tasks revealed two subgroups. Each plot shows the proportion of choices on test tasks (y-axis) that lead to each feature triplet (x-axis). π* denotes feature triplets associated with optimal test policies. A: The predicted choice profile across test tasks according to an SF&GPI algorithm. B: The choice profile for a subgroup of participants (n = 17) showing a strong recapitulation of the SF&GPI predictions. The choice profile for the remaining participants (n = 19) showing a partial recapitulation. B, C: Dots show individual participant data. Lines connect individual participant data across the feature triplets. A–C: Materials to reproduce this figure are available at https://gin.g-node.org/sam.hall-mcmaster/sfgpi-behavioural-analysis.

https://doi.org/10.1371/journal.pbio.3003174.s002

(TIF)

S3 Fig. Choice behavior on individual test tasks.

Participants’ most common action matched the SF&GPI predictions in 3/4 cases. Each plot shows the proportion of choices (y-axis) that lead to each feature triplet (x-axis). Each panel shows the choice profile for an individual test task. The cue for an individual task was a weight vector, w, which is indicated in the plot titles. These cues were called market values during the gem collector game. Each panel includes three plots: the theoretical predictions of a model-based algorithm, the theoretical predictions of an SF&GPI algorithm, and empirical choices from human participants. Bars indicate the standard error of the mean in plots with human data. To formally examine human choice profiles on an individual task level, we compared whether participants made the SF&GPI predicted choice significantly more often than the alternatives. For example, using data from the test task w=[2,−1,−1], we compared whether the proportion of choices leading to feature triplet 1 was significantly higher than the proportions leading to feature triplets 2, 3, and 4. Repeating this process for each individual test task revealed that participants made the SF&GPI-predicted choice significantly more often in all cases except one (all t-values without the exception > 4.54, all corrected p-values without the exception < 0.001). The exception can be seen in panel B where the proportion of SF&GPI-predicted choices and MB-predicted choices were comparable (M_ɸ(4) = 40.24% of test trials, SD_ɸ(4) = 27.40, M_ɸ(3) = 41.83%, SD_ɸ(3) = 32.91, t(37) = −0.166, corrected p = 0.869). Choice proportions were compared using paired samples t-tests, and p-values were corrected for the 12 tests in this section using the Bonferroni–Holm correction. Materials to reproduce this figure are available at https://gin.g-node.org/sam.hall-mcmaster/sfgpi-behavioural-analysis.

https://doi.org/10.1371/journal.pbio.3003174.s003

(TIF)

S4 Fig. Predictions from a Universal Value Function Approximator (UVFA) algorithm.

Each plot shows the proportion of choices (y-axis) that lead to each feature triplet (x-axis). Each panel shows the choice profile for an individual test task. The cue for an individual task was a weight vector, w, which is indicated in the plot titles. These cues were called market values during the gem collector game. Each panel includes three plots: the theoretical predictions of a UVFA algorithm, the theoretical predictions of an SF&GPI algorithm, and empirical choices from human participants. Bars indicate the standard error of the mean in plots with human data. UVFA predictions were generated post hoc using the implementation in [7]. Materials to reproduce this figure are available at https://gin.g-node.org/sam.hall-mcmaster/sfgpi-behavioural-analysis.

https://doi.org/10.1371/journal.pbio.3003174.s004

(TIF)

S5 Fig. Choices on individual test tasks were not explained by model-free perseveration.

A model-free algorithm learns the optimal policies on training tasks but has no way to evaluate their success on test tasks. The result is that the algorithm reuses the optimal training policies in an unselective fashion. Each plot shows the proportion of choices (y-axis) that lead to each feature triplet (x-axis). Each panel shows the choice profile for an individual test task. The cue for an individual task was a weight vector, w, which is indicated in the plot titles. These cues were called market values during the gem collector game. Each panel includes three plots: the theoretical predictions of a model-free algorithm, the theoretical predictions of an SF&GPI algorithm, and empirical choices from human participants. Bars indicate the standard error of the mean in plots with human data. Materials to reproduce this figure are available at https://gin.g-node.org/sam.hall-mcmaster/sfgpi-behavioural-analysis.

https://doi.org/10.1371/journal.pbio.3003174.s005

(TIF)

S6 Fig. Post scan feature estimates and confidence ratings.

Participants were asked to estimate the number of triangle, square, and circle gems that had appeared in each city during the final block of the scanning task. Estimates were restricted to a range between 0 and 250 gems. Participants were also asked to rate their confidence for each estimate on a scale from 0 (not confident at all) to 100 (completely confident). A: The mean absolute error (y-axis) for each feature triplet (x-axis). The mean error in each bar is an average across the three estimates provided for each triplet. No significant differences in mean absolute error were detected when correcting for multiple comparisons (all corrected p-values > 0.09). B: The relationship between the mean error for feature triplets associated with the suboptimal training policies and policy reuse. Feature triplets 2 and 3 were associated with suboptimal training policies. Policy reuse is the proportion of test trials in which the more rewarding optimal training policy was selected. We detected a significant correlation between these variables after correcting multiple comparisons, suggesting that individuals with less accurate estimates of features linked to the suboptimal training policies were less likely to select those policies on test trials. C: The same plot as B, except showing the mean absolute error for feature triplets associated with the optimal training policies. D: Mean confidence ratings (y-axis) for each feature triplet (x-axis). Mean confidence in each bar is an average across the three ratings provided for each triplet. Participants were significantly more confident in their ratings for feature triplets associated with the optimal training policies [ɸ(1) and ɸ(4)] compared to ɸ(3) which was associated with one of the suboptimal training policies (t(37)_{ɸ(1) vs. ɸ(3)} = 3.08, corrected p = 0.046; t(37)_{ɸ(4) vs. ɸ(3)} = 3.91, corrected p = 0.0054, all remaining corrected p-values > 0.05). E: The relationship between participants’ confidence in their estimates for the features associated with suboptimal training policies and the proportion of test trials in which the more rewarding optimal training policy was selected. F: The same plot as E, except showing the participants’ confidence in their estimates for the features associated with the optimal training policies. A, D: Error bars indicate standard error of the mean. B, C, E, F: Black lines indicate linear fits to the data and gray lines indicate 95% confidence intervals of the fits. A–F: Significance thresholds were corrected for 14 exploratory tests (5 pairwise comparisons for each bar plot and four correlations). The Bonferroni–Holm correction was used to correct for multiple comparisons. Materials to reproduce this figure are available at https://gin.g-node.org/sam.hall-mcmaster/sfgpi-behavioural-analysis.

https://doi.org/10.1371/journal.pbio.3003174.s006

(TIF)

S7 Fig. Validation of the neural decoding pipeline.

To validate our decoding approach, we trained neural decoders to distinguish the city observed during feedback on training trials. We then tested how well we could recover the chosen city on held out training trials. One functional run was held out at a time to assess decoding performance. As training trials were not related to our hypotheses, we used the validation procedure to explore three possible time shifts (+4 s, + 5 s, + 6 s following feedback onset) and three possible smoothing levels (0 mm, 2 mm, 4 mm) that could be used to optimize decoder training. The validation results were based on 25 iterations. Data were subsampled at random on each iteration to match the number of trials from each class. The highest-performing decoding parameters were selected for each ROI based on the validation results. These parameters were later used when testing our hypotheses about neural reactivation during test tasks. A: Validation results for occipitotemporal cortex (OTC). The left subplot shows the mean decoding evidence for the selected city on held-out training trials for each combination of decoding parameters. The mean is taken across a 5 s window (3.75–8.75 s) following feedback onset. Bars indicate the standard error of the mean. The blue box denotes the decoding parameters with the highest validation performance. The right subplot shows the decoding time course based on those parameters. B–D: The remaining panels follow the same structure as A but present validation results for different brain regions. These include the medial temporal lobe (MTL) in B, the orbitofrontal cortex (OFC) in C, and the dorsolateral prefrontal cortex (DLPFC) in D. A–D: Materials to reproduce this figure are available at https://gin.g-node.org/sam.hall-mcmaster/sfgpi-neural-analysis.

https://doi.org/10.1371/journal.pbio.3003174.s007

(TIF)

S8 Fig. Comparison of cue-locked and feedback-locked decoding.

It is possible that the activation of city stimuli could occur directly following task cue onset. We therefore performed additional validation tests, examining the decoding evidence on training tasks when decoder training was locked to task cue onset, rather than feedback onset. Decoding based on feedback onset was found to be more effective in all ROIs. Panels A–D were generated using the same approach described in S7 Fig. For cue-locked decoding, the decoder was trained using data with a time shift of +4 s, + 5 s, or +6 s from cue onset and evaluated from cue onset on held-out trials. Decoding evidence above represents evidence for the city selected on each training trial, averaged over a 5 s window (3.75–8.75 s) following the relevant event onset (cue presentation or feedback presentation). A: Validation results for occipitotemporal cortex (OTC). Bars indicate the standard error of the mean. The blue box denotes the decoding parameters with the highest validation performance. B–D: The remaining panels follow the same structure as A but present validation results for different brain regions. These include the medial temporal lobe (MTL) in B, the orbitofrontal cortex (OFC) in C, and the dorsolateral prefrontal cortex (DLPFC) in D. A–D: Materials to reproduce this figure are available at https://gin.g-node.org/sam.hall-mcmaster/sfgpi-neural-analysis.

https://doi.org/10.1371/journal.pbio.3003174.s008

(TIF)

S9 Fig. Decoding evidence for policies on each individual test task.

Each panel shows average decoding evidence during a specific test task (y-axis) as a function of brain region (x-axis). Colors denote different policy categories, and small translucent circles show data from individual participants. Panel B is especially relevant for testing whether the neural data are more consistent with SF&GPI or Universal Value Function Approximation (UVFA). The theoretical choice profiles for SF&GPI and UVFA are distinct for test task w = [1,1,−1] (S4 Fig). The theoretical choice profile for UVFA predicts: 1) that the more rewarding training policy and objective best policy will be activated on test task w = [1,1,−1] and 2) that the decoding evidence for these two policies will be comparable in magnitude. In contrast, we found that the more rewarding training policy was activated in OTC (t(37) = 3.87, corrected p = 0.003) but did not detect evidence that the objective best policy was activated (t(37) = −0.90, corrected p = 0.750). Decoding evidence for the more rewarding training policy was also significantly higher than the objective best policy (t(37) = 3.21, corrected p = 0.014). Equivalent tests in DLPFC were not significant (t-values < 1.66, corrected p-values > 0.094). These follow-up tests were restricted to OTC and DLPFC based on the decoding results in Fig 3 of the main text. p-values were corrected for six tests using the Bonferroni–Holm correction (2 policy categories × 2 ROIs and two follow-up tests assessing the difference in decoding evidence in each ROI). Materials to reproduce this figure are available at https://gin.g-node.org/sam.hall-mcmaster/sfgpi-neural-analysis.

https://doi.org/10.1371/journal.pbio.3003174.s009

(TIF)

S10 Fig. Associative memory paradigm used to train the feature decoders.

On each trial, participants were shown a retrieval cue and need to select a corresponding target stimulus. Targets were either feature numbers or cities that would later appear during the gem collector game. Participants were pre-trained on the associations before scanning, and two retrieval cues were used per target stimulus. Feature decoders were then trained using neural patterns that arose from the response phase onset. For a full description, please see Session One and Feature Decoding in the Methods.

https://doi.org/10.1371/journal.pbio.3003174.s010

(TIF)

S11 Fig. Validation tests of feature decoding during the associative memory paradigm.

To test the validity of our feature decoding approach, we trained neural decoders to distinguish the target number presented during the response phase. Response phase onset was used for training to be as consistent as possible with the policy decoding approach in S7 Fig, given the associative memory paradigm did not have feedback after correct responses. The validation tests used the optimal smoothing and time shift for each ROI identified in S7 Fig. Once the feature decoders were trained, we tested how well we could recover the target number on held-out trials. One functional run was held out at a time to assess decoding performance. The process was repeated 25 times. Data were subsampled at random on each iteration to match the number of trials from each target class. A: Decoding time course for occipitotemporal cortex (OTC). Bars indicate the standard error of the mean. The dotted line indicates the chance based on 12 possible classes. B–D: The remaining panels follow the same structure as A but present validation results for different brain regions. These include the medial temporal lobe (MTL) in B, the orbitofrontal cortex (OFC) in C, and the dorsolateral prefrontal cortex (DLPFC) in D. The target number could be recovered in OTC, OFC, DLPFC but not MTL using this approach. Materials to reproduce this figure are available at https://gin.g-node.org/sam.hall-mcmaster/sfgpi-neural-analysis.

https://doi.org/10.1371/journal.pbio.3003174.s011

(TIF)

S12 Fig. Decoding evidence for feature triplets on test tasks.

Another neural hypothesis formulated based on the SF&GPI algorithm is that feature triplets associated with the optimal training policies should be reactivated on test trials. This figure shows average decoding evidence for features associated with the more and less rewarding training policies on test trials (y-axis) as a function of brain region (x-axis). Feature information could not be decoded above chance in the four brain regions of interest (corrected p-values > 0.05). Materials to reproduce this figure are available at https://gin.g-node.org/sam.hall-mcmaster/sfgpi-neural-analysis.

https://doi.org/10.1371/journal.pbio.3003174.s012

(TIF)

S13 Fig. Comparison of city decoders trained on session one and session two data.

Target cities could be recovered based on neural activity in all ROIs during the associative memory paradigm (session one). However, decoding evidence for cities during the gem collector paradigm (session two) was often higher when decoders were trained on the data from that same paradigm. A: Decoding evidence in OTC. The left subplot shows decoding evidence for target cities during the associative memory paradigm. The right subplot shows decoding evidence for cities selected on training trials of the gem collector paradigm, when the decoder was trained on data from session one (blue) or session two (orange). Bars indicate the standard error of the mean, and the dotted line indicates chance. B–D: Panels follow the same structure as A but present decoding time courses for different ROIs. A–D: Decoder training used the optimal smoothing and time shift for each ROI identified in S7 Fig, and 25 subsampling iterations were performed to match the number of trials from each target class. Decoding evidence presented above is cross-validated, based on assessment on held-out test data. Materials to reproduce this figure are available at https://gin.g-node.org/sam.hall-mcmaster/sfgpi-neural-analysis.

https://doi.org/10.1371/journal.pbio.3003174.s013

(TIF)