INTRODUCTION
Imagine going for a walk in the park on a sunny Saturday morning. As you stroll along, you notice many other people who are likewise strolling about. However, you are not just passively observing them; instead, you are continuously trying to anticipate or predict what they are about to do, so as to avoid any awkward collisions. As you exit the park and cross the street busy with cars, you are again engaged in prediction, this time to avoid more serious collision. As this scenario makes clear, adaptive interaction with the environment requires prediction of both human actions and non-biological motion, and implementing these tasks is, therefore, a central function of the human brain (Bubic, von Cramon, & Schubotz, Reference Bubic, von Cramon and Schubotz2010; Clark, Reference Clark2015; see also Fuster & Bressler, Reference Fuster and Bressler2015).
According to one influential account, action prediction is instantiated by internal forward models that are used to predict the proprioceptive and exteroceptive sensory consequences of one’s own motor commands. This prediction is compared with the actual movement as it unfolds, and any discrepancy between the two is used for online correction and learning of movements (Wolpert, Ghahramani, & Jordan, Reference Wolpert, Ghahramani and Jordan1995). It has been argued that the same forward models can also be deployed to simulate others’ actions so as to predict how they will unfold (Wolpert, Doya, & Kawato, Reference Wolpert, Doya and Kawato2003; see also Friston, Mattout, & Kilner, Reference Friston, Mattout and Kilner2011). By contrast, the prominent account of Pickering and Garrod (Reference Pickering and Garrod2013) suggests that prediction may not always use forward models, but may under some circumstances (such as during prediction of non-biological motion) rest upon a perceptual associative process. In the latter case, observers compare the current trajectory with past perceived trajectories and in this way come to predict how it will unfold (see also Stadler, Springer, Parkinson, & Prinz, Reference Stadler, Springer, Parkinson and Prinz2012).
There is extensive behavioral evidence supporting the close link between action production and prediction (e.g., Aglioti, Cesari, Romani, & Urgesi, Reference Aglioti, Cesari, Romani and Urgesi2008; Balser et al., Reference Balser, Lorey, Pilgramm, Stark, Bischoff, Zentgraf and Munzert2014; Brattan, Baker, & Tipper, Reference Brattan, Baker and Tipper2015; Calvo-Merino, Glaser, Grèzes, Passingham, & Haggard, Reference Calvo-Merino, Glaser, Grèzes, Passingham and Haggard2005; Colling, Thompson, & Sutton, Reference Colling, Thompson and Sutton2014; Knoblich & Flach, Reference Knoblich and Flach2001; Makris & Urgesi, Reference Makris and Urgesi2013). For example, superior prediction performance has been reported for videos of self compared to other-generated actions, possibly because the used forward models are more accurate in the former case (Colling et al., Reference Colling, Thompson and Sutton2014; Knoblich & Flach, Reference Knoblich and Flach2001). On the other hand, there is evidence that subjects may use a visual matching strategy to predict actions, suggesting the possibility that the task does not necessarily require involvement of the motor system (Springer, Brandstädter, & Prinz, Reference Springer, Brandstädter and Prinz2013; see also Springer & Prinz, Reference Springer and Prinz2010).
Several brain imaging studies suggest the existence of a fronto-parietal network that supports human action prediction. Typical activations include premotor cortex (PM), supplementary motor area (SMA), inferior frontal gyrus (IFG), supramarginal gyrus (SMG), superior parietal lobule (SPL), and the cerebellum, with increasing left-lateralization for cortical regions as a function of motor expertise and/or visual familiarity (see Yang, Reference Yang2014 for a meta analysis). Whether prediction of non-biological motion trajectories is subserved by overlapping neural mechanisms is an issue of contention.
According to Schubotz (Reference Schubotz2007), while forward models in PM cortex may be ideally suited to produce and predict human actions, they are nevertheless also the best available neural resource for predicting many other events beyond (visually perceived) human actions. In line with this proposal, Coull, Vidal, Goulon, Nazarian, and Craig (Reference Coull, Vidal, Goulon, Nazarian and Craig2008) found activations in left pars opercularis (a portion of IFG), ventral PM, and SMG during collision judgments of inanimate objects. At variance with these data are studies that report differences between prediction of human and non-biological motion trajectories, with shared activity in left supplementary motor area (SMA) but additional activations in right extrastriate body area (EBA) in the former, and left lateral occipital cortex (LOC) in the latter case (Cross, Stadler, Parkinson, Schütz-Bosbach, & Prinz, Reference Cross, Stadler, Parkinson, Schütz-Bosbach and Prinz2013; see also Stadler, Springer, et al., Reference Stadler, Springer, Parkinson and Prinz2012).
One important limitation of brain imaging studies is that they cannot provide information about the necessity of activated brain regions for a given task (cf. Caramazza, Anzellotti, Strnad, & Lingnau, Reference Caramazza, Anzellotti, Strnad and Lingnau2014; Fellows et al., Reference Fellows, Heberlein, Morales, Shivde, Waller and Wu2005). In this regard, the study of brain lesion patients is highly informative. In one of the few available patient studies addressing the neural substrate of prediction, Schubotz, Sakreida, Tittgemeyer, and von Cramon (Reference Schubotz, Sakreida, Tittgemeyer and von Cramon2004) instructed right or left parietal- or premotor-lesioned patients, as well as right, left, and bilateral prefrontal-lesioned patients to predict sequences of geometrical shapes and spatial locations. They found impaired performance in both the parietal and premotor patients, whereas the prefrontal-lesioned patients performed normally, despite having larger lesion volumes. These data suggest that prediction of sequential information critically depends on parietal and premotor (but not prefrontal) cortices. Roth, Synofzik, and Lindner (Reference Roth, Synofzik and Lindner2013) extended these findings to include an additional critical role for the cerebellum. Neither study assessed prediction of human action stimuli.
Patient studies are also useful for informing the question of whether the proposed relationship between action production and prediction is contingent on spatiomotor planning abilities, motor production abilities, or both. While these questions have not been addressed in patient populations, related research has focused on the ability of apraxic patients (known to have deficient spatiomotor planning) and hemiparetic patients (who exhibit impaired motor production) to simulate actions, which many theorists consider a precursor to or component of prediction (Barsalou, Reference Barsalou2009; Mulligan, Lohse, & Hodges, Reference Mulligan, Lohse and Hodges2015; Stapel, Hunnius, Meyer, & Bekkering, Reference Stapel, Hunnius, Meyer and Bekkering2016; but see Vannuscorps & Caramazza, Reference Vannuscorps and Caramazza2015).
For example, we (Buxbaum, Johnson-Frey, & Bartlett-Williams, Reference Buxbaum, Johnson-Frey and Bartlett-Williams2005; Dawson, Buxbaum, & Duff, Reference Dawson, Buxbaum and Duff2010; see also Coslett, Buxbaum, & Schwoebel, Reference Coslett, Buxbaum and Schwoebel2008) showed that patients with left parietal stroke and limb apraxia, a disorder of skilled action associated with prominent bimanual deficits in tool pantomime (but better performance with tools in hand), were unable to reliably indicate how they would grasp a dowel if they were to reach out and perform the action, whereas they performed normally in actual grasping. These data suggest that patients with limb apraxia may have deficient predictive models of action in the face of relatively intact feedback-driven motor control.
The evidence with regard to hemiparesis is inconclusive, with reports of intact simulation in hemiplegic stroke patients (Johnson, Sprehn, & Saykin, Reference Johnson, Sprehn and Saykin2002; see also Sirigu et al., Reference Sirigu, Cohen, Duhamel, Pillon, Dubois, Agid and Pierrot-Deseilligny1995), and conversely, of impairment in hemiparetic individuals born with cerebral palsy, especially for simulated actions of the affected hand (van Elk et al., Reference van Elk, Crajé, Beeren, Steenbergen, van Schie and Bekkering2010). In at least some circumstances, then, motor planning and production abilities may be associated with the ability to simulate one’s own actions. However, to our knowledge, the relationship between motor planning or production deficits and the ability to predict either the observed actions of others or non-biological motion trajectories has not been directly assessed.
An additional question of interest concerns the relationship between action recognition and understanding and prediction of others’ actions. On the one hand, if action perception relies upon direct matching through simulation processes in the motor system, as has been suggested (e.g., Rizzolatti, Fogassi, & Gallese, Reference Rizzolatti, Fogassi and Gallese2001), and simulation is, in turn, a component of prediction, then a strong relationship may be expected (Urgesi, Candidi, & Avenanti, Reference Urgesi, Candidi and Avenanti2014). That is, on such accounts, one should be impaired at recognizing actions that one cannot simulate/predict. However, according to Csibra (Reference Csibra2008), action recognition and understanding is instantiated through top–down processing in temporal and inferior frontal association cortices resulting in a bias toward expecting certain outcomes, with the output feeding into the motor system to enhance its efficiency in simulating and hence predicting actions. Given that this account argues that action recognition is not dependent on simulation (but rather that simulation is influenced by recognition), it may assume a weaker relationship between recognition and prediction.
The aims of the present study were to identify critical neural substrates of prediction of observed human and non-biological motion trajectories and to assess the behavioral correlates of this ability. We used a video occlusion paradigm (e.g., Stadler et al., Reference Stadler, Schubotz, von Cramon, Springer, Graf and Prinz2011) to examine motion prediction ability in chronic left-hemisphere stroke patients as well as in a matched sample of neurologically intact controls. Patients exhibited a full range of hemiparesis and limb apraxia.Footnote 1 We also included our laboratory’s measure of patients’ action recognition impairment (Kalénine, Buxbaum, & Coslett, Reference Kalénine, Buxbaum and Coslett2010; Tarhan, Watson, & Buxbaum, Reference Tarhan, Watson and Buxbaum2015). Video stimuli in the prediction task consisted of pantomimed and actual tool use actions, and non-biological motions of comparable complexity.
Based on prior neuroimaging studies that found activation of left SMA, PM, IFG, IPS, and SMG during prediction of either human or non-biological motion (Cross et al., Reference Cross, Stadler, Parkinson, Schütz-Bosbach and Prinz2013; Schubotz & von Cramon, Reference Schubotz and von Cramon2002; Stadler et al., Reference Stadler, Schubotz, von Cramon, Springer, Graf and Prinz2011; Wiener, Turkeltaub, & Coslett, Reference Wiener, Turkeltaub and Coslett2010; Yang, Reference Yang2014), we hypothesized that lesions in these regions would be associated with impairments in prediction of all three motion types. In terms of the behavioral correlates of action prediction ability, accounts that invoke forward models (Friston et al., Reference Friston, Mattout and Kilner2011; Wolpert et al., Reference Wolpert, Doya and Kawato2003) predict an association between action planning and action prediction. Thus, given that apraxia is thought to be, in part, a disorder of action planning (Buxbaum, Johnson-Frey, et al., Reference Buxbaum, Johnson-Frey and Bartlett-Williams2005; Eidenmuller, Randerath, Goldenberg, Li, & Hermsdorfer, Reference Eidenmuller, Randerath, Goldenberg, Li and Hermsdorfer2014; Jax, Buxbaum, & Moll, Reference Jax, Buxbaum and Moll2006; Mutha, Sainburg, & Haaland, Reference Mutha, Sainburg and Haaland2010; Ochipa et al., Reference Ochipa, Rapcsak, Maher, Rothi, Bowers and Heilman1997; Wheaton, Fridman, Bohlhalter, Vorbach, & Hallett, Reference Wheaton, Fridman, Bohlhalter, Vorbach and Hallett2009), such accounts would expect an association between praxis ability and motion prediction ability.
Under the assumption that forward models underlying action planning are used not only to predict human but also non-biological motions (Schubotz, Reference Schubotz2007), this association should hold for all three movement types. By contrast, the perceptual associative account of prediction of Pickering and Garrod (Reference Pickering and Garrod2013) argues that forward models are unlikely to be used for non-biological motions, and thus would presumably expect praxis ability to be associated with human but not non-biological motion prediction ability. Finally, based on evidence that hemiplegic stroke patients may display intact action simulation abilities (Johnson et al., Reference Johnson, Sprehn and Saykin2002), we hypothesized that there would not be an association between motor production ability (i.e., hemiparesis severity) and prediction ability for either human or non-biological motion trajectories.
METHODS
Participants
Forty chronic left-hemisphere stroke patients were recruited through a dedicated stroke patient research registry (Schwartz, Brecher, Whyte, & Klein, Reference Schwartz, Brecher, Whyte and Klein2005). All patients were premorbidly right-handed according to self-report and at least 6 months post-stroke at the time of testing. To increase the likelihood of task instruction comprehension, patients with scores≤4/10 (“severe” or “very severe” impairment) on the language comprehension subtest of the Western Aphasia Battery (WAB; Kertesz, Reference Kertesz1982) were excluded pre-data acquisition, as is customary in our lab (e.g., Watson & Buxbaum, Reference Watson and Buxbaum2015). Data collection for two patients was aborted because it was discovered that their vision was insufficient to perform the tasks, while two other patients were excluded for demonstrably not following task instructions.
For nine patients, prediction performance was at chance for all three conditions (i.e., tool use, pantomimed tool use, non-biological motion) according to binomial tests (α=.05). Because it is unclear whether this was due to inability to understand the task instructions or inability to perform the task, these patients were also excluded. The final sample comprised 27 patients. Table 1 shows the patients’ demographic, neuropsychological, and lesion information.
Demographic, neuropsychological, and lesion information
Note. Patients excluded from the final sample are flagged with asterisks. Scores of .00 in the grip strength and finger tapping frequency columns indicate patients’ inability to perform the task with the right hand.
rh/lh=right hand divided by left hand; n/a=datapoint not available; freq=frequency; comp=comprehension; vol=volume.
In addition, 16 neurologically intact control participants were recruited from a control subject research registry. All control participants were right-handed according to self-report and had a score of≥27 of 30 on the Mini-Mental State Exam (Folstein, Folstein, & McHugh, Reference Folstein, Folstein and McHugh1975). One control participant was excluded because her prediction performance was at chance in all three conditions. Two control participants were excluded for having an average score in one or more conditions of the prediction task more than two SDs below the group mean. There was no significant difference in age or years of education between the patients (41% female; mean age 58.7 years, range 35–76 years; mean education 14.7 years, range 11–27 years) and controls (38% female; mean age 60.2 years, range 38–80 years; mean education 15.9 years, range 12–22 years) included in the final sample (ps>.30).
Patient and control participants with a history of co-morbid or pre-morbid neurological disorders, alcohol or drug abuse, or psychosis were excluded from participating in the study. All participants gave informed consent for the behavioral portion of the experiment and were paid for participation and reimbursed for travel expenses. Thirty-eight patients additionally consented to participate in a structural magnetic resonance imaging (MRI) or computed tomography (CT) scanning protocol; brain scans of two patients were obtained from clinical records. The behavioral study was approved by the Institutional Review Board (IRB) of Einstein Healthcare Network and the scanning protocol was approved by the IRB of the Hospital of the University of Pennsylania.
Stimuli and Apparatus
Nine videos were constructed in which an actor at a desk performed left-handed everyday actions with a tool and its recipient object. In addition, nine videos were constructed showing the same actor performing pantomimed versions of the tool use actions. By flipping the videos 180° across the vertical axis, nine videos were created in which the same actions appeared to be performed with the right hand.Footnote 2 Furthermore, six non-biological motion sequences were constructed using wind-up toys (Kikkerland Design Inc., New York, NY) and six using a programmable robotic ball (Sphero, Boulder, CO). The non-biological motion sequences were filmed on the same desk and were also flipped across the vertical axis to create 12 additional videos.
For the human action sequences, the camera was angled at an approximately 45° angle relative to the actor’s body midline which ensured the maximal amount of visible movement during each action. Non-biological videos were filmed using the same camera angle. Videos were presented in color on an Acer G215H monitor at a resolution of 640×480 pixels and a frame rate of 30 frames per second (see Figure 1 for static images taken from the tool use, pantomimed tool use, and non-biological motion videos).
Frames from tool use (A), pantomimed tool use (B), and non-biological (C) motion videos. The human actions in panels A and B consist of (pantomiming) the use of a fork.
To select videos for the experimental protocol that were of appropriate difficulty, preliminary data on the tool use, pantomimed tool use, and non-biological motion prediction tasks were collected from 10 young neurologically intact participants (age range: 21–26 years) who were recruited in accordance with guidelines of the IRB of Einstein Healthcare Network (for task procedure, see Section Motion Prediction Task below). Based on these data, four tool use videos and matching pantomimed tool use videos (i.e., using an iron, fork, razor, wrench) as well as four non-biological videos that featured the robotic ball were selected for which performance was on average 81.5% correct and did not differ significantly between the three motion types (p=.75). Given that performance on many cognitive tasks decreases with age, videos were selected with high but not ceiling performance in young controls to increase the likelihood that prediction performance for patients and matched older controls would be above floor but below ceiling.
Each video was preceded by a 1500 ms fixation cross, followed by the beginning of the motion sequence. After a mean duration of 4167 ms (range: 2300–5100 ms), the videos were fully covered by a black occluder for a duration of 800 ms. Occlusion onset always occurred during a large amplitude phase of the movement (e.g., during the fork lifting phase rather than while picking up the food; during a rolling phase, rather than while the ball made a slow turn) and, therefore, varied between videos. At occlusion offset, the videos continued to play in a way that was either congruent with the motion sequence pre-occlusion (i.e., the videos continued to play normally during occlusion) or incongruent (i.e., the videos were shifted 800 ms into the future; see Graf et al., Reference Graf, Reitzner, Corves, Casile, Giese and Prinz2007; Stadler et al., Reference Stadler, Schubotz, von Cramon, Springer, Graf and Prinz2011, for similar paradigms).
Tool use, pantomimed tool use, and non-biological videos lasted ~6500 ms (range: 4600–8500 ms) and did not differ significantly in terms of total length, time from start of video until occlusion onset, and time from occlusion offset until end of video (all ps>.25). Each combination of stimulus factors, motion type, video orientation, and congruence, was presented three times, leading to a total of 12 (4 tool use, 4 pantomimed tool use, 4 non-biological)×2 (flipped, non-flipped)×2 (congruent, incongruent)×3 repetitions=144 experimental trials. The human (tool use and pantomimed tool use) and non-biological motion sequences were presented separately within two counterbalanced blocks of 96 and 48 randomly presented stimuli, respectively.
E-Prime 2.0 stimulus presentation software (Psychology Software Tools, Pittsburgh, PA) was used to control stimulus presentation and a yellow and a blue colored response key on an E-Prime serial response box were used to collect responses and associated reaction times (RT). A lever attached to a mechanical counter (Veeder-Root, Hartford, CT) was used to collect finger tapping frequency data, and a Jamar dynamometer (Sammons Preston Rolyan, Bolingbrook, IL) to collect grip strength data. Ten three-dimensional familiar tools (fork, razor, scissors, watch, toothbrush, comb, bottle opener, cigarette lighter, eraser, and nail clippers) were used to collect data on pantomimed tool use production impairment.
Procedure
Motion prediction task
Participants first learned which response key to press for stimuli that they perceived as congruent (“in time”) and which key to press for stimuli that they perceived as incongruent (“not in time”). Stimuli in this practice block consisted of the printed word “in time,” in response to which participants were required to press the key associated with congruent stimuli, or the printed word “not in time,” in response to which participants were required to press the key associated with incongruent stimuli. Stimulus-response mappings were counterbalanced between participants. This preliminary key-mapping practice was repeated if necessary until participants reached a criterion of≥13 of 16 trials correct. All participants used their left hand to provide the responses, since the right hand is potentially paretic following left-hemisphere stroke.
In the second practice task, participants were presented with video stimuli, first in unoccluded form to familiarize participants with the appearance of the motion trajectories, followed by (occluded) congruent and incongruent versions of the same stimuli. Participants then performed a 32 trial video practice block in which they were randomly presented with the congruent and incongruent video stimuli and were instructed to indicate whether they perceived the stimulus to be “in time” (i.e., congruent) or “not in time” (i.e., incongruent). Participants were instructed to respond as quickly and accurately as possible and received performance feedback after each response. Trials timed out 5 s after video offset, at which time a prompt would appear asking the participant to attempt to respond faster.
Participants were encouraged to ask questions about the task requirements at any time during practice, in which case the trial was paused manually and the experimenter provided additional instruction. The stimuli used in the practice block were not used in the experimental block. At the end of practice, the experimental stimuli were played once in unoccluded form. Next, the first experimental block was administered. The task was the same as during video practice, except that performance feedback was not provided. Participants received a ~2 minute break after every 48 trials.
Additional Behavioral Tasks
Pantomimed tool use production task
Pantomime production was used as a measure of praxis ability. Participants were presented with one tool at a time. The instruction was to show how to correctly use the tool, without actually holding it, using the non-paretic left hand. Participants completed one practice trial with feedback from the experimenter. The 10 experimental trials were video recorded and scored offline for accuracy according to a detailed error taxonomy by a trained coder who obtained at least 85% agreement with previous coders (Buxbaum, Kyle, & Menon, Reference Buxbaum, Kyle and Menon2005). Based on normative performance, scores<.895 are abnormal (Buxbaum, Kyle, et al., Reference Buxbaum, Kyle and Menon2005; see Table 1).
Grip strength and finger tapping tasks
To obtain an index of motor production ability, grip strength and finger tapping frequency were measured in both hands. Three grip strength trials were performed for each hand and the ratio of mean grip force (kg) for the left versus right hand was obtained. For finger tapping frequency measurements, patients were instructed to tap as fast as they could on the counter’s lever for 10 s using their index finger. Two trials were performed for each index finger, and the ratio of mean tapping speed for the left versus right hand was obtained (see Table 1).
Pantomimed tool use recognition task
To assess participants’ ability to correctly identify pantomimed tool use actions, a 24-item pantomimed tool use recognition task was administered (Buxbaum, Kyle, et al., Reference Buxbaum, Kyle and Menon2005; Kalénine et al., Reference Kalénine, Buxbaum and Coslett2010). Participants heard an action verb-derived noun in the gerund form repeated twice, and simultaneously viewed the written noun (e.g., sawing). Next, they were presented with two videos of an actor performing pantomimed tool use actions. In one video, the action matched the noun (e.g., sawing), while the other did not because of a spatially incorrect hand posture, arm posture, or incorrect amplitude and timing. The task was to select the action that matched the noun. (A pre-test ensured that participants comprehended the verbs, see Buxbaum, Kyle, et al., Reference Buxbaum, Kyle and Menon2005.) The order of the target and foil action within a trial was randomized. There were no time constraints for responding. Based on normative data, scores<.87 are abnormal (Buxbaum, Kyle, et al., Reference Buxbaum, Kyle and Menon2005; see Table 1).
Imaging Methods
For 20 patients, research quality high-resolution whole-brain T1-weighted magnetic resonance (MR) images were collected on a 3T scanner (Siemens Trio, repetition time=1620 ms, echo time=3.87 ms, field of view=192×256 mm, 1×1× 1 mm voxels) or a 1.5 T scanner (Siemens Sonata, repetition time=3000 ms, echo time=3.54 ms, field of view=24 cm, 1.25×1.25×1.2 mm voxels) using a Siemens eight-channel head coil. Seven patients were contraindicated for MR imaging and underwent whole-brain research quality computed tomography (CT) scans without contrast (60 axial slices, 3–5 mm slice thickness) on a 64-slice Siemens SOMATOM Sensation scanner.
For patients with research quality MR scans, lesions were segmented manually by trained research assistants on the patients’ T1-weighted MR images. Images were then registered to standardized space [the Montreal Neurological Institute (MNI) “Colin27” template brain; Holmes et al., Reference Holmes, Hoge, Collins, Woods, Toga and Evans2015] using a symmetric diffeomorphic registration algorithm (Avants, Schoenemann, & Gee, Reference Avants, Schoenemann and Gee2006; http://www.picsl.upenn.edu/ANTS). Subsequently, each lesion map was binarized so that lesioned voxels had a value of 1, and preserved voxels had a value of 0. To ensure that no errors had occurred during this process, lesion maps were inspected by the team neurologist who was naive to the behavioral data. For patients with CT scans, the team neurologist drew the lesions directly onto the Colin27 template brain using MRIcron (http://www.mccauslandcenter.sc.edu/mricro/mricron/index.html) after rotating its pitch to match the pitch of the patient’s scan. Schnur et al. (Reference Schnur, Schwartz, Kimberg, Hirshorn, Coslett and Thompson-Schill2009) have previously shown high intra and inter-rater reliability for this method.
Lesion Proportion Difference Analysis
To identify brain regions that are critically involved in motion prediction, a lesion proportion difference analysis was performed to identify voxels that are significantly more likely to be lesioned in low performing as compared to high performing patients. The 27 patients were divided into three groups of nine patients on the basis of their d’ score (a measure of participants’ sensitivity to stimuli that is unaffected by response bias) averaged across the three motion types (tool use, pantomimed tool use, non-biological). The nine lowest and nine highest scoring patients were included. Following Mirman and Graziano (Reference Mirman and Graziano2013), we excluded the middle third of patients to contrast severely impaired patients with mildly impaired/unimpaired patients, eliminating patients whose scores were ambiguous.Footnote 3
We included voxels lesioned in at least 10% of these 18 patients (2 or more). Setting a lower limit for the inclusion of voxels in the analysis is a common procedure in lesion analysis, used to ensure the stability of results (Kemmerer, Rudrauf, Manzel, & Tranel, Reference Kemmerer, Rudrauf, Manzel and Tranel2012). Figure 2 shows an overlap map on the Colin27 template of the lesions of the 18 patients included in the analysis. There is good coverage of the parietal and frontal lobes, the regions of greatest interest in this study.
Overlap map of lesions included in the lesion proportion difference analysis (n=18, i.e., the 9 highest and 9 lowest scoring patients on the prediction task). The map is displayed on the Colin27 template with z-coordinates of horizontal slices corresponding to MNI standardized space. The color bar represents the number of patients with lesions in a particular voxel (min=2; max=18). The cortical surface rendering is displayed at a search depth of 8 voxels.
For each of the included voxels, the proportion of patients with low prediction performance and a lesion in the voxel and the proportion of patients with high prediction performance and a lesion in the voxel was determined, and the observed distributions were compared to the expected distributions under the null hypothesis, resulting in a χ2 value for each voxel. The voxel-wise χ2 value map was then thresholded to 6.6349, which is the value corresponding to a significant χ2 test result at α=.01 and 1 degree of freedom (see Kemmerer et al., Reference Kemmerer, Rudrauf, Manzel and Tranel2012; Mirman & Graziano, Reference Mirman and Graziano2013 for more details of this approach). The Automated Anatomical Labeling (AAL; Tzourio-Mazoyer et al., Reference Tzourio-Mazoyer, Landeau, Papathanassiou, Crivello, Etard, Delcroix and Joliot2002) map in MRIcron was used to determine the neuroanatomic loci of voxels that surpassed the threshold. Significant clusters≥100 voxels were subsequently used as predictors of behavioral performance in regression analyses (see below).
Multilevel Regression Analysis
Participants’ ability to distinguish between congruent and incongruent stimuli on the prediction task was indexed by d’. RT data for correct responses were also collected. Trials in which button presses occurred before occlusion offset or less than 250 ms after occlusion offset were removed from the analysis, as were trials with RTs>2 SDs from the mean RT for that participant and condition. This led to elimination of approximately 5% of RTs of all correct responses for both patients and controls.
RT and d’ were analyzed using a multilevel regression approach in which prediction performance was modeled as a function of fixed behavioral and neuroanatomical factors of interest (to be described below), random intercepts for participants, and random slopes where possible. That is, because there is only one d’ value per cell of the design (one d’ per subject per condition), adding random slopes would in this case result in oversaturated models, precluding or leading to unreliable estimation of model parameters (e.g., Bates, Kliegl, Vasishth, & Baayen, Reference Bates, Kliegl, Vasishth and Baayen2015). Hence, we only included random slopes in the analysis of RTs, which was performed over unaggregated data.
The regressions were conducted with the “lme4” package in the R software environment (version 2.15.3; R Core Team, 2015). To detect and remove cases with undue influence on the multilevel model parameter estimates, the “influence.ME” package was used (Nieuwenhuis, te Grotenhuis, & Pelzer, Reference Nieuwenhuis, Te Grotenhuis and Pelzer2012). Cases were regarded as too influential if their associated values of Cook’s distance were larger than 4/n, where n=number of participants. Model results presented below are based on data excluding any influential cases. To assess whether fixed factors significantly contributed to model fit, they were added individually and comparisons of models with and without the fixed factor were performed with likelihood ratio tests in which the statistic is −2 times the change in model log-likelihood, distributed as χ2 with degrees of freedom equal to the change in number of estimated parameters.
The p-values for the parameter estimates of the fixed factors were calculated using Satterthwaite’s approximation for degrees of freedom. We also report 95% confidence intervals (CI) around the parameter estimates. Marginal R 2 (R 2 LMM(m)), which denotes the proportion of variance explained by the fixed factor(s) of a multilevel regression model (see Johnson, Reference Johnson2014; Nakagawa & Schielzeth, Reference Nakagawa and Schielzeth2013), is reported as a measure of effect size.
In the first analysis, patients’ behavior was compared to controls with model comparisons that assessed the fixed factors Group (patient, control), Motion Type (tool use, pantomimed tool use, non-biological), and their interaction. A second analysis focused on the behavioral predictors of patient performance. Thus, we assessed the continuous behavioral predictors described in the Section Additional Behavioral Tasks (i.e., Praxis Ability, Grip Strength, Finger Tapping Frequency, and Pantomime Recognition) in separate model comparisons for each predictor. We also entered interaction terms for the behavioral predictors and the factor Motion Type to assess whether the strength of the behavioral predictors was conditioned by motion type.
A third regression analysis followed up on the results of the lesion proportion difference analysis performed on patients’ average d’ scores. First, we computed for each patient the % damage to each of the significant voxel clusters identified in that analysis. Subsequently, we performed separate model comparisons for each neuroanatomical cluster predictor to assess main effects of % Damage to Cluster and interactions with the factor Motion Type.
The fourth and final analysis assessed the behavioral and neuroanatomical cluster predictors simultaneously to determine whether behavior, lesion, or both influenced prediction performance when they were considered together. We first tested for the presence of a Pearson correlation between each of the behavioral and neuroanatomical cluster predictors. In case of a significant correlation, we assessed the predictive strength of the correlated predictors together in one multilevel regression model. All of the regressions in the second, third, and fourth analysis included the covariate TLV (see Table 1) to control for general effects of stroke severity on performance. The levels “control” and “non-biological” of the categorical fixed factors were treated as the baseline (reference) and model parameters were estimated for the levels “patient”, “tool use”, and “pantomimed tool use.”
RESULTS
Behavioral Results
Patients versus controls
In the model comparing d’ of patients and controls, including the predictor Group (patient, control) improved model fit relative to a reduced model that included only a random intercept for each participant. Patients were overall less accurate than controls. Adding the interaction term Group×Motion Type (tool use, pantomimed tool use, non-biological) relative to a reduced model that included only main effects for Group and Motion Type and random intercepts also improved model fit. Figure 3 shows the observed mean d’ scores for patients and controls separately for tool use, pantomimed tool use and non-biological motions. Table 2 shows the χ2 test results for each of the model comparisons, the marginal R 2 of the full models, as well as the β estimates with associated SE, 95% CI and p value of each pairwise comparison of interest. The pairwise comparisons indicate that the interaction is driven by the fact that the difference between predicting non-biological motions versus pantomimed actions as well as between tool use versus pantomimed actions was significantly different between controls and patients, while the difference between non-biological motions versus tool use actions was not.
Observed mean prediction performance as indexed by d’ for Patients (right panel) and Controls (left panel) plotted separately for each level of Motion Type (tool use, pantomimed tool use, non-biological; see legend). Error bars represent ±1 SE.
χ2 test results for the analyses comparing patients’ prediction performance with that of controls (see body text for description of reduced models)
Note. The marginal R 2 of the full models, as well as β estimates with associated SE, 95% CI, and p value for the pairwise comparison(s) of interest are also displayed.
We also tested models with the same predictors, but with RT as the dependent variable. Adding the predictor Group improved model fit relative to a reduced model that included only random intercepts and random slopes for participants [χ2(1)=11.07; p<.001]. Patients were slower overall than controls. Inclusion of the interaction Group×Motion Type did not significantly improve model fit relative to the reduced model without the interaction term (p=.707). Given the lack of effects on RT beyond the expected main effect of Group, the remaining analyses focus on d’.
Analysis of behavioral predictors
Using the data from patients only, we next examined the behavioral predictors Praxis Ability, Grip Strength, Finger Tapping Frequency, and Pantomime Recognition to evaluate their association with patients’ prediction d’ scores. A strong positive Pearson correlation was observed between grip strength and finger tapping frequency (r=.82; p<.001). We, therefore, used the average of these measures as a composite index of Motor Production Ability. As shown in Table 3, including the predictors Motor Production Ability and Praxis Ability (in separate models) significantly improved the fit of these models relative to reduced models that included TLV and random intercepts. This indicates that motion prediction performance is positively associated with both motor production ability and praxis ability, even when controlling for general non-specific effects of stroke severity (as indexed by TLV). By contrast, including the predictor Pantomime Recognition did not significantly improve model fit relative to a reduced model that included TLV and random intercepts.
χ2 test results for model comparisons assessing main effects of the behavioral predictors of patients’ prediction performance (see body text for description of reduced models)
Note. The marginal R 2 of the full model, as well as β estimates with associated SE, 95% CI, and p value of the behavioral predictors are also displayed.
We next assessed (separately for each of the three behavioral predictors) whether adding the interaction term behavioral predictor×Motion Type improved model fit relative to reduced models that included only main effects for the behavioral predictor, Motion Type, TLV, and random intercepts. None of these model comparisons approached significance (all ps>.486). This indicates that reduced motor production ability and praxis ability were equivalently associated with deficits in all three prediction conditions.
Neuroanatomical Results
Lesion proportion difference analysis
As shown in Figure 4, the lesion proportion difference analysis identified three neuronatomical clusters≥100 voxels that, when damaged, are associated with low mean d’ scores collapsed across tool use, pantomimed tool use and non-biological motion types. A dorsal cluster in frontal cortex included voxels in the precentral gyrus and middle frontal gyrus, extending slightly into the postcentral gyrus (81 voxels). A second cluster included voxels in the insula. Finally, a region in the parietal cortex included voxels in the inferior parietal lobe, superior parietal lobe and the postcentral gyrus. Table 4 lists the coordinates of the peak value of each of the three clusters.
Lesion proportion difference map for mean prediction performance (d’) displaying voxels associated with χ2 values surpassing a statistical threshold for significance (α=.01). Voxels are displayed on the Colin27 template; z-coordinates of horizontal slices correspond to MNI standardized space. The cortical surface rendering is displayed at a search depth of 8 voxels. Color bars represent χ2 value ranges.
Significant (α=.01) clusters from the lesion proportion difference analysis
Note. X, Y, and Z coordinates represent the location of the peak value within each cluster in MNI standardized space. Only clusters containing≥100 voxels are reported.
Analysis of neuroanatomical predictors
We next assessed (in separate models) whether damage to these clusters selectively affected the ability to predict the three motion types. For each cluster, the comparison between a reduced model that only included random intercepts and TLV and a full model that additionally included a main effect of % Damage to Cluster was significant, corroborating (not surprisingly) the results of the lesion proportion difference analysis. Of greater interest, follow-up model comparisons assessed, for each cluster, whether damage was differentially associated with deficits in prediction of the different motion types by comparing reduced models that included only main effects of % Damage to Cluster, Motion Type, random intercepts and TLV to models that additionally included the interaction % Damage to Cluster×Motion Type. The interaction term did not approach significance for any of the clusters (all ps>.375). Thus, for each cluster, an increase in % damage was equivalently associated with deficits in the three prediction conditions.
Analysis of neuroanatomical and behavioral predictors
It is possible that behavioral predictors, rather than having a primary association with d’ scores, are merely subserved by brain regions having neuroanatomic proximity to or overlap with the regions critical for prediction. This hypothesis would predict that when neuroanatomic and behavioral predictors are included in the same model of prediction ability, only the former (or possibly neither) will remain significant. The final models we assessed included both neuroanatomic and behavioral predictors of d’ scores.
Given that we can assume independence of un-correlated predictors, we first tested for a correlation between % damage to each of our three neuroanatomic clusters and the behavioral predictors. Motor Production Ability correlated significantly with all of the neuroanatomical predictors (r ranging from −.54 to −.66; ps<.01). Praxis Ability correlated significantly only with % Damage to IPL Cluster (r=−.41; p=.034; other ps>.183).
Given these correlations, we performed model comparisons to assess changes in model fit when adding % Damage to Cluster to a reduced model that included random intercepts, TLV and Motor Production Ability, separately for each cluster. The addition of % Damage to Cluster resulted in a significant improvement in model fit for all clusters (ps<.05). Conversely, however, adding the predictor Motor Production Ability to a reduced model that included random intercepts, TLV and one of the cluster predictors (separately for each cluster) did not lead to a significant improvement in model fit for any of the clusters (ps>.166).
The same approach was followed for Praxis Ability and % Damage to IPL Cluster. Adding % Damage to Cluster to a reduced model that included random intercepts, TLV and Praxis Ability led to a significant improvement in model fit (p=.012) while adding Praxis Ability to a reduced model that included random intercepts, TLV and % Damage to IPL Cluster led to an improvement in model fit that just missed the significance threshold (p=.05). These analyses suggest that motor production ability is predictive of d’ only when damage to insula, IPL, or dorsal frontal cortex is not considered. Praxis ability, however, remains a significant predictor even when considering IPL damage.
DISCUSSION
This study identified neural substrates and behavioral correlates associated with the prediction of human and non-biological motion trajectories in patients with left-hemisphere stroke. Relative to the prediction performance of controls, patients’ performance was impaired. This impairment was associated with two distinct measures of action ability, limb apraxia and hemiparesis, but not with a measure of action recognition. Moreover, action impairment negatively affected prediction of not only human, but also non-biological motion. Furthermore, impaired prediction of both motion types was equivalently associated with voxel clusters in the left inferior parietal and dorsal frontal cortices as well as the left insula; motor production ability only predicted performance when lesions in these regions were not also taken into account. By contrast, praxis ability remained a significant predictor of performance even when IPL damage was considered. In all cases, these predictors remained significant even when we controlled for total lesion volume, a proxy for stroke severity.
The importance of the integrity of sensorimotor regions in left parietal and dorsal frontal cortex for motion prediction is consistent with previously observed neuroimaging activations in these regions in neurologically intact individuals engaged in prediction tasks (Balser et al., Reference Balser, Lorey, Pilgramm, Stark, Bischoff, Zentgraf and Munzert2014; Cross et al., Reference Cross, Stadler, Parkinson, Schütz-Bosbach and Prinz2013; Schubotz & von Cramon, Reference Schubotz and von Cramon2002; Stadler et al., Reference Stadler, Schubotz, von Cramon, Springer, Graf and Prinz2011; van Elk, Reference van Elk2014; Wiener et al., Reference Wiener, Turkeltaub and Coslett2010; Yang, Reference Yang2014). Although bilateral activations are commonly reported, meta-analyses indicate that left hemisphere activations are most reliable (Wiener et al., Reference Wiener, Turkeltaub and Coslett2010; Yang, Reference Yang2014). Van Elk (2014), for example, found activations in left PM and IPL when participants observed an actor holding a tool (e.g., a wine opener), a target (wine bottle), and a distractor object (soda bottle) and were asked to predict the direction in which the tool would be moved.
Importantly, our lesion data strongly suggest that these regions in the left hemisphere play a critical (and not merely supportive or incidental) role in these tasks (see also Schubotz et al., Reference Schubotz, Sakreida, Tittgemeyer and von Cramon2004; Stadler, Ott, et al., Reference Stadler, Ott, Springer, Schubotz, Schütz-Bosbach and Prinz2012). We, furthermore, observed critical involvement of a cluster located in the insula. Although the left insula has not been implicated in prediction of perceptual stimuli, several recent neuroimaging studies have reported activations in the insula during motor planning (Bernardi et al., Reference Bernardi, Ricciardi, Sani, Gaglianese, Papasogli, Ceccarelli and Pietrini2013; Di Russo et al., Reference Di Russo, Lucci, Sulpizio, Berchicci, Spinelli, Pitzalis and Galati2016) which, we will suggest below, is closely associated with prediction. The critical role of the insula cluster may also be related to the proximity of this cortical region to the superior longitudinal fasciculus, a prominent white matter pathway connecting frontal and parietal cortices (Makris et al., Reference Makris, Kennedy, McInerney, Sorensen, Wang, Caviness and Pandya2005; Schmahmann et al., Reference Schmahmann, Pandya, Wang, Dai, D’Arceuil, De Crespigny and Wedeen2007).
The present finding that motor production ability is associated with prediction task performance only when frontoparietal lesions are not considered is consistent with previous work focusing on the relationship between hemiparesis and the ability to simulate one’s own actions. Johnson (Reference Johnson2000) asked hemiplegic patients to indicate how they would grasp a dowel if they were to reach out and perform the action. Patients were able to perform the simulation task unless their lesions involved frontoparietal cortex (see also Johnson, 2002). Given these reports, the present association between motor production ability and prediction ability may be explained by the co-occurrence of hemiparesis and lesions in the left premotor and parietal cortices in our sample. That is, motor production ability and prediction ability, rather than being directly causally related, may simply share similar or adjacent neuroanatomic substrates.
Unlike hemiparesis, the effect of limb apraxia (as indexed by performance on the pantomime production task) remained significant even when considering left IPL damage, suggesting a robust association between praxis ability and motion prediction ability. Apraxic patients have previously been shown to be impaired at planning and simulating their own actions, whereas motor tasks that can be performed adequately by relying on visual feedback (such as reaching to targets under visual guidance) are relatively intact (e.g., Buxbaum, Johnson-Frey, et al., Reference Buxbaum, Johnson-Frey and Bartlett-Williams2005; Coslett et al., Reference Coslett, Buxbaum and Schwoebel2008; Dawson et al., Reference Dawson, Buxbaum and Duff2010; Eidenmuller et al., Reference Eidenmuller, Randerath, Goldenberg, Li and Hermsdorfer2014; Jax et al., Reference Jax, Buxbaum and Moll2006; Mutha et al., Reference Mutha, Sainburg and Haaland2010; Ochipa et al., Reference Ochipa, Rapcsak, Maher, Rothi, Bowers and Heilman1997; Sirigu et al., Reference Sirigu, Duhamel, Cohen, Pillon, Dubois and Agid1996; Wheaton et al., Reference Wheaton, Fridman, Bohlhalter, Vorbach and Hallett2009).
For example, Buxbaum, Johnson-Frey et al. (Reference Buxbaum, Johnson-Frey and Bartlett-Williams2005) found that apraxics performed abnormally on Johnson’s (Reference Johnson2000) grip simulation task, but executed the same grasps successfully. Similarly, Sirigu et al. (Reference Sirigu, Duhamel, Cohen, Pillon, Dubois and Agid1996; see also Jeannerod, Reference Jeannerod1986) found that apraxics were impaired at simulating finger opposition sequences, with performance on the actual finger opposition task within the control range. Lastly, Dawson et al. (Reference Dawson, Buxbaum and Duff2010) and Eidenmuller et al. (Reference Eidenmuller, Randerath, Goldenberg, Li and Hermsdorfer2014) observed relatively impaired anticipatory grip force scaling in apraxics when asked to lift everyday objects (but see Li, Randerath, Goldenberg, & Hermsdörfer, Reference Li, Randerath, Goldenberg and Hermsdörfer2011). Given that the anticipatory planning component of action production appears to be affected in apraxia, the present results are consistent with the proposal that action prediction relies on mechanisms that are shared with motor planning, which may include use of the same forward models (Friston et al., Reference Friston, Mattout and Kilner2011; Wolpert et al., Reference Wolpert, Doya and Kawato2003).
Importantly, however, praxis ability was associated with deficits in prediction of both human action and non-biological motion, and there was no indication that these relationships were of different strengths. These data are broadly supportive of the prediction account of Schubotz (Reference Schubotz2007) in which forward models may be ideally suited to predict human actions, but also deployed to predict non-biological motion. By contrast, the observed robust association between praxis ability and the ability to predict non-biological motion is not well explained by the perceptual associative account of prediction by Pickering and Garrod (Reference Pickering and Garrod2013) in which forward models are used only to predict actions performed by agents that are “similar enough” to the observer, while trajectories lacking human characteristics are predicted based on past perceptual experience (see also Springer et al., Reference Springer, Brandstädter and Prinz2013; Stadler, Springer, et al., Reference Stadler, Springer, Parkinson and Prinz2012). This account would seem to suggest that non-biological motion should be predicted normally in at least some patients with apraxia.
Notably, we observed that neurologically intact controls were better at predicting pantomimed tool use actions than both actual tool use actions and non-biological motion. At present, we can only speculate about the cause of this pattern. One possibility is that controls paid closer attention to the pantomimed actions than to the other two motion types, perhaps because the former were more novel. However, this account predicts that patients, too, should benefit from the combination of novelty and increased attention, unless one posits an additional deficit in patients’ ability to allocate additional attention to the task. Further research is required to explain this aspect of the findings.
Finally, the absence of a significant association between action recognition and prediction enables us to hypothesize that prediction may rely on processes further “downstream” in the motor planning process than those tapped in action recognition tasks. Additionally, the weak association between action recognition and prediction is inconsistent with accounts suggesting that action recognition is critically dependent upon action simulation (Rizzolatti et al., Reference Rizzolatti, Fogassi and Gallese2001; Urgesi et al., Reference Urgesi, Candidi and Avenanti2014). Consistent with recent prior findings from our laboratory (Kalénine et al., Reference Kalénine, Buxbaum and Coslett2010; Tarhan et al., Reference Tarhan, Watson and Buxbaum2015) and others (e.g., Negri et al., Reference Negri, Rumiati, Zadini, Ukmar, Mahon and Caramazza2007) showing that brain regions beyond the human “mirror system” are critical for action recognition, the present data suggest that action recognition is not reducible to action simulation.
CONCLUSION
It has been argued that action production and prediction rely upon shared underlying mechanisms (e.g., Aglioti et al., Reference Aglioti, Cesari, Romani and Urgesi2008; Colling et al., Reference Colling, Thompson and Sutton2014; Stapel et al., Reference Stapel, Hunnius, Meyer and Bekkering2016). We report data from left-hemisphere stroke patients that enables us to refine this account. Consistent with the hypothesis that forward models are used both for action planning and prediction (Friston et al., Reference Friston, Mattout and Kilner2011; Wolpert et al., Reference Wolpert, Doya and Kawato2003), our data suggest that the feedforward components of action trajectory planning are critical for prediction of human as well as non-biological motion trajectories. Moreover, our results suggest that the left insula, inferior parietal, and dorsal frontal cortices are critical to the ability to predict both motion types. On first inspection, these data may appear to be inconsistent with widely cited prior findings that action expertise has a specific effect on the ability to predict the relevant action, but not other related skills (Aglioti et al., Reference Aglioti, Cesari, Romani and Urgesi2008). However, action planning capacities are presumably not all-or-none. Motor planning mechanisms may be used for prediction of all types of events that contain spatiotemporal trajectory information, with high levels of motor expertise serving to “tune” the precision with which exquisitely timed events can be predicted.
Acknowledgments
The authors declare no conflict of interest. This research was supported by National Institutes of Health grants R01NS099061 (L.J.B.) and T32HD071844 (M.M.D.W., trainee), and the Moss Rehabilitation Research Institute Peer Review Committee. We thank Branch Coslett for help with lesion segmentation, Christine Watson for script writing and suggestions regarding neuroanatomical analyses, and several research assistants, in particular Leyla Tarhan, for pantomime coding and help with data collection.
