Tracking the Continuity of Language Comprehension Computer Mouse Trajectories Suggest Parallel Syntactic Processing
Thomas A. Farmera, Sarah A. Cargilla, Nicholas C. Hindya, Rick Daleb, Michael J. Spiveya
Abstract
Although several theories of online syntactic processing assume the parallel activation of multiple syntactic representations, evidence supporting simultaneous activation has been inconclusive
continuous and non-ballistic properties of computer mouse movements are exploited
procure evidence regarding parallel versus serial processing
Participants heard structurally ambiguous sentences while viewing scenes with properties either supporting or not supporting the difficult modifier interpretation
The curvatures of the elicited trajectories revealed both an effect of visual context and graded competition between simultaneously active syntactic representations
Introduction
Sentences such as, “The adolescent hurried through the door tripped,” are difficult to process because, at least temporarily, multiple possible structural representations exist
what feel like garden-path effects are due to the incorrect syntactic alternative winning much of the competition during the early portion of the sentence, and then nonconforming information from the latter portion of the sentence inducing a laborious reversal of that activation pattern
As a result, one can expect that some garden-path events may be very mild, some moderate, and some extreme such that a wide variety of sentence readings should all belong to one population of events with a relatively continuous distribution.
When ambiguous sentences like 1a are heard in the presence of visual scenes where only one possible referent is present (an apple already on a towel), along with an incorrect destination (an empty towel), and a correct destination (a box), as in the top portion of Fig. 1, about 50% of the time participants fixate the incorrect destination after hearing the first PP.
After the second disambiguating PP is heard, eye movements tend to be redirected to the correct referent and then to the correct destination
This garden-path effect can, however, be modulated by contextual information contained within the visual scene
it seems that when two possible referents are present, an expectation is created that they will be discriminated amongst, thus forcing a modifier interpretation of the
ambiguous PP
The attenuation of looks to the incorrect destination by the presence of two possible referents, then, is evidence for an early influence of non-syntactic (even nonlinguistic) information on the parsing process and is problematic for traditional syntax-first accounts discussed earlier.
However, because saccadic eye movements are generally ballistic, they either send the eyes to fixate an object associated with a garden-path interpretation or they do not.
The evidence from this paradigm, therefore, is also consistent with the Unrestricted Race model, where the various constraints are combined immediately, but on any given trial only one syntactic representation is initially pursued
across experimental trials, distributions of eye-movement patterns are almost always bimodal because the fixations are coded as binomial
There are saccades to locations on the display corresponding to either one of the possible representations, but almost never to a blank region in between those two potential targets
In the following experiment, we examined the dynamics of hand movement in the same sentence comprehension scenario with the goal of determining whether the non-ballistic, continuous nature of computer mouse trajectories can serve to tease apart these two remaining theoretical accounts.
Experiment
computer mouse movements can serve as an informative indicator of the cognitive processes underlying spoken-word recognition (Spivey, Grosjean, & Knoblich, 2005)
In addition, whereas self-paced reading affords 2 to 3 data points (button presses) per second, and eye-movement data allow for approximately 3 to 4 data points (saccades) per second, “mouse tracking” yields somewhere between 30 and 60 data points per second, depending on the sampling rate of the software used.
The context and garden-path effects reported in the visual world paradigm are highly replicable when tracking eye movements
As such, recording mouse movements in the visual world paradigm can serve as a strong test case by which to evaluate the efficacy of the mouse-tracking procedure for the study of language processing in real time.
Expected Prediction
Averaged trajectories recorded in response to ambiguous sentences in the onereferent context should show significantly more curvature toward the incorrect destination than the averaged trajectories elicited by unambiguous sentences—a pattern corresponding to the garden-path effect. 2)
The curvature of averaged trajectories in the two referent condition should not differ statistically between ambiguous and unambiguous sentences, thus demonstrating an influence of referential context on the garden-path effect.
The second purpose of this study, then, was to exploit the continuity of the mousemovement trajectories to discriminate between these two remaining theoretical accounts
a measure of curvature magnitude was used to determine the amount of spatial attraction toward the incorrect destination that was exhibited by the ambiguousand unambiguous-sentence trajectories in the one-referent context.
If only one representation were active at any one time, as the unrestricted race account predicts, then the trial-by-trial distribution of trajectory curvatures in the ambiguous-sentence condition should be either (a) bimodal—comprised of highly curved garden-path movements and noncurved, correct-interpretation movements, or (b) uniformly in the more extreme curved range, indicating that almost every trial exhibited a garden-path effect
Method
Participants
Forty right-handed, native English-speaking undergraduates from Cornell University participated in the study for extra credit in psychology courses
only right-handed individuals to avoid variability associated with subtle kinematic differences in leftward and rightward movement of the left versus the right arms.
Materials and Procedures
Sixteen experimental items 102 filler sentences
The unambiguous version (1b) of each of the 16 experimental items was recorded first, and then the “that” was removed to produce the ambiguous (1a) sentence condition
Each visual context corresponding to the 16 experimental items was varied to produce a oneand two-referent condition
The one-referent visual context (illustrated in Fig. 1, top) contained the target referent (an apple on a towel), an incorrect destination (a second towel), the correct destination (a box), and a distracter object (a flower). In the two-referent context, all items were the same except that the distracter object was replaced with a second possible referent (such as an apple on a napkin). Twenty-four filler scenes, designed to accompany filler sentences, were also constructed.
In critical trials for both the oneand two-referent conditions, the target referent always appeared in the top left corner of the screen, the incorrect destination always appeared in the top right corner of the screen, and the correct destination
was always located at the bottom right portion of the screen.
Given that the scene layout was held constant across all items in each experimental condition, a left-to-right movement was always necessary
Although there could exist a systematic bias toward specific locations in the display when moving rightward, this was viewed as unproblematic given that the bias would be held constant across both the ambiguous and unambiguous sentences, which were directly compared in all statistical analyses, for each context.
In each scene, participants saw four to six color images, depending on how many objects were needed for the scene
Results
Data Screening and Coding
A trajectory was considered valid and submitted to further analysis if it was initiated at the top left quadrant of the display and terminated in the bottom right quadrant, indicating that the correct referent had been picked up and then placed at the correct destination. This screening procedure resulted in 27 deleted trials, accounting for less than 5% of all experimental trials.
To make sure that trajectories in one condition were not initiated (or that objects were not grabbed) at a systematically different region of the display than in the other conditions, we conducted two 2 (Context) × 2 (Ambiguity) ANOVAs on the x and y coordinates, separately.
There was no significant main effect or interaction for either the x or the y coordinates (all ps were nonsignificant) indicating that, across conditions, the trajectories were initiated at approximately the same location of the display
Subsequently, all analyzable trajectories were “time normalized” to 101 timesteps by a procedure described in Spivey et al. (2005) and Dale et al. (2007).
Context and Garden-path Effects
The mean trajectories from ambiguous and unambiguous sentences in the onereferent context, illustrated in Fig. 1 (top), demonstrate that the average ambiguoussentence trajectory was more curved toward the incorrect destination than the average trajectory elicited by the unambiguous sentences
Thus, in the presence of the garden-path effect, it seems clear that there exists more spatial attraction toward the incorrect destination for the ambiguous sentences.
In addition, in line with the time-normalized analyses presented above, none of the
eight time bins in the two-referent context showed the ambiguousand unambiguous-sentence trajectories significantly diverging for either the x or the y coordinates.
Serial Versus Parallel Activation
garden-path trials and some non-garden-path trials, the majority of the trajectories elicited in this condition fell somewhere in between those two extremes, forming a single population of non-, somewhat-, and highly curved responses.
To determine whether any bimodality is present in the distribution of responses, we computed the area under the curve on a trial-by-trial basis
The b value for each distribution is less than .555, indicating no presence of bimodality within the distributions.
Notably, with regard to the distribution of responses in the one-referent, ambiguous-sentence condition, b < .555 indicates that the graded spatial attraction effects elicited in this condition came not from two different types of trials but from a single population of trials.
Finally, one might argue that bimodality was not detected (thus, b < .555) in the crucial one-referent, ambiguous-sentence condition due to a lack of statistical power resulting from the relatively small number of trials in the garden-path distribution.
To address this concern, we created an artificial distribution with a sample size almost identical to our crucial gardenpath distribution by randomly sampling 50% of the trials from the one-referent, ambiguoussentence condition (where garden pathing was observed) and 50% of the trials from the onereferent, unambiguoussentence condition.
By examining the distributional properties of the area-under-the-curve values produced by the garden-path and non-garden-path trials together, we can thus determine whether the bimodality statistic (b) we used to assess the bimodality of the garden-path distribution (above) is capable of detecting bimodality in a case where the response distribution should clearly be bimodal
General Discussion
In the one-referent context, participants’ mouse movements in response to the ambiguous sentences curved significantly closer to the top right of the screen (toward the incorrect destination) than in response to unambiguous sentences.
Thus, it would seem that when only one referent was present, the incorrect destination (e.g., the towel) was partially considered relevant, until disambiguating information was processed—a trend corresponding to the garden-path effect associated with this condition.
The fact that most mouse trajectories began while the speech file was still being heard suggests that the effect of visual context modulating the garden path took place during early moments of processing the linguistic input, not during a second stage of syntactic reanalysis.
In addition, by capitalizing on the continuous, non-linear, and non-ballistic properties of trajectories produced by computer mouse movements, mouse tracking has the potential to answer questions that have been difficult to answer with more traditional methodologies.
What does distinguish between these two accounts is the gradiency observed in the curvature of the trajectories in the garden-path condition
If the Unrestricted Race model posits that only one syntactic representation is pursued at any one time, then it must predict that mouse movements in a gardenpath condition should initially move either in the direction of the correct destination or in the direction of the incorrect destination (producing either a bimodal distribution or an all-curved distribution)
In contrast, because the constraintbased account posits simultaneous graded activation of multiple syntactic alternatives, it predicts that mouse movements can move in directions that are dynamically weighted combinations of the two competing destinations (producing a unimodal distribution of moderate curvatures).
Fig. 4 shows that although approximately 5% of the trajectories moved all the way to the incorrect destination before changing direction, the vast majority of the trajectories responsible for the mean curvature were unmistakably graded in their degree of spatial attraction toward the incorrect destination.
The lack of bimodality in the distribution of trial-by-trial trajectory curvatures suggests that the garden-path effect so frequently associated with this manipulation is not an all-or-none phenomenon—that is, the activation of one structural representation does not forbid simultaneous activation of other possible representations
Through a large-scale survey of children’s computer use, for example, Calvert, Rideout, Woolard, Barr, and Strouse (2005) found that the mean age at which a child was able to point and click a computer mouse was 3.5 years, and that the mean age of the onset of autonomous computer use was 3.7 years
we believe mouse tracking can serve as “the poor man’s eye tracker,” providing detailed indices of cognitive processing to laboratories that cannot afford expensive eye-tracking equipment.
