In their target article, Goldin-Meadow & Brentari (G-M&B) ask in section 7.3 what technology is needed to further study sign language and gesture. There are several novel methods recently developed or under development that focus on investigating the biological and information-transfer properties of dynamic visual data, such as that produced in sign language and gesture.

1. 1. Motion capture methods have evolved from trajectory-tracing to analysis of multiple dynamic features, such as velocity, acceleration, or within- and between-participant indices of movement stability; to date, these approaches have been applied to American Sign Language, Croatian Sign Language, and Finnish Sign Language (Malaia & Wilbur 2010; Malaia et al. Reference Malaia, Wilbur and Milković2013; Puupponen et al. Reference Puupponen, Wainio, Burger and Jantunen2015). Recent analyses indicate that kinematics of natural narrative production (from a single signer) produces results equivalent to analyses of multiple signers in controlled elicitation phrases (Wilbur & Malaia, Reference Wilbur, Malaia, Hübl and Steinbachin press), which makes the technique generalizable to single-subject studies of lesser-known sign languages, as well as clinical cases. Another method that has been proven useful is analysis of eye-tracking in combination with stimulus data or video recording of dyadic interactions (Oben & Brône Reference Oben and Brône2015; Romero-Fresco Reference Romero-Fresco2015).

2. 2. Current technology and software make possible use of natural stimuli (e.g., gestures with movement, or signed-sentence videos), rather than degraded (static) versions, for neuroimaging and neurophysiological experiments. This is important for identifying the roles of language and sensory perception in the processing of complex input (Malaia et al. Reference Malaia, Ranaweera, Wilbur and Talavage2012; Reference Malaia, Borneman and Wilbur2016); use of degraded stimuli, on the other hand, does not yield sufficient information about natural processing of sign language or gesture (Malaia Reference Malaia, Gonzalez-Castillo, Weber-Fox, Talavage, Wilbur, Mandouilidou and de Almeida2014a; Reference Malaia, Talavage and Wilbur2014b).

3. 3. Analysis of optical flow in video data allows quantitative evaluation of content, both at specific temporal or spatial frequencies (speeds of motion, size of articulator) and with regard to mathematical information content (Shannon entropy in visual signal) (cf. Malaia et al. Reference Malaia, Borneman and Wilbur2016). This method does not require intrusive wearable markers, can be used on natural video data, and allows separation of communicative signal (e.g., sign language) from other biological motion in the data.

The need for additional technology development is driven further by unanswered research questions on sign language and gesture. It is still unclear as to how information is encoded in the communicative visual signal. What parameters of the visual stimuli (e.g., frequencies of visual spectrum, or timing of motion in sign language syllabification and phrasing) are used by the brain during perception to extract linguistic information from the signed visual input, and extralinguistic information from gesture? Further, how is that information integrated with other (auditory or haptic) input? To answer the larger questions in the field, improved technologies for data collection and analysis are needed; in comparison with spoken language research, the capabilities of the visual communication field still lag behind.

The data collection stage of visual communication research would be significantly enhanced by a capability of automatic capture for motion of hands, head, body, and face (separately and coordinated) without the need of sensors, gloves, or markers. Hypothesis testing requires software equivalent to Praat (http://praat.org) for spoken language, aimed at analyzing spectral components of the visual signal across spatial and temporal frequencies, including motion, recognition of meaningful handshapes and nonmanuals, and data annotation (see McDonald et al. Reference McDonald, Wolfe, Wilbur, Moncrief, Malaia, Fujimoto, Baowidan and Stec2016 for a prototype). Such techniques should be applicable to noisy real-world data (the “cocktail party” problem is as real in visual communication as it is in the auditory domain). One of the directions to solve this problem is the use of biologically inspired cognitive architectures. For example, quantitative analysis of visual properties of the signal (cf. Bosworth et al. Reference Bosworth, Bartlett and Dobkins2006) can be combined with neural data during perception (electroencephalography [EEG], functional magnetic resonance imaging [fMRI], etc.) to investigate the models of information transfer between complex systems using, for example, complex systems analysis and machine learning techniques (cf. Malaia et al., Reference Malaia, Borneman and Wilburin press; Barbu et al. Reference Barbu, Barrett, Chen, Siddarth, Xiong, Corso, Fellbaum, Hanson, Hanson, Helie, Malaia, Pearlmutter, Siskind, Talavage, Wilbur and Fleet2014).

Development of technologies (quantitative analysis methods) and their acceptance in research is based on a feedback loop with testable, predictive hypotheses, which can ground (sign) language and communication in neural processing and biological development. Parallel studies of natural perception and production in communication, such as correlational measurements of Shannon entropy in neural (EEG, fMRI) and linguistic (speech and signing) data to assess information transfer are likely to yield the most insight into the unique phenomena of human communication, such as sign language and gesture.