1. Introduction
Perception is a complex and multifaceted process and involves various sensory inputs that determine how we experience reality. In case of multistable visual stimulation, sensory activity continually triggers reorganisation of neuronal activity, leading to attentional modulation (Leopold and Logothetis Reference Leopold and Logothetis1999). The effects of multistable stimulation can also be observed in the auditory domain, manifesting themselves, for example, as auditory grouping (Bregman Reference Bregman1990). Here, an effect is a consequence or outcome of some event or condition (physical domain), or refers to a specific perception when presented an external stimulus (perceptual domain) (cf. Féron Reference Féron2015).
However, the information obtained through our sensory inputs may not suffice for an unambiguous representation of reality, and consequently lead to misinterpretations by our brain, that is, illusory perceptions (Matthen Reference Matthen2015). In that sense, an illusion can be generally defined from a perceptual perspective as a ‘misinterpretation of an objective reality’ (Féron Reference Féron2015). Note that in this article, we consider perceptual illusions only, that is, we exclude semantic misidentifications of sounds like, for example, perceiving distant traffic like surf. We further distinguish between illusions and perceptual effects: while illusions involve a perception of physically non-existing stimulus components, perceptual effects are caused by physically existing stimulus components.
Perceptual illusions can occur in any of our senses, including vision, hearing and a combination of both (Bregman Reference Bregman1990; Matthen Reference Matthen2015).
A visual illusion may result from a misinterpretation of reality through our eyes, triggering a discrepancy between what we expect to see and what we actually see. Visual illusions often rely on the interactions between light, colour, shape, perspective and depth (Escher Reference Escher2000; Weststeijn Reference Weststeijn2013; Todorović Reference Todorović2020). Studies on their perception provide valuable insights into the underlying physiological, psychological and neurophysiological aspects (Bruce et al. Reference Bruce, Green and Georgeson2003). Seen in this light, an alternative way of defining visual illusions is to regard them as the result of brain mechanisms aiming to ‘meaningfully’ interpret visual sensory inputs.
Analogously, auditory illusions are misinterpreted perceptions of an external sound stimulus: listeners may hear sounds which are not present or sounds that should not be heard given the circumstances under which they were produced. Auditory illusions can be influenced by various psychological and physiological aspects such as expectation, attention, memory, sensory sensitivity and perceptual grouping (Bregman Reference Bregman1990). Also, neural mechanisms related to central-auditory and hierarchical processing play an essential role (Husain et al. Reference Husain, Lozito, Ulloa and Horwitz2005; Kuriki et al. Reference Kuriki, Numao and Nemoto2016; Mehta et al. Reference Mehta, Jacoby, Yasin, Oxenham and Shamma2017). In that respect, auditory illusions can alternatively be viewed as products of inherent aspects in peripheral and central hearing processes, rather than as misinterpretations or distortions of reality.
Auditory-visual illusions occur as a biased misinterpretation of the auditory and visual stimuli (for a review, see O’Callaghan Reference O’Callaghan2017). Our brain selectively weights and combines the streams of information from both perceptual domains (Opoku-Baah et al. Reference Opoku-Baah, Schoenhaut, Vassall, Tovar, Ramachandran and Wallace2021; Shams et al. Reference Shams, Kamitani and Shimojo2000), leading to a biased interpretation of reality. Consequently, auditory-visual illusions can be viewed as simply revealing characteristics of perceptual processes or modal dominance (e.g., McGurk and MacDonald Reference McGurk and MacDonald1976).
Note that in our article, we use the term ‘auditory-based’ to stress that illusions in both perceptual domains require an auditory stimulus to work.
Illusions and effects represent an intriguing compositional tool for sound art (Knight-Hill Reference Knight-Hill1996; Gibbs Reference Gibbs2007; Wishart Reference Wishart and Emmerson2016; Maes and Leman Reference Maes, Leman, Cobussen, Meelberg and Truax2017). Their use as a compositional tool aligns more closely with the original concept embodied by the Latin term ‘illusio’, meaning to deceive or mislead – which can be pushed to the extreme in auditory magic tricks (Kuhn et al. Reference Kuhn, Gibgot, Thomas and Ekroll2024). By creating an experience where listeners perceive something that is physically not present, auditory-based illusions effectively capture the essence of this idea.
The transdisciplinary creative practice of sound art (Gál Reference Gál2022) has become increasingly important for contemporary music. It involves genres such as experimental and electroacoustic music, installation art and media art, as well as scientific disciplines such as musicology, art history and open-research domains. Sound art connects various disciplines including aesthetics, acoustics, architecture, music psychology, neuroscience, philosophy of art and sociology. Related cross-media works are often reproduced or ‘exhibited’ over extended periods of time (Gibbs Reference Gibbs2007; Knight-Hill Reference Knight-Hill2020).
Sound installations are central manifestations of sound art and combine physical or virtual spaces with sound objects to create an immersive sonic experience (Bandt Reference Bandt2005). They can be characterised in terms of various aspects (Fraisse et al. Reference Fraisse, Giannini, Guastavino and Boutard2022; Gal Reference Gál2022). Unlike traditional music performances that usually have specific durations and listening positions, sound installations leave many choices to the listeners including, for example, the duration of visit, the preferred listening spot and perspective, or whether to explore the presented scene statically, or dynamically (Fraisse et al. Reference Fraisse, Wanderley and Guastavino2021). In this sense, visitors are turned into ‘co-authors’ who complement the ‘proposed’ art work through individual interaction.
Understanding the underlying fundamentals of auditory-based illusions and effects (Blauert Reference Blauert1996; Fastl and Zwicker Reference Fastl and Zwicker2007; Bruce et al. Reference Bruce, Green and Georgeson2003) and their unimodal or multimodal perception (Spence and Driver Reference Spence and Driver2004; O’Callaghan Reference O’Callaghan2017) is beneficial for their application as an experimental tool in sound installations. In this sense, this article focuses on the connection of objective sound-installation aspects and auditory-based illusions and effects. Hence, it not only provides an overview of sound-installation aspects (cf. Section 2), and auditory-based illusions and effects supplemented with application examples (cf. Section 3), but also aims at promoting an effective implementation of auditory-based illusions and effects in sound installations, as well as inspiring new ideas on how to combine sound installations with scientific fundamentals.
2. Sound-installation aspects
In this section, we review formal taxonomies enabling analysis and comparison of sound installations. Section 2.1 describes selected relevant objective sound-installation aspects (Fraisse et al. Reference Fraisse, Giannini, Guastavino and Boutard2022). Section 2.2 provides perceptual and philosophical aspects selected from Gál (Reference Gál2022), which help to complement the description and analysis of sound installations. Section 2.3 extends the scope to selected social and cultural aspects.
2.1. Objective aspects
Inspired by Fraisse et al. (Reference Fraisse, Giannini, Guastavino and Boutard2022), Table 1 presents a categorisation of selected objective sound-installation aspects, and establishes a connection to the auditory-based illusions and effects presented in Section 3.
Auditory-based illusions and effects, and their connection to selected objective sound-installation aspects
Note: ▴ = required; ●● = used in the provided reference(s); (●●) = assumed due to unclear methodology in the provided reference(s); ● = considered experimental for sound installations; ○ = potentially influential; empty = not relevant or not applicable; LS = loudspeaker. This table makes no claim to be complete.
The focus is on indoor sound installations with room acoustics that may have perceptual effects on the presented auditory-based illusions and effects (Franssen Reference Franssen1960; Wallach et al. Reference Wallach, Newman and Rosenzweig1949; Haas Reference Haas1972). Although not considered by Fraisse et al. (Reference Fraisse, Giannini, Guastavino and Boutard2022), indoor sound installations can also be presented in an integrated way, see application examples below, or oppositionally, depending on the artist’s compositional intention and the level of distinction required from room-specific background sounds or noise levels. Both interactive or non-interactive installation settings are conceivable, opening up interesting interaction possibilities; for example, to change time sequences in stimulus presentations and to control the positioning and reproduction level of sound sources. Consequently, such interactions may have reactive or generative effects on the presented composition in real-time or deferred time (Gau et al. Reference Gau, Redi, Redi and Kampel2012).
Integrated indoor sound installation: application examples
The sound installation ‘Roth’ (2024/25) by Bernhard Gál (b. 1971) focuses on existing sound elements in SP•CE (Gál Reference Gál2025), a sound laboratory for electroacoustic and experimental compositions. Faint and often ignored sounds unconsciously define the perceptual experience, blending actual sounds with artificially added recordings of the same acoustic environment.
The sound work ‘REACH: New York’ (Janney Reference Janney1995) is an interactive public indoor art installation designed to engage the city’s residents in a sensory audio experience, blending sound art and technology. Implemented at the 34th Street/Herald Square station in New York City, it uses light sensors to invite interaction with participants who trigger sounds through visual movement, fostering a sense of connection in an otherwise impersonal urban environment.
Various devices can be used for the reproduction of auditory-based illusions and effects, ranging from electroacoustic devices such as headphones, a single or multiple loudspeakers, to customised appliances or musical instruments, and visual displays. The required reproduction levels are specified by the auditory-based illusion (e.g., Schouten et al. Reference Schouten, Ritsma and Cardozo1962), or left open to the composer. Some of the presented illusions originally investigated with real sound sources were recreated with virtual sound sources already in early primary studies (e.g., Wallach et al. Reference Wallach, Newman and Rosenzweig1949), or in later secondary studies (e.g., Stawicki et al. Reference Stawicki, Majdak and Başkent2019). State-of-art auralisation methods (see, e.g., Vorländer Reference Vorländer2020; Schröder Reference Schröder2011; Zotter and Frank Reference Zotter and Frank2019) are likely to facilitate a flexible and effective presentation of auditory-based illusions and effects in sound installations.
Regarding visiting modalities, we encourage the listener, whenever feasible, to dynamically explore the presented illusions as part of a sound installation, entailing a dynamic listening spot. Note that for plausible reproduction of a virtual scene in which auditory-based illusions and effects are to be binaurally reproduced via headphones (McAnally and Martin Reference McAnally and Martin2014) or loudspeakers (Masiero and Vorländer Reference Masiero and Vorländer2014), a dynamic listener requires to be motion-tracked for real-time scene updates.
Visual aspects such as the visibility of sonic or non-sonic elements can play a crucial role in sound installations involving auditory-based illusions and effects. While auditory-visual illusions absolutely need non-sonic, specifically visual, elements in order to work (McGurk and MacDonald Reference McGurk and MacDonald1976; Bermant and Welch Reference Bermant and Welch1976), for the auditory illusions, the visibility of each type of element is optional (Wallach et al. Reference Wallach, Newman and Rosenzweig1949) or even unnecessary (Schouten et al. Reference Schouten, Ritsma and Cardozo1962).
2.2. Perceptual and philosophical aspects
Activation is an important aspect of sound installations. It refers to the process of a listener engaging with the sound installation and consequently triggering a change in perception. Activation leads to anti-hierarchical individual ‘realisations’ of open works of art and thus can be understood as a democratisation of the perceptual process.
Activation may also result in an increased perceptual awareness of the own environment. For example, the presence of a temple may ‘elevate’ its surroundings (Rebentisch Reference Rebentisch2003). In this way, the process of aestheticisation leads to the experience of atmosphere and to the ‘self-observation’ of one’s own modes of perception. Experiencing sound installations may similarly result in an aestheticisation of everyday sounds or acoustic environments.
Owing to their spatial and site-specific nature, sound installations typically demand an increased level of attention and listening effort. Listeners are challenged to decode complex sound stimuli, recognise rhythmic or melodic patterns, integrate multiple sound sources into a coherent auditory image, and develop an understanding for the composition. Conversely, increased attention, as the result of the art experience, may lead to an increased awareness and finally trigger the aestheticisation processes.
The term Eigenzeit (i.e., own time) encompasses the temporal and spatial structuring when experiencing sound installations. Eigenzeit is largely specific to the acting listener (Heiniger Reference Heiniger and Tadday2008). Instead of experiencing a pre-specified art work, the eventfulness of one’s own experience determines the perceptual result, encouraging an individual ‘journey of discovery in one’s own time’ (Gál Reference Gál2022).
While the visual sense facilitates a separation of subject and object, our auditory sense does not enable such decoupling, automatically resulting in some level of immersion. Auditory immersion can therefore be considered a ‘somatic’ immersion (Böhme Reference Böhme2000).
Sound installations can be presented in an open form, that is, continuously, without having a dedicated beginning or end. This can be achieved by looping pre-recorded sound sequences or generating loops in real-time. Similar to exhibitions of visual arts, the resulting perceptual experience depends on individual decisions, involving factors such as the time spent, the focus of attention, the listening positions and physical motion within the exhibition space.
The process of self-reflection describes the experience of one’s own physical presence. It may go along with a transformational process leading to the ‘unveiling of the intrinsic relevance of people and things’ (Fischer-Lichte Reference Fischer-Lichte2004). Thus, in sound installations, the self-referential observation of one’s own observation (Luhmann Reference Luhmann2020) can itself become the subject and starting point of artistic creation.
2.3. Social and cultural aspects
On a social and cultural level, sound installations provide the possibility of exploring individual listening habits and serve as a bridge from conscious awareness experienced in an artistic context to listening experiences in daily routines and environments. As most sound installations allow the audience to choose their position in relation to sounding elements − in a literal as well as a metaphorical sense − they continuously challenge the participants about what and how they perceive and act, raising questions about their attitude and position towards themselves and the world. Auditory-based illusions and effects may overturn what was perceptually taken for granted so far. This aspect can be exploited to meet the social and cultural challenges of our time on an abstract, yet comprehensible level. This is in line with DeNora (Reference DeNora2003: 71) who stated that ‘conventional cognitive forms can be defied or undermined through unconventional aesthetic forms, namely, through forms that deliberately introduce ambiguity, that confuse, and thus, that require active interpretation’.
When presented in publicly accessible indoor spaces, sound installations are turned into a public and shared event engaging on an artistic as well as social levels, facilitating participation, interaction and exchange (Gau et al. Reference Gau, Redi, Redi and Kampel2012). For example, while the integrated indoor sound installation ‘Roth’ (cf. Section 2.1) potentially stimulates such social aspects and creates an intensified perceptual interaction with the physical environment, ‘REACH: New York’ reflects a deep cultural perspective and encourages a moment of shared communal experience in an otherwise uninviting place.
Generally, the consideration of aesthetic and philosophical concepts of sound such as the meaning of sound as a sign, object or situation (Demers Reference Demers2010), and possible relationships between humans, technology and nature, including aspects of new materialism, can open up novel possibilities of exploring auditory-based illusions and effects as a methodical and compositional tool. Such a broad-based approach can shape contemporary communication models in interactive sound installations, leading to innovative forms of artistic expression that blur traditional boundaries (Chibalashvili et al. Reference Chibalashvili, Kharchenko, Bezuhla, Savchuk and Sydorenko2022).
3. Auditory-based illusions and effects: fundamentals, experiments, and application examples
This section provides an overview and a classification of various auditory-based illusions and effects, links them to selected aspects of sound installations, and presents application examples. Table 1 separates the auditory-based illusions and effects into perceptual domains. It further structures the auditory illusions in categories from a psychoacoustic perspective, and thus, is different from other classification schemes (Wessel and Risset Reference Wessel and Risset1979; Féron Reference Féron2015). Additional primary and secondary literature are provided per illusion. A categorical dot matrix connects each illusion to the sound-installation aspects introduced in Section 2.1.
3.1. Auditory illusions and effects
3.1.1. Spatial illusions
Spatial illusions rely on manipulations of the sound-source characteristics with the goal to influence the perceived distance or direction of a sound event.
A distance illusion can occur when listening to a stimulus simultaneously reproduced via headphones and a loudspeaker. The combination of the two reproduction paths creates a single auditory image whose perceived egocentric distance can be controlled by the relative loudness contributions. Listeners are able to align the perceived egocentric distance of a stimulus with a visual reference (Butler and Naunton Reference Butler and Naunton1964). The perception of such and other distance cues have been extensively investigated in various related studies (Kolarik et al. Reference Kolarik, Moore, Zahorik, Cirstea and Pardhan2016).
An elevation illusion is observed when listening to a notch-filtered stimulus via a single loudspeaker at a fixed elevation. The perceived elevation can be lowered by decreasing the centre frequency of the notch filter (Bloom Reference Bloom1976). The perceptually determined centre frequency for a perceived elevation agrees well with the naturally occurring notch frequency of the head-related transfer functions (HRTF; Blauert Reference Blauert1996) measured for the corresponding elevation. Binaural synthesis can facilitate the generation of virtual sound sources from arbitrary target elevations by filtering a sound with the corresponding pair of HRTFs (Blauert Reference Blauert1996; Vorländer Reference Vorländer2020; Masiero and Vorländer Reference Masiero and Vorländer2014).
Elevation illusion: application example
The composition ‘head transfer #1’ (2024) by Veronika Mayer (b. 1977) used HRTFs to create illusory directional perceptions similarly as done by Bloom (Reference Bloom1976). The stimuli were pre-generated by filtering continuous streams of pink noise or clicks featuring various inter-stimulus intervals with a left or right HRTF that was randomly selected from a set of directions, and played back via one of three loudspeakers. Depending on the stimulus type, the selected HRTF, the listener’s head position and viewing direction, as well as room reflections, the stimuli were either perceived statically from discrete directions or as vertically moving, often not matching the physical loudspeaker directions. The piece was premiered as part of the opening event of SP•CE (Gál Reference Gál2025) within the project ‘AUDIO GHOSTS’ (Gál et al. Reference Gál, Mayer, Majdak and Pausch2025), in Vienna.
A front-or-back illusion can be created by virtually rotating a sound source twice as fast as the listener’s head rotation. It was investigated in experiments on sound localisation in the horizontal plane (Wallach Reference Wallach1940) where participants sat on a chair which, when rotated by the experimenter, triggered playback via one of the circularly arranged loudspeakers, creating a moving sound source. Doubling the rotation speed of the sound source leads to a confused sound-source perception in the opposite half space. The illusion has been reproduced and confirmed in modern experiments using virtual sound sources (Brimijoin and Akeroyd Reference Brimijoin2012).
Binaural beats describe the perception created by two pure tones with a small frequency difference presented to each ear via headphones (Oster Reference Oster1973; Licklider et al. Reference Licklider, Webster and Hedlun1950). The position of binaural beats is often perceived within the head and varies depending the frequency difference (Oster Reference Oster1973).
Binaural beats: application example
Binaural beats were proposed as a non-invasive clinical-intervention tool against anxiety. Their embedding in sound compositions was investigated in a collaborative interdisciplinary study (Macarow et al. Reference Macarow, Weiland, Brown, Jelinek, Samartzis, Grierson and Winter2012). Although, in patient groups exposed to music and sound interventions, the anxiety decreased, no significant effect of binaural beats has been observed. This finding is further supported by Chaieb et al. (Reference Chaieb, Wilpert, Reber and Fell2015), who concluded that studies claiming potential effects of binaural beats on cognitive processes and mood states, as well as their applicability as a therapeutic tool, are of limited validity.
In the context of artistic applications, binaural beats have been used in electroacoustic compositions. Examples are ‘Operation:Kracpot’ (2002) by First (Reference First2003), and ‘Parisflâneur’ (2017), a series of compositions by Rumori and Marentakis (Reference Rumori and Marentakis2017).
3.1.2. Spatio-temporal illusions and effects
Spatio-temporal illusions manipulate the perceived sound-source direction but additionally require the stimuli to be presented in a specific chronological order.
The precedence effect (Wallach et al. Reference Wallach, Newman and Rosenzweig1949) entails that the direction of a leading stimulus determines the perceived direction when succeeded by the same but time-delayed (i.e., lagging) stimulus from another direction. In primary experiments, clicks were sequentially reproduced via two spatially separated loudspeakers at various delays (ibid.). This facilitated the determination of temporal delay thresholds below, around and above which humans will likely perceive a single fused auditory event, an auditory event coming from the direction of the leading click, and two distinct sequential auditory events from the individual click directions, respectively (for reviews, see Litovsky et al. Reference Litovsky, Colburn, Yost and Guzman1999; Brown et al. Reference Brown, Stecker and Tollin2015).
Precedence effect: application example
In the sound installation ‘ththbrsh’ (2024), sound artist Christine Schörkhuber (b. 1982) used four electric toothbrushes and triggered various vibrational states in them by setting the electric current through microcontrollers. The toothbrushes were positioned at different spatial positions around the listener. In one of the three sound scenes, the precedence effect was created by activating the devices in defined chronological sequences. The installation was premiered as part of the project ‘AUDIO GHOSTS’ (Gál et al. Reference Gál, Mayer, Majdak and Pausch2025) and the transdisciplinary festival for music and sound art ‘shut up and listen! 2024’ at SP•CE (Gál Reference Gál2025) in Vienna.
The Clifton effect (Clifton Reference Clifton1987) demonstrates that a sudden switch of the order of the leading and lagging clicks, with a delay entailing a precedence effect, initially leads the listener to perceive simultaneous clicks appearing from the two loudspeaker directions, before the perceived source direction becomes dominated by the leading click (Litovsky et al. Reference Litovsky, Colburn, Yost and Guzman1999; Brown et al. Reference Brown, Stecker and Tollin2015). The effect can be explained by perceptual buildup and breakdown mechanisms (Djelani and Blauert Reference Djelani and Blauert2001; Brown et al. Reference Brown, Stecker and Tollin2015).
The Haas effect (Haas Reference Haas1951, Reference Haas1972) is closely related to the precedence effect but is situated in the realm of speech. Specifically, a delay of up to about 30 ms in the lagging speech signal merely increases the apparent source width and results in a perceived directional shift of the leading speech signal without creating an annoying echo. Above approximately 30 ms, chances of detecting an annoying echo increase, potentially leading to the complete loss of speech intelligibility at higher delays. The effects depend on speech rate, reverberation time and characteristics of the lagging speech signal such as its angle of incidence and timbre (Haas Reference Haas1972).
The Franssen effect (Franssen Reference Franssen1960) entails an illusory directional perception dominated by a leading stimulus with sharp onset in a reverberant environment. In experiments with two loudspeakers, listeners were presented a pure tone with sharp onset played via the first loudspeaker (Hartmann and Rakerd Reference Hartmann and Rakerd1989). The amplitude exponentially decayed, while the second loudspeaker simultaneously played the same pure tone but with an exponential fade-in. This triggers the Franssen effect: in a reverberant environment, the perceived sound direction corresponds to the direction of the loudspeaker initiating the playback with the sharp onset. Note that this effect works poorly when presented in an anechoic chamber, indicating that human sound-localisation mechanisms overweigh transients while failing to extract sound-localisation cues from steady-state pure tones that are smeared in the presence of room reflections.
Auditory saltation (Bremer et al. Reference Bremer, Pittenger, Warren and Jenkins1977) can be considered the auditory equivalent of the cutaneous ‘rabbit’, a tactilely perceptible illusion in which a sequence of three tap groups, each group being sequentially presented to the forearm at spatially separated positions, may be perceived as being spread across these three locations (Geldard and Sherrick Reference Geldard and Sherrick1972). To investigate auditory saltation, three loudspeakers were arranged in the horizontal plane sequentially reproducing groups of clicks per direction (Bremer et al. Reference Bremer, Pittenger, Warren and Jenkins1977). Depending on the inter-stimulus interval, the number of clicks per group, and the temporal click regularity, participants either perceived the clicks coming clearly from the three loudspeaker directions, from three blurred directions but still perceivable in three groups of directions, or as (sometimes evenly) distributed between the three loudspeaker directions.
3.1.3. Pitch-related illusions
Pitch perception describes the ability to assign tonal sounds to a frequency-based reference system. A melody is a chronological and rhythmical sequence of individual pitches perceived as a salient musical entity. Pitch-related illusions mainly influence pitch perception but are largely independent of auditory illusions from other categories (see Table 1).
Dichotic pitch describes an illusory pitch perception based on manipulations of the interaural time differences or interaural phase relationships in the dichotically presented noise stimuli (Culling et al. Reference Culling, Marshall and Summerfield1998a, b). A dichotic melody leads to the perception of an illusory melody and can be created by presenting pure-tone clusters via headphones with a specific interaural time difference and applying interaural phase shifts or instantaneous frequency shifts at the onset times of the targeted melodic tones (Cramer and Huggins Reference Cramer and Huggins1958; Kubovy et al. Reference Kubovy, Cutting and McGuire1974). Listeners reported hearing a chime-like melody defined by the onsets but were not able to hear a melody when the sequence of stimuli was presented monaurally. Alternatively, a target melody can be embedded in a sequence of binaural beats (cf. Section 3.1.1), centred around the target pitches (Kubovy et al. Reference Kubovy, Cutting and McGuire1974). Consequently, the resulting dichotic melody is perceived as more salient, while the binaural beats sometimes add roughness and appear as a rather independent perceptual entity.
Ambiguous circular scales or glissandi reveal how pitches are established by our auditory system and cause an auditory illusion of a seemingly infinite continuation of pitch. The Shepard scale (Shepard Reference Shepard1964) can be regarded as the auditory counterpart of visual infinite-staircase illusions (Escher Reference Escher2000). This auditory illusion is based on a superposition of pure tones arranged in octaves, so-called Shepard tones, whose pitches are simultaneously increased or decreased stepwise over time. Gradually increasing and decreasing the loudness over time of the lowest and highest octave harmonics, respectively, with advancing pitch levels creates the auditory illusion of an infinitely ascending or descending scale. This concept has been extended in the Shepard–Risset glissando (Risset Reference Risset1969; Shepard Reference Shepard1983), in which pitch and loudness of the harmonic components change continuously over time.
Seemingly infinite pitch continuations: application examples
Shepard scales and Shepard-Risset glissandi have been extensively used in various compositions (for reviews, see Braus Reference Braus1995, Wannamaker Reference Wannamaker2008). For example, the pieces ‘Mutations’ (1969) and ‘Computer Suite for Little Boy’ (1969), both by Risset (Reference Risset1994), are early electroacoustic compositions using pitch paradoxes. James Tenney (1934–2006) applied Shepard-Risset glissandi in his electroacoustic composition ‘For Ann (rising)’ (1969) and later adopted the composition for instruments, entitled ‘For 12 Strings (rising)’ (1971; Wannamaker Reference Wannamaker2008). Imitations of infinite scales can also be found in György Ligeti’s (1923–2006) piano etudes ‘No. 9: Vertige’ (1990), ‘No. 13: L’escalier du diable’ (1993) and ‘No. 14: Coloana infinitǎ’ (1993) (for analyses, see Steinitz Reference Steinitz1996).
For a quadrophonic sound installation ‘earlift’ (2011), Gál (Reference Gál2015) used sounds pre-recorded in the lobby of an elevator. The word ‘lift’ in the installation title also means to ‘elevate’, hinting the use of sound layers consisting of continuously ascending or descending tonal spirals as a tribute to Shepard (Reference Shepard1983) and Risset (Reference Risset1969).
A perpetual melody is a perceptually seemingly never-ending melody and is an extension of Shepard tones to musical melodies (Patrício Reference Patrício2012). Two consecutive Shepard tones are either perceived as rising or falling, determined by the direction of the shorter distance between the first and second tone along the pitch-class circle (Shepard Reference Shepard1964). This observation led to the discovery of a tritone paradox (Deutsch Reference Deutsch1986), which occurs when the distances are equal in both directions along the pitch-class circle. Pairs of Shepard tones, split into various pitch classes and presented to listeners, created a perceived pitch direction which depended on the stimulus-presentation order and the individual internal pitch template. The internal template specifies the division point of the pitch-class circle below and above which the stimulus pairs are perceived as falling or rising, respectively.
Combination tones (for a historical review, see Plomp Reference Plomp1965) are tones perceived but physically not present in a sound consisting of two pure tones. Given two pure tones at frequencies
${f_1}$
and
${f_2}$
, with
${f_1} \lt {f_2}$
, combination tones of first order (quadratic difference tone), second order (cubic difference tone), and fourth order are most salient, and can be calculated as
${f_{{\rm{C1}}}} = {f_2} - {f_1}$
,
${f_{{\rm{C2}}}} = 2{f_1} - {f_2}$
, and
${f_{{\rm{C3}}}} = 3{f_1} - 2{f_2}$
, respectively. The mechanism of perceiving combination tones is the mechanical non-linear nature of the human cochlea (Plomp Reference Plomp1965; Goldstein Reference Goldstein1967; Goldstein and Kiang Reference Goldstein and Kiang1968; Ashmore et al. Reference Ashmore, Avan, Brownell, Dallos, Dierkes, Fettiplace and Grosh2010). In the medical field, combination tones can be measured and are commonly referred to as distortion-product otoacoustic emissions (Kim et al. Reference Kim, Molnar and Matthews1980), the absence of which is considered an indicator for sensorineural hearing loss. Combination tones can represent an intriguing compositional tool because they turn the ear itself into an instrument (Haworth Reference Haworth2011, Reference Haworth2012).
Combination tones: application examples
Early examples are the media opera ‘Intelligent Life’ (1981–2) and ‘Head Rhythm’ (1999), both by Maryanne Amacher (1938–2009) (for notes, see Cimini Reference Cimini2017). Further, Kendall et al. (Reference Kendall, Haworth and Cadiz2014) describes methods facilitating sound synthesis based on combination tones. Later, in ‘ADS 4 (Correlation Number One)’ (2010), an eight-channel computer-music piece (Haworth Reference Haworth2011, Reference Haworth2023a), combination tones were supposed to be consciously stimulated and explored in the scope of electroacoustic compositions (Haworth Reference Haworth2023b).
The audio-architectural installation ‘Defragmentation/blue’ (1999; Gál Reference Gál2000) was created in collaboration with the Japanese architect Yumi Kori (b. 1960) and premiered at the sound and intermedia gallery Studio Five Beekman in New York City (1999). The composition integrates combination tones and psychoacoustic masking effects (Fastl and Zwicker Reference Fastl and Zwicker2007). Progressing ‘frozen time-windows’ led to a superposition and an interference of partial tones, which varied depending on the position of the listener in the room.
In a second scene of ‘ththbrsh’ (2024) by Christine Schörkhuber (cf. Section 3.1.2), different vibrational states of the toothbrushes created different pitches, which led to the perception of combination tones.
3.1.4. Pitch-related and spatio-temporal illusions
Temporal pitch changes influence the perceived directions of sound sources or musical entities (Deutsch Reference Deutsch1975, Reference Deutsch and Deutsch2013). Such illusions cannot be clearly assigned to the categories of pitch-related or spatio-temporal illusions. The auditory brain seems to ‘re-organise’ or ‘group’ perceptual input, stressing the crucial role of cognitive behaviour in shaping perceptual pattern recognition (Bregman Reference Bregman1990; Deutsch Reference Deutsch and Deutsch2013).
The octave illusion (Deutsch Reference Deutsch1974, Reference Deutsch1975) can be created by two pure tones spaced at an octave and presented via headphones over the left and right channel, respectively, in an alternating way switching every 250 ms. Listeners reported hearing a single tone with constant or slightly varying pitch spatially oscillating between ears, hearing a single tone which pitch was changing by one octave while spatially oscillating between ears, or having a more complex spatial and pitch-related percept. These percepts were influenced by the handedness, which also dominated on which side the listeners perceived the higher and lower pure tone.
Melodic patterns can lead to illusory spatial percepts. In the scale illusion, the notes of an ascending and a descending scale were reproduced sequentially at the left and right ear via headphones (Deutsch Reference Deutsch1975) or loudspeakers (Butler Reference Butler1979). Each tone pair was presented simultaneously, but after each tone pair, the channels were switched, resulting in an alternating presentation of the tones belonging to the respective scale variant. Consequently, some listeners reported to perceive two symmetric melodies around the scale’s centre note, suggesting perceptual grouping by pitch proximity and spatial reorganisation (Deutsch Reference Deutsch and Deutsch2013). Depending on whether the listeners were right- or left-handed, the higher notes were often perceived on the right or left side, respectively. Being based on melodic patterns, the chromatic illusion (Deutsch Reference Deutsch and Deutsch2013) and the cambiata illusion (Deutsch Reference Deutsch and Deutsch2013, Reference Deutsch2003) produce similar percepts as the scale illusion.
Scale illusion: application examples
In Ichimiya and Ichimiya (Reference Ichimiya and Ichimiya2023), the scale illusion has been extended to dichotically presented note patterns other than just scales for the creation of musical melodies, opening up new possibilities for its use in electroacoustic compositions.
In the experiments on the glissando illusion (Deutsch Reference Deutsch1995; Deutsch et al. Reference Deutsch, Hamaoui and Henthorn2007), participants listened to an oboe tone of constant pitch, and an ascending or descending glissando. Both stimulus types were reproduced spatially alternating between two symmetrically arranged loudspeakers. While the oboe was perceived as alternating, the glissando variants were not only perceived as continuous streams but also as continuously changing their spatial position on elliptic trajectories within the loudspeaker positions depending on their pitch progression and the handedness of the listeners.
3.1.5. Rhythmic illusions
Fundamentally, rhythm can be regarded as ‘the ordered characteristic of succession’ (Fraisse Reference Fraisse and Deutsch1982). Specifically related to melodies in music, rhythm can be defined as ‘the serial pattern of variable note durations in a melody’ (Schulkind Reference Schulkind1999).
The perceptual concepts applied for the creation of illusory pitch circularity (see Section 3.1.3) can be adapted for the generation of seemingly infinitely accelerating or decelerating rhythms.
To create a Risset rhythm, different instances of a rhythmic stream are superimposed (Risset Reference Risset1986). Each instance is running in parallel but on a different relative timescale. Slower and faster instances increase and decrease in level over time, respectively. Presenting the rhythmic streams in a loop provokes the illusion of an infinitely accelerating or decelerating rhythm. Risset (Ibid.) argued that the perceived rhythm circularity can probably be attributed to proximity effects and stream-segregation mechanisms (Bregman Reference Bregman1990), and implied a connection to the self-similarity of Mandelbrot (Reference Mandelbrot1979) sets (for formal definitions of this rhythmic illusion, see Stowell Reference Stowell2011).
Risset rhythm: application examples
The piece ‘Moments Newtoniens’ (1977) by Risset (Reference Risset1994) combines synthetic and acoustic sounds, and includes passages with rhythm paradoxes. Stowell (Reference Stowell2011) picked up on Risset’s concept and proposed a mathematical framework that facilitates a mapping of note patterns or audio clips to Risset-rhythm metabars. The flexibility of this framework for electroacoustic compositions was demonstrated by means of various compositional considerations (ibid.).
3.1.6. Temporal illusions
Time perception per se is not directly linked to other senses, but can be influenced by the sense of hearing, creating temporal illusions (for reviews, see ten Hoopen Reference Ten Hoopen2008; ten Hoopen et al. Reference Ten Hoopen, Miyauchi, Nakajima and Grondin2008).
The divided-time illusion makes a time interval containing clicks to be perceived longer than the same but empty interval (Hall and Jastrow Reference Hall and Jastrow1886). The illusion is also perceivable in rhythmic sequences (Repp and Bruttomesso Reference Repp and Bruttomesso2010). While diotically being presented a sequence of isochronous tone beats via headphones, musicians and non-musicians were asked to either continue the beat in synchronised mode, or to continue the beat in synchronised mode and fill in one or two additional subdivisions between two beat tones, both times by tapping on a percussion pad. Across the tested target inter-stimulus intervals, both participant groups perceived slower overall beat tempi when the beat sequences were to be filled with intermediate strokes compared with the unfilled variants.
In the sustained-sound illusion, the duration of an interval filled with a continuous sound can be perceived longer or shorter than without that sound (Meumann Reference Meumann1896; Repp and Marcus Reference Repp and Marcus2010). Note that this illusion is not perceived when extended to rhythmic sequences (Repp and Marcus Reference Repp and Marcus2010).
3.1.7. Complex illusions and effects
This section describes illusions and effects related to various auditory categories such as pitch and location of a sound source. As a common ground, central auditory processes, which can be partially explained via principles of Gestalt psychology (Bregman Reference Bregman1990), are required for their perception.
The term missing fundamental refers to an illusory pitch perception when simultaneously listening to two or more pure tones that are multiple integers of a missing fundamental frequency (for a historical review, see Greenberg et al. Reference Greenberg, Marsh, Brown and Smith1987). The perceived pitch of the missing fundamental is equal to the greatest common divisor of the individual pure-tone frequencies and represents the inverse period between the periodic amplitude peaks in the time signal of the complex harmonic sound (Chialvo Reference Chialvo2003). This illusion cannot be attributed to non-linear distortions in the inner ear (cf. Section 3.1.7), but is of central-auditory origin (Schouten Reference Schouten1938). However, when presented a sound consisting of only two pure tones, the greatest common divisor also corresponds to the combination tone of first order. In the event that the greatest common divisor is acoustically measurable (Kim et al. Reference Kim, Molnar and Matthews1980), it is referred to as a combination tone. If it is not measurable but perceivable (Schouten Reference Schouten1938), it is referred to as a missing fundamental. Various models predicting the missing-fundamental frequency for harmonic and inharmonic sounds have been proposed (Chialvo Reference Chialvo2003), among which, the maxima of the predicted distributions of neural firing rates show a high consistency with the perceived pitch (Schouten et al. Reference Schouten, Ritsma and Cardozo1962).
An auditory-continuity illusion occurs when listening to a sound interrupted for a short time interval but having the gap filled with noise. It makes the sound appear as continuous (Bregman Reference Bregman1990) and works for various types of signals such as constant pure tones, frequency-modulated sweeps, familiar melodies and speech. This illusion can be explained via auditory-masking mechanisms with auditory filters having specific critical auditory-filter bandwidths across the signal frequencies (Riecke et al. Reference Riecke, van Opstal and Formisano2008).
Auditory-continuity illusion: application example
In a third sound scene of ‘ththbrsh’ (2024) by Christine Schörkhuber (see Section 3.1.2), the spatial arrangement and sequential activation of the toothbrushes facilitated the creation of an auditory-continuity illusion, which was perceived as a single melody.
Phonemic restoration lets listeners perceive phonemes that were removed from a speech signal or masked by noise. The percentage of missed speech information correlates well with the percentage time of irregularly interrupting silence intervals. The information loss seems to be robust against periodic interruptions if occurring more than 10,000 times per second (Miller and Licklider Reference Miller and Licklider1950). Intelligibility was not impaired when speech and noise were alternated ten times per second, and phonemes can still be perceived when replaced by coughing or tonal sound (Warren Reference Warren1970). However, listeners do not seem to be able to correctly assign the temporal phoneme position. This illusion seems to have its origin in language redundancies and depend on the language skills and familiarity.
Phantom words can be perceived in a looped sequence consisting of two words or a single two-syllabic word. This illusion works when reproduced via two headphone channels (one of which is delayed) and via two loudspeakers (with an inter-channel delay, Deutsch Reference Deutsch2019). The perceived phantom words range from lexical items, through meaningful or nonsense words in the language presented, a second language (Broersma and Cutler Reference Broersma and Cutler2008) or a foreign language, to melodised speech perception. This illusion can be linked to individual expectations, memories, language abilities and related central-auditory processing of spoken language (Deutsch Reference Deutsch2019).
In the Yanny-or-Laurel illusion, a short audio recording of speech can be heard as one of two words: Yanny or Laurel. The words were perceived dichotomously by different groups of listeners. A spectral analysis has shown that the spectra of both words are present in the mixed recording. Some listeners focus on the higher frequencies of the word ‘Yanny’ and ignore the lower frequencies of the word ‘Laurel’ (Pressnitzer et al. Reference Pressnitzer, Graves, Chambers, de Gardelle and Egre2018). It is possible to drive perception towards either word by means of spectral contrast enhancement (Bosker Reference Bosker2018). The illusion is not limited to the aforementioned pair of words but can be extended to other pairs (Guan and Valiant Reference Guan and Valiant2019).
Phantom words: application example
The sound installation ‘PHANTOM WORDS SPATIAL’ by Deutsch et al. (Reference Deutsch, Werner and Meyer2021) was premiered at HKW Berlin as part of the festival ‘The Sound of Distance’. The installation extended the originally proposed stereo loudspeaker setup (Deutsch Reference Deutsch2019) to multiple distributed loudspeakers, repeatedly presenting speech stimuli in an open form. The application of such a spatial loudspeaker configuration in combination with room acoustics aimed at increasing the level of immersion and efficiency in provoking the perception of phantom words.
3.1.8. Cross-signal-type illusions
Signals of a certain type may provoke an illusion in the respective other space of signal type.
In the speech-to-song illusion (Deutsch Reference Deutsch1995), a looped spoken phrase developed to be perceived as melodised, which can also occur as part of a phantom-words illusion (cf. Section 3.1.7). The results from experimental investigations indicated a failing illusory perception if the presented phrase was changed with respect to pitch or syllabic composition across repetitions (Deutsch et al. Reference Deutsch, Henthorn and Lapidis2011). Deutsch et al. (ibid.) hypothesised that speech and music information, though processed along common neuronal pathways, may ultimately lead to different complex percepts.
Speech-to-song illusion: application examples
In the composition ‘It’s Gonna Rain’ (1965) by Steve Reich (b. 1936), the title phrase is repeatedly presented over two channels while undergoing inter-channel phase manipulations to provoke a speech-to-song illusion. In ‘Come Out’ (1966), Reich applied similar methods with the same perceptual objective (for analyses of both pieces, see Schwarz Reference Schwarz1993).
In ‘bestimmung new york’ (1998; Gál Reference Gál1999), the musical qualities of language are explored. This series of works contains 15 short electro-acoustic pieces featuring carefully selected and combined excerpts of voice recordings from 15 people in New York repeating the same sentences and words in their mother tongues. These pieces aimed at demonstrating that language can be perceived as music when detached from its semantic and functional context, focusing on speech characteristics such as melody, rhythm and dynamic features.
Music-to-speech illusions require that a music signal is processed in such a way that it is perceived as a corresponding speech-like signal when reproduced via an electroacoustic device or an instrument. In this context, Ablinger (Reference Ablinger2006) coined the term ‘phonorealism’ as auditory analogy to photorealism.
Music-to-speech illusion: application example
The composition ‘Quadraturen III (“Wirklichkeit”): A Letter From Schoenberg’ (2006) by Ablinger (Reference Ablinger2006) was implemented as a sound installation together with Ritsch (Reference Ritsch2011), imitating speech in the music domain using a piano. For this purpose, spectro-temporal speech features that are vital for speech intelligibility were extracted from the target speech and mapped to technical parameters. Psychoacoustic masking effects (Fastl and Zwicker Reference Fastl and Zwicker2007) were also considered and helped reduce mapping complexity. Based on this mapping, a custom-made apparatus with electromechanical fingers facilitated operating the piano keys (Ritsch Reference Ritsch2011).
3.1.9. Combined auditory illusions and effects
Multiple auditory illusions can also be combined in a single electroacoustic composition. However, such a combination not only requires thorough knowledge about the involved individual auditory-based illusions and effects, but also knowledge and experience about their combined perception. In this regard, amplifying or attenuating perceptual effects can be exploited consciously by the composer.
Example for a combined application of various auditory illusions
The composition ‘Vertizontal Hearing (Up & Down, I Then II)’ (2011–12) by Christopher Haworth (b. 1981) combined the tritone paradox, the Haas effect, combination tones and phantom words, and explored auditory-grouping mechanisms (Bregman Reference Bregman1990) (for a commentary by the composer, see Reimer et al. Reference Reimer, Stein, Hennies, Colbert, Costa, Gormley and Haworth2016). Haworth stressed the importance of analysing and listening to the original demonstration material in a ‘prescribed’ way, that is, as described in supplementary explanations, to be able to actually perceive the intended perceptual objective and effectively aestheticise the material in the form of an electroacoustic composition.
3.2. Auditory-visual illusions and effects
The combination of the auditory and visual domains opens up a space for various illusions and their cross-modal perception (for a discussion, see O’Callaghan Reference O’Callaghan2017).
The McGurk effect (for a review, see MacDonald 2018) demonstrates how vision may dominate speech perception. In the experiments by McGurk and MacDonald (Reference McGurk and MacDonald1976), children and adults both were either presented an audio or an audio-video sequence in which a woman was articulating utterances consisting of various syllable pairs. Participants had to repeat the respective utterances in blocks with auditory-only and auditory-visual presentation. In the auditory-visual blocks, the auditory information was either congruent or incongruent with the visually articulated utterance. Utterances in the auditory-only blocks were repeated with high accuracy by both children groups and the adults. However, the perceptual results in the auditory-visual blocks differed across age groups: while the syllable pairs repeated by children were more influenced by the auditory information, the ones repeated by adults were biased towards the visual stimuli. Such cross-modal conflicts led to high errors and even to the reporting of utterances that were not present in the target stimuli in either perceptual domain.
The ventriloquist illusion (Thomas Reference Thomas1941; for a recent review, see Bruns Reference Bruns2019) occurs when synchronously presenting a visual stimulus that is spatially inconsistent with an auditory stimulus, triggering the localisation of the auditory stimulus at the position of the visual stimulus. It is an example for the perceptual dominance of stimuli with high acuity (here, visual information) over those with a low acuity (here, sound). The ventriloquist illusion has been reproduced using visual stimuli reproduced on a display and virtual acoustic stimuli binaurally reproduced via headphones (Stawicki et al. Reference Stawicki, Majdak and Başkent2019). Closely related, the ventriloquist aftereffect leads the auditory stimulus remaining spatially biased towards the visual one due to unisensory auditory recalibration (Bruns Reference Bruns2019).
The sound‑induced flash illusion serves as a disproving example that vision is not always the prevailing sense in multisensory perception and demonstrates how perception of a visual stimulus changes when synchronously presented with an auditory stimulus (Shams et al. Reference Shams, Kamitani and Shimojo2000, Reference Shams, Kamitani and Shimojo2002). When asked to report the number of white flashes appearing on a black display, participants perceived a single flash as multiple flashes when presented together with two or more beeps. This illusion can be explained via polysensory brain mechanisms (ibid.) and works also with auditory and visual stimuli being spatially separated from each other (Innes-Brown and Crewther Reference Innes-Brown and Crewther2009).
The cross-bounce illusion describes a phenomenon in which the visual perception of the trajectories of two objects crossing paths can be influenced by an auditory stimulus (Sekuler et al. Reference Sekuler, Sekuler and Lau1997). Presenting a short sound at the moment the objects bump into each other can lead to the perception that the objects bounce off each other instead of passing through each other. The illusion can be explained via effects related to multisensory integration, auditory grouping, as well as experience and imagination (Watanabe and Shimojo Reference Watanabe and Shimojo2001; Berger and Ehrsson Reference Berger and Ehrsson2017).
4. Conclusions
This article aims at bridging the gap between a scientific understanding of auditory-based illusions and effects, and artistic sound installations. First, we define objective, perceptual, philosophical, social and cultural aspects of sound installations. Then, we describe psychoacoustic fundamentals, scientific experiments and application examples of selected auditory-based illusions and effects. Finally, in an extensive table supported by key references to methods and findings, we link various objective aspects of sound installations with the auditory-based illusions and effect, inspiring their integration in sound installations underpinned with psychoacoustic knowledge. Such an interdisciplinary approach aims at helping composers and sound artists to increase the perceptual appeal of sound installations even further.
