2. TIBETAN BUDDHISM AND SONIC AWARENESS

Tibetan Buddhism and its contemplative cultural arts, which uniquely embrace the legacy of both Indian and Chinese Buddhism (Stein Reference Stein1972), includes stunning arts such as the Mandala Sand Art (Bryant and Dalai Lama Reference Bryant1992), monastic throat-singing (Smith, Stevens and Tomlinson Reference Smith, Stevens and Tomlinson1967) and Cham dance (Pearlman Reference Pearlman2002) as well as clinically proven effective contemplative techniques to improve well-being (Rajagopal, Mackenzie, Bailey and Lavizzo-Mourey, Reference Rajagopal, Mackenzie, Bailey and Lavizzo-Mourey2002; Kabat-Zinn Reference Kabat-Zinn2003; Ospina et al. Reference Ospina, Bond, Karkhaneh, Buscemi, Dryden and Barnes2008; Loizzo et al. Reference Loizzo, Peterson, Charlson, Wolf, Altemus, Briggs, Vahdat and Caputo2010). Tibetan cultural arts are a crucial part of Tibetan Buddhism, which exteriorise the profound and abstract philosophy into tangible metaphors and forms. Practising Tibetan cultural arts is an example of embodied cognitive practice in Tibetan Buddhism.

Tibetan Buddhism, or broadly speaking, the Vajrayāna Buddhist tradition, includes somatic practices that engage a practitioner’s physical body and their six sensory consciousness (eyes, ears, nose, tongue, body and mind) as a vehicle to reach enlightenment (Baker Reference Baker, Baier, Maas and Preisendanz2018). For example, a mantra or a dharani practice is a sonic meditation practice that lets practitioners experience their minds through sounds. Through reciting specific sacred utterance (e.g., ‘Oṃ Āḥ Hūṃ’), a numinous sound (e.g., throat-singing), a seed syllable Bīja, phonemes, a group of words or phrases in Sanskrit that are believed to have psychological and/or spiritual powers practitioners can achieve different stages of self-awareness and consciousness depending on their understanding in the ultimate Buddhist view of emptiness, which sometimes is interchangeable with the concept of ‘non-self’ or ‘non-duality’ (Lingpa, Mahapandita and Rinpoche Reference Lingpa, Mahapandita and Rinpoche2006; Lowe Reference Lowe, Clarke and Clarke2011).

Another example of embodiment practice in Vajrayāna is Mudras meditation. This method requires practitioners to perform specific symbolic ritual finger/hand/arm/gestures to materialise a specific dharma doctrine. Sometimes whole-bodied postures and internal actions (e.g., controlling diaphragm, abdomen and other parts of the internal body to regulate the energetic flow of ‘praṇā vāyu’ – the essence of breathing and vitality) (Gyatso Reference Gyatso2007; Nair Reference Nair2013). Most of the time, specific ritual musical instruments and chants are performed to accompany the meditation. Depending on the purpose of the meditation, sometimes the human thighbone trumpet (‘Kangling’) and/or drums made of two skull halves (‘Damuru’) are used to sonify/materialise the ultimate wisdom of Buddhist philosophy – non-duality and emptiness (Vandor Reference Vandor1974).

Similar to Mudras meditation, Cham dance also can be seen as an embodied cognitive practice. The main Cham dancer usually is an accomplished Buddhist teacher who performs a lively masked and costumed Yidam Footnote ¹ dance with sacred music ensembles and chants, while conducting an intense meditation in their mind at a grand ritual gathering (Figure 1). During the event, other monks perform as backing dancers, wearing masks of deities, humans or animals. The dancers use their body expressions and meditative mind coupled with the auditory stimuli to help them embody the supreme power and eventually become enlightened beings (Samuel and David Reference Samuel and David2016). These physical-to-spiritual Tibetan contemplative practices through sounds, visuals and bodily activities anchor the philosophical root and provide rich multisensory design inspirations for embodied sonic meditation.

Figure 1.

A Cham dance meditation with sacred music ensemble in Garzê Tibetan Autonomous Prefecture.

Besides Tibetan Buddhism, there are three Western composers’ works that are highly relevant to embodied sonic meditation practice. The first one is Pauline Oliveros’s practice of ‘deep listening’. In 1974, Oliveros composed a series of works called ‘Sonic Meditations’. Through this work, she disrupted traditional Western music by practising an ancient eastern philosophical concept – meditating through sound. More than 2,500 years ago, meditating through sound has been called ‘experiencing Sonic Vedanā’ in Buddhist practice (De Silva Reference De Silva, Marked and Ames1995). This ancient practice encourages practitioners to treat the sound as is, with no difference between like and dislike, and this non-duality way of thinking can be implemented to all aspects in life to achieve a ‘non-self’ emptiness without any attachment (Dalai Lama and Berzin Reference Dalai and Berzin1997).

Oliveros adopted this concept and further developed it into a contemporary artistic practice of improvising, composing and teaching experimental music. She advocated a deep listening ^{Footnote 2} practice that trains our ears and mind to consciously appreciate all sounds to increase sonic awareness, thus creating an open space for music making and listening (Oliveros Reference Oliveros2005). Actual sound making in Oliveros’s sonic meditations was ‘primarily vocal, with sometimes hand clapping or other body sounds. Occasionally, sound-producing objects and instruments are used’ (Osborne Reference Osborne2000). Her work mainly focused on sound and mind.

Another female composer, Éliane Radigue, is well known as an electronic musician using ARP2500 modular synthesisers and magnetic tape to sonify her lifelong Tibetan Buddhist contemplative practices, insights and reflections. Her electronic music compositions demonstrate how electronic musical instruments and abstract sounds have the ability to vividly narrate different meditative states of mind. Although Radigue has shifted gears to acoustic music composition during the past two decades, her main music vocabulary rooted in Tibetan Buddhism has never changed. Her work in Trilogie de la Mort (Radigue Reference Radigue1998) is a musical materialisation of one of the most important Tibetan Buddhist teachings in Bardo, which is the deepest meditative practice that guides people through their next inevitable stage of life – the death (Fremantle Reference Fremantle2000).

Phillip Glass’s Symphony No. 5, Requiem, Bardo, Nirmanakaya (Glass Reference Glass2000), tells the sonic story of Saṃsāra – a cyclic view of time, life and the universe. This Buddhist view is commonly demonstrated in the mother nature: in the rhythm and energy of the orbits of the heavenly bodies, the changing of the seasons, and the in and out of human breath. As we live our lives, we navigate these cycles, a wheel of time between birth and death, as we already have for millions of years, without beginning or end. Although Glass has been a Tibetan Buddhist practitioner since the late 1960s, he rarely talks about the relationship between his spiritual practice and creative work. His Symphony No. 5 is perhaps the most well-known work with explicit references to Tibetan Buddhism. However, it could be argued that his minimal music practice is always meditative and delivers an experience of Sonic Vedanā. It is also notable that he strives to avoid cultural appropriation and authentically conveys his internal experiences only through music itself.

There are many other influential composers’ works that have been built upon Tibetan Buddhism or culture, such as musique concrète pioneer Pierre Henry’s Le voyage (Reference Henry1962) and Zhang Xiaofu’s Nuo-rilang (Reference Zhang1996). While these practices treat Tibet as a source of inspiration, they are rather an artistic exploration instead of contributing to the contemplative culture. For example, Henry’s musical narrative of the stages of transcendence after death is undoubtedly splendid. However, without any meditative practice or opportunities for close contact with Tibetan Buddhism in his lifetime, Le voyage is a musical journey of pure fantasy. Similarly, despite the stunning soundscape that Zhang created using both electric sounds, percussions, and Tibetan Buddhist ritual instruments, Nuo-rilang is named after a male God of Bön, a primitive Tibetan religion that is self-distinguished from Tibetan Buddhism. Therefore, I argue that while both Zhang’s and Henry’s creative perspectives focused on exploring the beauty of the Tibetan cultural mysteries, they inhabit a creative space separate from the practice of embodied sonic meditation. In other cases, many composers use mind-expanding drugs to help recreate a meditative mind that generates transcendent sounds. These creative works also exist in a separate space, as the stages of mind are completely different between using drugs versus self-induced experience through long-time meditation practice. It is like comparing organic food with artificial flavours and they are never the same, from both cause and effect. Indeed, it is difficult for either a Western or an Eastern mind to comprehend this meditative journey without dedicatedly practising it.

3. BODY AS AN INSTRUMENT

As a vocalist, my body is both a musical instrument and an instrument for expressing my emotions and intentionality on the stage. I can position the body and manipulate the vocal system in a way that improves tone quality and/or conveys non-verbal communication to the audience. In the sonic world of electroacoustics, these natural body expressions and gestures can be adapted, tracked and implemented as a three-dimensional human interface to control real-time audio processing parameters, thus enriching vocal expressions, computer-generated sounds or pre-recorded samples in an intuitive way.

As early as 1984, Michel Waisvisz (Reference Waisvisz and Truax1985) built a three-dimensional, gestural-control, human interface called ‘Hands’ to explore the body–sound–mind relationship. He observed: ‘My work has been trying to bring back a unity between physical and mental activities … There is a kind of physical excitement … Moving my hands apart is definitely showing that things (sounds) are going bigger and wilder’ (Chadabe Reference Chadabe1997). Many musicians, such as Laetitia Sonami, Pamela Z and Imogen Heap have been using body-controlled human interfaces to communicate their identities in gender, race, nation and digital culture. At the beginning of the twenty-first century, Cadoz and Wanderley (Reference Cadoz, Wanderley, Wanderley and Battier2000) established the field of ‘gesture-music’, which investigates the body–sound–mind connections in computer music and how we can treat the technology-augmented body as an instrument to better convey musical ideas to the audience through gesture-sound mapping strategies.

Indeed, the cognitive perceptions of body-sound mappings and sensory-motor couplings are naturally embedded in us, so we can read body language, voice and other sonic events and apply meanings to them. This begins at infancy and is refined to an art by adulthood (Darwin and Prodger Reference Darwin and Prodger1998). Furthermore, clinical research has supported that auditory cues (sonic sense) can effectively regulate a person’s attention (mind) and modify the gait of a person with Parkinson’s disease, thus enhancing the patients’ walking performances (motor/body) (Howe, Lövgreen, Cody, Ashton and Oldham Reference Howe, Lövgreen, Cody, Ashton and Oldham2003). Scientific study also explored the role of valence and arousal in the cross-modal bias, in the context of dance movements and music stimuli. It was found that the audience’s emotional experiences to dance were enhanced through music (Christensen, Gaigg, Gomila, Oke and Calvo-Merino Reference Christensen, Gaigg, Gomila, Oke and Calvo-Merino2014). These auditory sensory-motor coupling studies provide evidence to support the potential of sound–body–mind integration not only in music-making but also in personal training and daily practice.

Marshall McLuhan ([1964] Reference McLuhan1994) stated that media and technology extend the human body. The machines we built have augmented us, so we can see, hear, touch, taste, smell and feel what we previously could not. From computer-assisted algorithmic composition beginning in the 1950s to the 1960s biofeedback experiments; from VR DMIs and wearable interfaces to intersections between music and artificial intelligence, the music technology that we have been designing and using gradually become a part of our bodies and have changed our collective musical behaviours. In other words, we have become augmented musical post-humans (Cecchetto Reference Cecchetto2013). Therefore, the sonic events we create and the technology-augmented post-human bodies we possess work together to form our sonic minds.

4. DESIGNING EMBODIED SONIC MEDITATION

Designing DMIs for embodied sonic meditation is both an instrument design and a meditative cultural experience design process. Therefore, I divide the design thinking process of ‘embodied sonic meditation’ into three individual parts, each backed by interdisciplinary research, from STEM to social science and humanities perspectives. These three components are embodiment, sonic empathy and the meditative experience. Fifteen design principles are proposed to guide the design process.

First, the word ‘Embodiment’ has been broadly used. The following four meanings are most relevant: 1) a sound being heard and ‘seen’ simultaneously with the performer’s body; 2) cognition that is grounded in sensorimotor experience such as bodily activities and senses; 3) ownership: having, controlling, possessing – the instrument is acting like a part of the performer’s body; 4) a performer learning the subtleties of playing an instrument partly due to the feedback that they receive while playing. ‘Embodiment’ in this DMI design context is a combination of the above meanings: the DMI is working seamlessly with the performer and augmenting the performer’s voice or generated-sound so well that it becomes the performer’s extended body/voice; the sound must be heard and morph simultaneously with the performer’s body movement being seen; the gestural control and mapping design must be based on our cognitive capacity grounded in our bodily activities, environment and learned knowledge.

Wessel and Wright (Reference Wessel and Wright2002) advocate designing DMIs in a metaphorical ‘embodied’ intimate way, so that users can better understand the designed features, effectively adapt the instrument and feel good about using it. Fels (Reference Fels2004) defines his ‘embodiment’ as achieving a ‘high degree of intimacy such that [she or] he embodies the instrument’. From the multisensory design perspective, sonic, visual, tactile and kinesthetic feedback has been studied and used to achieve embodiment. Wang proposes ‘human-centred’ design principles in visual feedback for computer music, such as ‘induce [the] viewer to experience substance, not technology; hide the technology’ and ‘be whimsical, organic: glow, flow, pulsate, breathe: imbue visual elements with personality’ (Wang Reference Wang2014). Essl and O’Modhrain (Reference Essl and O’Modhrain2006) underscore the significance of ‘the embodiment of specific knowledge’ and ‘how the musician’s working model of the dynamics of a system as complex as a musical instrument is built up’.

As such, the following principles are proposed to achieve ‘embodiment’:

• Low latency for control intimacy.

• Ensure software/hardware’s reliability and predictability.

• Ergonomic and intuitive control (so the DMI becomes the performer’s extended body).

• Hide the technology and make it natural yet magical.

• Engage in multisensory interactions.

Second, from the ‘sonic’ perspective, Roads’s ‘selection of economy’ in musical decision-making and aesthetic pursuit (Reference Roads2015) is aligned with Wessel and Wright’s DMI design suggestion of ‘parsimony’, which means using the simplest method to produce and not compromise a specific musical result. Simon’s studies in decision-making and in designing artificial systems also address the concept of ‘efficient strategy’ (Simon Reference Simon1996). An example of applications of this in technology development is the recent trend in the reduction of Size, Weight, and Power consumption as well as of cost (SWaP + cost), especially in the area of designing sensing technology and embedded systems.

Furthermore, a designer, composer or performer’s sonic mind can be really divergent. Building on Peirce’s (Reference Peirce and Buchler1902) philosophy of semiotics, Cumming (Reference Cumming2000) studied music subjectivity from a performer’s perspective. She argued that music contributes to its own subjectivity. This ‘sonic self’ can be negotiated and shaped by performer through the instantaneous way of playing a note (short-time scale), the musical gestures created during a phrase (mid-time scale) and the ‘willed direction’ to which a performer intends to lead the whole piece (long-time scale). Eventually, the negotiation between the composition and the performer leads to a non-dual music subjectivity (Legg Reference Legg2002). If we follow Cumming and expand her perspective, the sonic subjectivity between the designer, performer/user, DMI and music itself should be negotiable and fluid. Therefore, it is vital for the designer to engage both the performer, users, audience, engineers or other important stakeholders during the DMI design and validation process.

Moreover, the sonic affordance of the DMI determines its design. First coined by Gibson (Reference Gibson1966), ‘affordance’ in design refers to the capacity of understanding by the design’s observers/users and based on this capacity, the perceived function that this design could potentially offer to the observers/users, depending on their current intentions. Tanaka (Reference Tanaka2010) summarises a series of research in affordance in computer music. He points out that the studies of music affordance should concern ‘not just the primary user, the performer, but also the audience as observer and perceiver’, because music affordance is ‘situated in music as a social, public activity’. This suggests that the gesture-sound mapping strategies should be intuitive and transparent to the audience, the performer and whomever intends to use it.

Meanwhile, Magnusson (Reference Magnusson2010) suggests that the affordances of a DMI change when they are examined within different cultural scopes. Indeed, culture is essential and fluid. From the anthropological approach, culture is the total way of life of ‘a people, a period, or a group’. It is a way of thinking, feeling and believing (Swidler Reference Swidler1986). From the literary perspective, culture is defined as ‘a general process of intellectual, spiritual, and aesthetic development’ (Williams Reference Williams2013). From the social science perspective, Geertz used the metaphor of ‘spinning webs of significance’ to relate how we human beings search for, make and impose meaning through the webs of culture to create a sense of belonging, inclusion and significance in society (Geertz Reference Geertz1966). Since meaning can be constantly created, renewed and adapted through one’s body and mind, I communicate my understanding (meaning-making) of Tibetan contemplative philosophy and its way of life to people cross-culturally through a shareable musical, physical and spiritual experience by implementing my DMIs to compose proof-of-concept pieces. The goal is to create sonic empathy and awareness, as well as make authentic connections between people and with oneself. As a powerful form of digital cultural arts, embodied sonic meditation carries ideologies, projects identities, creates symbolic meanings and embodies social implications.

As a result, the following principles are proposed to create meaningful sonic interactions and sonic empathy:

• Achieve aesthetics through efficiency.

• Consider dramatic and obvious gestures and intuitive mappings to achieve transparency.

• Tailor the DMI to the specific piece and fit the selected cultural context of the piece.

• Create abstract visual and sonic stimuli to make the piece meet the general audience’s contemplative needs from diverse cultural backgrounds.

• Validate the DMI from the performer, user and audience perspectives.

Third, for designing meditative experience, there are at least eight stages of meditative states before a practitioner can reach the first state of enlightenment in Tibetan Buddhism. Most of them are highly advanced and have not been scientifically studied. Currently, embodied sonic meditation focuses on designing multisensory bodily activities that facilitate the entrance-level Focused Attention (FA) meditation that promotes calm, improves attention function and induces self-consciousness with far-reaching goals such as a heightened sense of well-being and the cultivation of empathy (Lutz, Slagter, Dunne and Davidson Reference Lutz, Slagter, Dunne and Davidson2008). FA meditation entails voluntary focusing attention on a chosen object in a sustained fashion, such as a visual object, a visualised image, breath sensations and auditory stimuli. From a top-down perspective, training in core cognitive processes using FA meditation has been shown to cause increased performance in smaller processes and it may also produce lasting changes in mental function that translate to novel contexts (Slagter, Davidson and Lutz Reference Slagter, Davidson and Lutz2011).

On the other hand, Csikszentmihalyi’s Flow Theory (Reference Csikszentmihalyi1975) describes ‘a state of peak enjoyment, energetic focus, and creative concentration’ when people are doing highly engaging bodily and creative activities such as sports, dance and music composition/performance. This described state of ‘flow’ is very similar to FA meditation. Studies using Flow State Scale-2 (FSS-2) questionnaire (Jackson and Marsh Reference Jackson and Marsh1996) in music performance suggest that high flow can be achieved especially when: 1) the challenges of performing this activity and the performer’s skill is well-balanced; 2) the experience of performing the task is highly intrinsically enjoyable; and 3) the performer has a sense of control over this activity (Wrigley and Emmerson Reference Wrigley and Emmerson2013).

The above scientific evidence and the nature of meditation indicate that designing a meditative DMI should first focus on attention regulation and user experience optimisation, especially for the general users who have no previous meditation experience. Cook (Reference Cook2017) proposed his DMI design principle in ‘Everyday objects suggest amusing controllers.’ In this case, this principle can be translated into ‘enjoyable movements/activities suggest high-flow, meditative gestural control mechanisms’. I also use the metaphor of a hook to describe clever DMI feedback features (e.g., sonic, visual, tactile and kinesthetic) that can either optimise the user’s experience to encourage the user to stick with the DMI or help the user and the audience to focus on or even ‘fall in love’ with the experience. By analogy to pop music, the hook is a catchy, memorable melody. An example of a good hook would be a set of natural mappings from the singer’s bodily interactions with her microphone stand to real-time vocal processing in Hewitt’s Emic (Hewitt and Stevenson Reference Hewitt and Stevenson2003). I define hook from the perspectives of both DMI design and compositional process design. For instance, a beautiful acoustic sound, a visually beautiful performance practice and a unique audio effect can all be considered as hooks. In brief, a perfect hook contains a variety of control feedback and/or engaging features so the design can hook in both the user and audience to create sonic empathy.

Subsequently, to design a meditative DMI, the following principles can be applied:

• Enjoyable movements/activities suggest high-flow, meditative gestural control mechanisms.

• Create a hook.

• Strike a balance between challenges and skills, as well as easy-to-use and virtuosity.

• Set up a safe environment that allows users and audience to have a sense of control.

• A ‘shut-down’ or ‘off’ gestural mechanism that allows the performer to have total control of the overall volume and density of the ongoing sound.

As we can see that the first five design principles for embodiment are also critical for designing meditative experiences to ensure a smooth high-flow state and user stickiness/engagement. It is clear that physicality (embodiment) and spirituality (meditation) are unified in an inseparable non-duality form through sound arts in embodied sonic meditation practice and designing its DMIs.

5. EMBODIED SONIC MEDITATION PRACTICES

Significant embodied sonic meditation practices have been done in the area of integrating bio-feedback techniques in meditative compositional environments since the 1960s, such as those famous bio-music pieces from Alvin Lucier, Richard Teitelbaum, David Rosenboom and Pauline Oliveros (Joseph Reference Joseph2011). These practices mainly focused on meditating with one’s brain activities using electroencephalography (EEG) or other bio-data monitoring devices in a musical way. Body movements are prohibited except the performer’s intense brain labour was monitored and processed by machines. A recent work that carries out this tradition is Music Re-Informed by the Brain by Alex Chechile (Reference Chechile2007). While Chechile developed this bio-feedback interactive music system, he and Oliveros held sessions of listening, meditating and performing music together while recording their brainwaves. This interactively changed their performances together as a whole. Chechile also experiments with other bio-feedback monitoring such as tracking heartbeat using an electrocardiogram (EKG) to create sonic meditation sessions.

On the other hand, meditation is never and should never only be static. Buddhist traditions encourage practitioners to meditate through walking, sleeping, eating and doing mundane daily activities (Hanh Reference Hanh1990). From this perspective, Tomie Hahn and Bryan Kip Haaheim’s works specifically explore the relationship between bodily activities and sonic meditation. Particularly, Hahn’s (Reference Hahn2007) work focuses on using sensing technology to track a dancer’s body motions that externalise her sensational knowledge and meditative states through Japanese Zen and culture. On the other hand, Haaheim’s Knowing as Feeling: Creating Meaningful Experience Using the Adaptive Use Musical Instrument (AUMI) (Stewart, Tucker, Williams and Haaheim Reference Stewart, Tucker, Williams and Haaheim2018) takes advantage of the camera on an iPad and use this platform to develop an interactive music system that tracks a performer’s partial body movement and real-time sonifies these body motions as a group meditation practice. In 2019, I participated in his group performance in Kansas City, where six identical AUMI systems were used by our ensemble. One of his pieces requires performers to move their bodies to initiate a sonic event only when a piece of thought is arisen. This embodied sonic meditation method is inspired by the ancient Dzogchen Tibetan Buddhism, which I find very engaging and effective.

To further investigate the potentials offered by embodied sonic meditation that engages body movement and sensing technology, I designed three DMIs and composed a series of interactive pieces using these DMIs. The backbone technologies of this work were developed based on the fifteen proposed design principles.

For the first case study, I simulate the Mandala sand arts with musical expression and aesthetically explore the ancient Buddhist philosophy of impermanence. The Virtual Mandala is composed of 4.5 million particles, directed by arm motions using a physical computing and motion-tracking system called ‘the Expressionist’. The Expressionist (Wu Reference Wu2015) is composed of a two-handed, magnetic, motion-sensing controller and a computer program that processes real-time sonic input and recorded sounds based on the performer’s body movements. The mapping between my hands and the models in the computer graphic simulation is performed using a technique called workspace expansion, which relies on progressively relocating the physical workspace of the devices mapped inside of the virtual environment towards the operator’s area of activity without disturbing my perception of the environment (Conti and Khatib Reference Conti and Khatib2005). This approach allows the performer to interact with high accuracy in a considerably large virtual environment while moving their arms within a comfortable small workspace, ensuring a smooth embodied experience when performing the piece.

Virtual Mandala was dynamically simulated and choreographed with the compositional improvisation for voice and electronic instruments. The piece includes three movements: Construction, Climax and Destruction. At the beginning of the piece, hand/arm motions from the vocalist were directed to initiate the Construction of the virtual Mandala. Mixed sounds of electronic guitar, rhythmic drumming, Buddhist mantra, synthesised sounds and abstract celleto started to create and build up a world from tranquility to intensified passions. During the Climax, musicians were focusing on improvisation while the virtual Mandala presented rotation, twisting and glitter effects. Granular synthesis, along with audio spatialisation that are real-time controlled by arm motions created bursts of variously pitched pieces of sound fractions. In the Destruction, which underscores the eternal nature of impermanence, experimental low-tone throat singing and explosions of violent soprano interjections come to symbolise duality – the extreme conflicts and ecstasy in life, the extraordinary highs and lows, the sharp rises and falls of life as it is experienced, ending with a sharp fade out into the deconstruction of the mandala. At the end, the virtual Mandala was brushed into emptiness, symbolizing the non-dual truth of the universe. Figure 2 shows the improvisation piece’s premiere at Stanford University. I also realised a fixed audio-visual piece based on these live performances, which can be viewed in Movie example 1.

Figure 2.

Live performance of the Virtual Mandala premiere at Stanford Bing Concert Hall.

The second case study, the Tibetan Singing Prayer Wheel (TSPW), was dedicated to my esteemed Buddhist teacher, the venerable Khenpo Sodargye Rinpoche – one of the most influential Tibetan Buddhist scholars. TSPW is a hybrid of a Tibetan spiritual tool called ‘prayer wheel’, a Tibetan singing bowl and a wireless 3-dimensional sensor-based human-computer interface that captures hand–arm motions to enrich musical expressions. TSPW utilises the ‘spinning the wheel’ hand motions to perform virtual Tibetan singing bowls and process human voice in real-time. It combines sounds and gestures, along with voice modulation, physical modelling and granular synthesis, using electronic circuits, sensors and customised software (Wu, Yeh, Michon, Weitzner, Abel and Wright Reference Wu, Yeh, Michon, Weitzner, Abel and Wright2015). The hook of this DMI is the soothing and repetitive circular motion coupled with the natural tactile/inertial force feedback. The calming circular motion hooks both the audience and the user while the enactive sensation hooks the user. Figure 3 shows the two instruments and their shared circular hand movements.

Figure 3.

Left: Spinning a prayer wheel (photo courtesy of Olivier Adam at www.dharmaeye.com). Right: playing a Tibetan singing bowl.

The performer gives the system three inputs: vocals via a microphone, spinning and intuitive gestures from a sensor-augmented prayer wheel, and button presses on a four-button RF transmitter to toggle different combinations of sound processing layers. The speed of the spinning controls the size of the grains of the vocal granulation – faster spinning results in a more intense granular effect and vice versa; the up-and-down movement of the wheel controls the pitch-shift of the vocal processing and the virtual singing bowl – higher hand gestures result in higher pitches and vice versa. There is also a pressure sensor on the wheel to simulate the sound effect of striking a singing bowl. This DMI uses ‘one-to-many’ mapping: using the same circular/spinning gestures to control multiple sound-making layers enriches the vocalist’s musical expression in a simple way; whereas the button controller provides ‘one-to-many’ mapping options and allows the performer to flexibly trigger eight different combinations of sound processing techniques during a performance.

The unit itself weighs less than 0.5 kg (1lb), which made it portable and efficient. Figure 4 demonstrates the inside of this wireless TSPW with all of its electronics fit within the cylinder of about 3.8 cm (1.5 inch) radius and 3.8 cm (1.5 inch) height, leaving room for the rod running through the middle. The hardware and software systems are robustly designed to avoid crashes or physical breakage. The DMI’s behaviours are predictable and it produces the same results every time. These factors create control intimacy and the state of high flow that can lead to a successful embodied sonic meditation session.

Figure 4.

The electronics inside of the wheel: (A) Arduino Pro Mini, (B) Xbee Series 1 2.4GHz RF Module, (C) Flora LSM9DS0 Magnetometer, (D) Li-Ion 400mAh battery pack, (E) charging port, (F) power switch.

Based on my observations from four interactive installations of the TSPW, with around 500 participants, using real-time body movement to manipulate sound helps laypersons, especially children, to experience sound in an intimate, fun and calm way, thus enhancing their embodied sonic meditation experience. Most of the participants seemed to be able to catch subtle changes in the sonic events that TSPW produces with their own gestural control and physical interactions. TSPW was also applied to a telematics concert named ‘Imagining the Universe’, held in honour of Khenpo Sodargye Rinpoche at Stanford University, sponsored by NASA and Stanford. The concert telemetrically connected a meditative live performance between musicians at five research institutions and explored ways of connecting cultures, collaborative artistic partners and the audience over long distances through embodied sonic meditation. An illustration of the performance is presented in Figure 5. A glimpse of the network meditative concert from the location of Stanford University can be viewed in Movie example 2.

Figure 5.

Live performance using TSPW at the NASA-Stanford ‘Imagining the Universe’ network concert.

For the third case study, I designed a DMI named Resonance of the Heart. This DMI uses an infrared sensing device and touchless hand gestures to control a real-time tracking system producing various audiovisual results. I chose optical sensing technology instead of wearable (e.g., gloves) because I aim to preserve the flexibility of the hands/fingers motion as well as the original beauty of the ancient barehanded Mudra meditation. Using a small, touchless sensor ensures that both the user/performer and the audience can focus on the experience and perceive the sound-body as a whole. The system uses supervised learning algorithms and an artificial neural network for tracking accuracy. Two novel electroacoustic vocal processing techniques, which include a Tibetan throat singing filter and a spectral-tilt filter, were implemented to simultaneously process voice input or recorded audio input based on the user’s fingers/hands gestures with a one-to-one mapping strategy (Wu, Smith, Zhou and Wright Reference Wu, Smith, Zhou and Wright2017). I create the hook of this DMI by designing visual feedback using Mudras mapping to both real-time vocal processing and 4-dimensional BuddhabrotFootnote ³ fractal deformations. Selected Buddhist Mudra hand gestures were mapped to seven novel Buddhabrot deformations. Based on the appearance of these newly discovered fractals in the Buddhabrot family, I coin the five terms of ‘Aquarius’, ‘Vajra’, ‘Double Buddha’, ‘Keyūra’ and ‘Shrivatsa’ to describe the Buddhist metaphor accordingly (in Figure 6: from the second left to the right, from the top to the bottom) (Wu and Ren Reference Wu and Ren2019). This abstract yet meditative visual hook enhances both users’ experience and increases audience engagement.

Figure 6.

Seven 4-dimensional Buddhabrot deformations real-time visual feedback.

In 2018, Resonance of the Heart was first used as a pedagogical tool in teaching an upper-division, college-level course at the College of Creative Studies at the University of California Santa Barbara and then has been offered to students at the College of Arts and Media at the University of Colorado. I encouraged students to explore the abstract concepts of meditative sound design using their bodily activities and sensing technology. At the end of the courses, five groups and five individual students showcased their own DMI design and demonstrated their ability to use these DMIs to communicate their sonic selves. The exit survey indicates that fifteen out of eighteen students enjoyed their embodied sonic meditation practice. Moreover, two solo performance pieces at the IEEE VIS and a2ru, ^{Footnote 4} coupled with a group improvisation at Arizona State University, show the capacity of the DMI as an expressive instrument that affords virtuosity. The mapping relationships between my gestures and the musical expressions are transparent both to the audience and to my collaborators, so we can share sonic empathy and a similar state of flow through meditating, observing, listening and doing. It is also worth mentioning that one of the six art installations’ venues of Resonance of the Heart is the National Museum of China, which is the second most visited museum in the world. The piece was exhibited there for a month. Public data shows that more than 3,000 people visited the museum every day (Figure 7). ^{Footnote 5} Through the series of high-trafficked exhibitions, the system demonstrates significant reliability, robustness, easy-to-use and setup, and portability. This solid technology helps participants focus on the current moment and cultivating a first-level entry meditative state of mind. The group improvisation at Arizona State University using Resonance of the Heart can be viewed in Movie example 3.

Figure 7.

A young audience experiencing embodied sonic meditation through Resonance of the Heart.