1. INTRODUCTION
This article summarises several findings in ubiquitous music (ubimus) research that underline the need for new theoretical frameworks to handle the volatile and distributed creativity factors involved in creative music-making (Keller, Lazzarini and Pimenta Reference Keller, Lazzarini and Pimenta2014a). As pointed out by Rudi (Reference Rudi2015), there has been a paradigm shift in technologically based creative practices. Contrasting with the historical electroacoustic music tendencies, the artistic value of recent technology-based music is not necessarily determined by experts (Rudi Reference Rudi2015: 36), blurring the lines ‘between composer, audience, composition, performance and mediation’. Musical activities that take place outside of traditional venues and that feature the audience as an active creative partner demand design techniques that are not currently supported by mainstream musical interaction approaches.Footnote 1 We discuss recent advances in ubimus research, highlighting the limitations of the current theoretical models and arguing for empirically grounded creativity-centred design techniques.
Moving towards an organic methodology for sound art (as idealised in the 1930s by Edgard Varèse), sonic eco-compositional approaches propose creative actions as by-products of action-perception cycles. The eco-compositional perspective rests on the empirical evidence provided by embedded-embodied cognition (Gibson Reference Gibson1979; Varela Reference Varela1992; Hutchins Reference Hutchins1995, Reference Hutchins2010). Rather than abstract manipulation of symbols, composition is thought as interaction among agents and objects (Keller Reference Keller1998, Reference Keller1999). Eco-compositional activity engages in the exploration of local resources as key creative ingredients (Burtner Reference Burtner2005). Similar to the soundscape compositional approach (Truax Reference Truax2002), it incorporates place as a creativity factor highlighting the interaction with the environment as one of the central aspects of the creative process (Rhodes Reference Rhodes1961). The open-ended creative methods adopted by eco-composers have helped to turn musicians (Nance Reference Nance2007) and audience (Keller Reference Keller2000) into active participants of the creative process. This increased reliance on (1) social interaction, (2) everyday settings, and (3) the open-ended, exploratory activities have underscored the limitations of the existing toolset for creativity support.
During the first decade of the twenty-first century, several composers started a search for methods to handle action-perception cycles within their compositional practice. Barrett (Reference Barrett2000), Opie and Brown (Reference Opie and Brown2006) developed data-extraction techniques that provided tools to deal with the complexity of the local sonic resources for musical purposes. Burtner (Reference Burtner2005) explored the usage of synthesis techniques in external settings as sources of creativity, coining the concept of socio-synthesis. Di Scipio (Reference Di Scipio2008) employed a venue’s acoustical properties to increase the available resources for performance-based compositional systems. Nance (Reference Nance2007) proposed the use of aural scores as a way to increase the openness of instrumental works. Cádiz (Reference Cádiz2012) employed ecologically based synthesis methods in the context of orchestral works and Aliel and Fornari (Reference Aliel and Fornari2014) applied these techniques in remote improvisatory performances. Basanta (Reference Basanta2010) and the Capasso+Keller+Tinajero Collective (Capasso, Keller and Tinajero Reference Capasso, Keller and Tinajero1999, Reference Capasso, Keller and Tinajero2000, Reference Capasso, Keller and Tinajero2012, Reference Capasso, Keller and Tinajero2014) made use of ecologically based interaction techniques to enhance the creative participatory potential of their installation artworks. All these initiatives have contributed to the methods presently gathered under the rubric ecologically grounded creative practice.
This article tackles the application of an ecologically based framework to the problem of supporting distributed and socially grounded manifestations of musical creativity. The remaining sections are structured as follows. First, we present two updated models of sonic ecologies. These models provide the conceptual basis to deal with three aspects of sonic creative practices: the material, the cognitive and the social factors. Then we analyse the contributions and limitations of a recent community-constructed definition of ubiquitous music phenomena. This definition is parsed into four components, encompassing the human factors, the material resources, the properties that emerge during the creative activities and the design strategies that foster creative outcomes – including both creative processes and products. The final section addresses the potential impact of the ecological frameworks within ubiquitous music-making on context-based creative practice.
2. MODELLING ECOLOGIES FOR MUSIC-MAKING
The relationships between the material resources and the ecological niche in music-making activities have been addressed in (Keller Reference Keller2012) and (Keller, Lazzaini and Pimenta Reference Lazzarini, Keller, Pimenta and Timoney2014b). From an ecologically grounded perspective, two strategies were identified in five representative musical projects of the late twentieth century: fixed and adaptive usage of material resources. These two strategies emerge from the relationships established between the creative potential of the local settings and the decision-making processes that impact the choice of material resources. These relationships have been modelled through three operations: constrain, expand and shift. Constrain implies the reduction of resources. Expand involves exploratory actions to introduce more resources into the ecological niche. Shift combines the two former operations yielding behavioural adjustments to new conditions. Compositional strategies that apply a strict separation between the decision-making process and the ecological niche do not necessarily depend on the local resources. Therefore, the creative products remain independent from the ecological niche. Multiple examples of this approach are provided both by the acousmatic practices (Wishart Reference Wishart2009) and by traditional soundscape composition (Westerkamp 1989). These sonic ecologies are classified as fixed (Figure 1).
Revised model of the fixed use of material resources in sonic ecologies (the two states reflect pre- and post-intervention status).
When the creative products are shaped through processes that depend on the local resources, the sonic ecologies are labelled adaptive. The mismatch between the eco-niche and the creative procedures seems to sever the link between the composer’s experience of the environment and the listener’s experience. For many years, Barry Truax and other soundscape composers have been working on techniques to reconnect these two separate realms (Truax Reference Truax2002). The interactive installation The Urban Corridor (Keller, Capasso and Wilson Reference Keller, Capasso and Wilson2002; Keller Reference Keller2012; Pimenta et al. Reference Pimenta, Flores, Capasso, Tinajero and Keller2009) suggests an alternative path by creating a setting that enables emergent properties defined by the participants’ behaviours. The material dimension is shaped by the interaction among agents and objects. The computational infrastructure developed for this piece uses accumulation as an organising strategy (Keller et al. Reference Keller, Capasso and Wilson2002). Event densities and timbre profiles depend both on community behaviour and on the compositional system’s activation of visual and sonic events. Thus, the material dimension is driven by the participants’ behaviours which in turn are constrained by the physical layout of the piece. Given that the agents’ behaviours impact the sonic outcome, what we see here is a process of mutual adaptation: the eco-niche determines the material resources of the piece but at the same time the piece also shapes the eco-niche. These processes cannot be separated. Their dynamic can be described as an adaptive sonic ecology: a habitat where agents and objects interact producing a creative sonic by-product that depends on local material and behavioural resources (Figure 2).
Revised model of the adaptive use of material and behavioural resources in sonic ecologies.
Based on the usage of technological tools by an individual composer, Jones, Brown and d’Inverno (Reference Jones, Brown and d’Inverno2012) strip the creative cycle down to two activities: generation and evaluation (henceforth the generation-evaluation model). Following McGraw and Hofstadter’s (Reference McGraw and Hofstadter1993) proposal, generation encompasses gathering materials and the internal processes that lead to new ideas. Evaluation involves processes of selection of the available resources. The generation-evaluation model gives support to the ecologically grounded view on musical creativity as laid out by Keller (Reference Keller2012). The procedural analysis of five representative musical works of the late twentieth century suggests that sonic ecologies involve interactions among human agents and material resources. These interactions can be described by three operations on the material resources: expand, when the pool of resources gets larger (or amplify, in Jones et al. Reference Jones, Brown and d’Inverno2012 nomenclature); constrain, when the pool of resources gets smaller; and shift, when a qualitative change of resources is involved (or impose). Jones and coauthors propose the no operation label, when the resources are unmodified.
The introduction of the nil operator, suggested by the lack of change in the resource pool is consistent with the incubation stage proposed by Wallas (Reference Wallas1926) and later incorporated by most musical creativity models (cf. reviews in Keller et al. Reference Keller, Lazzarini and Pimenta2014b). This is usually described as time spent away from the creative activity. But given the recent findings on the function of the default mental network – that is, neural pathways that stay active despite the lack of conscious processes which are usually linked to the area of the pre-frontal cortex (Beaty et al. Reference Beaty, Benedek, Kaufman and Silvia2015) – it is very likely that activities that have no impact on the material resources, such as sleeping or doing physical activity, provide an opportunity for reconfiguring the cognitive resources devoted to the creative act. Hence, a more accurate depiction of the nil operator is the reduction of explicit or conscious processes, enhancing the role of the implicit cognitive resources.
Despite Jones and his coauthors’ acknowledgement of the distributed nature of the creative act, the generation-evaluation model does not account for phenomena that involve the active participation of distributed agents and resources. There are two aspects to be considered. On the one hand, as multiple studies have shown since the early ecologically grounded proposals (Keller Reference Keller1999, Reference Keller2000; Burtner Reference Burtner2005; Keller and Capasso Reference Keller and Capasso2006), creative musical activities are embedded. That is, they do not depend exclusively on the cognitive resources of the participants. Eco-grounded musical activities feature emergent properties arising from interactions among material resources and behaviourial resources. Behaviours change materials and materials change behaviours, fostering a process of affordance formation. This process impacts the way the resources are employed to support creative activities, yielding dynamic relationships among resources and agents, also called relational properties (Keller et al. Reference Keller, Otero, Lazzarini, Pimenta, Lima, Johann and Costalonga2014c, Reference Keller, Otero, Lazzarini, Pimenta, Lima, Johann and Costalonga2015c; Keller, Otero and Costalonga Reference Keller, Otero and Costalonga2015b).
Complementarily, creative musical activities are social. This means that the material resources are not limited by the experiential knowledge of a single creator. Knowledge is accumulated and distributed among all the agents participating in the creative act, sometimes conforming a community of practice (Wenger Reference Wenger2010). The social aspects of creativity highlight the need to consider the distributed nature of the agents involved in the creative act. The stereotype of the genius-composer working in his studio – isolated from the hustle of mundane distractions – is currently being challenged by ubiquitous musical practices (Keller et al. Reference Keller, Lazzarini and Pimenta2014a). The situated methods that have emerged from ecologically grounded creative practices fit well within the context of community-oriented ubimus research (Lima et al. 2012; Pimenta et al. Reference Pimenta, Keller, Flores, Lima and Lazzarini2014). While the generation-evaluation model was developed keeping in mind a single participant, ecologically grounded practices have traditionally targeted the integration of the audience as an active shaper of the artistic experience (Keller and Capasso Reference Keller and Capasso2006; Basanta Reference Basanta2010; Burtner Reference Burtner2011; Keller et al. Reference Keller, Timoney, Costalonga, Capasso, Tinajero, Lazzarini, Pimenta, Lima and Johann2014d). Thus implying that a revised notion of agency is necessary to account for the socially distributed resources involved in community-based creative phenomena.
So far, we have dealt with two instances of the application of the concept of ecology or meshwork in creative music practice. Fixed sonic ecologies are characterised by a strict separation between the decision-making process and the ecological niche. Therefore, they do not necessarily depend on the local material resources. Adaptive sonic ecologies are habitats where agents and objects interact producing creative sonic by-products that depend on local material and behavioural resources. Recent developments in ecologically grounded creative practices may expand the frontiers of ubiquitous music research. Thus, it might be useful to revisit the definition of ubimus while considering the latest tendencies in ecologically grounded creative practices.
3. PUSHING THE DEFINITION OF UBIQUITOUS MUSIC FORWARD
A community-constructed definition of ubiquitous music proposes that the study of ubimus targets systems of 1) human agents and 2) material resources that 3) afford musical activities through 4) creative support tools (Keller et al. Reference Keller, Lazzarini and Pimenta2014a). Let us deal with each of the four components of this definition separately before we tackle the implications of the ecologically grounded perspective within the context of ubimus research.
3.1. Human agents
Component 1 of the definition refers to the factors that shaped hominid evolution. Two of the currently most influential perspectives on evolution theory are the social brain hypothesis (Shultz and Dunbar Reference Shultz and Dunbar2007) and the niche construction theory (Odling-Smee, Laland and Feldman Reference Odling-Smee, Laland and Feldman2003). The former highlights the importance of social interaction mechanisms for survival. Apparently, the ability to perceive and predict the intentions of others may have been one of the key demands of early hominid interactions that required a high investment of cognitive resources. The latter theory stresses the impact of the organisms on the local environment – leading to the formation of the ecological niches – and the environmental pressures on behaviour that may have demanded increasingly refined cognitive mechanisms to cope with uncertain conditions. Both approaches have gathered strong evidence from diversified fields, including biology, paleontology, anthropology, social psychology and cognitive science. These theories have several implications on creative practice. As suggested by Emmerson in 2001, installation art provides an ideal venue to experiment with the social factors underlying group behaviours (Emmerson Reference Emmerson2001). Interactions encompass visual, sonic and tactile cues produced by the agents’ behaviours and impacting the behavioural resources. Given that ecologically grounded creative practices have focused on social interaction (Keller et al. Reference Keller, Capasso and Wilson2002; Keller and Capasso Reference Keller and Capasso2006; Basanta Reference Basanta2010; Connors Reference Connors2015), it makes sense to think of this type of ubimus systems as behavioural ecologies.
3.2. Material resources
Component 2 of the definition of ubimus systems – material resources – is highlighted by the creative projects that make use of environmental features to constrain the behaviours of the agents or to generate and organise other material resources. Ecological modelling proposes creative heuristics – that is, decision helpers – that are based on the objects’ sonic behaviours in terrestrial settings (Keller Reference Keller1999, Reference Keller2000). These heuristics have been applied in sound synthesis (Keller and Truax Reference Keller and Truax1998; Keller and Berger Reference Keller and Berger2001; Burtner Reference Burtner2005; Di Scipio Reference Di Scipio2008), haptics design (Barrass and Adcock Reference Barrass and Adcock2002; Castle, Adcock and Barrass Reference Castle, Adcock and Barrass2002), interaction design (Keller et al. Reference Keller, Capasso and Wilson2002; Basanta Reference Basanta2010; Keller et al. Reference Keller, Barreiro, Queiroz and Pimenta2010; Keller et al. Reference Keller, Flores, Pimenta, Capasso and Tinajero2011), instrumental composition (Nance Reference Nance2007; O’Callaghan Reference O’Callaghan2013), and environmental data extraction and sonification for artistic purposes (Barrett Reference Barrett2000; Opie and Brown Reference Opie and Brown2006; Gomes et al. Reference Gomes, Pinho, Lopes, Costa, Dias, Tudela and Barbosa2014; Connors Reference Connors2015). All these techniques make use of local resources providing open approaches to sonic organisation. They highlight the emergent qualities of behavioural patterns across modalities, sometimes exploring contradictions and paradoxes through the expansion of the sonic properties of real-world events. Thus, we could classify the ubimus systems that make heavy use of environmental features as multimodal ecologies.
3.3. Relational properties
Both component 1 and component 2 interact to shape component 3 – affordances. The concept of affordance – that is, opportunities for action provided by the features of the environment as perceived by the active agent – as coined by psychologist Gibson (Reference Gibson1977), has a very long tradition of use and misuse. Particularly in the area of human–computer interaction (Norman 1988/Reference Norman2002), it has fostered multiple confusions. Norman’s perspective implies that somehow affordances could be designed and attached to objects or that material properties of objects could be equated to social usage of objects (see Keller et al. Reference Keller, Barreiro, Queiroz and Pimenta2010 for a critical discussion of the implications of the concept of affordance for creative practice). Affordances remain problematic until today. For this reason, we have opted to adopt the more neutral label relational properties for the design qualities that emerge from interaction (Keller et al. Reference Keller, Otero and Costalonga2015b, Reference Keller, Otero, Lazzarini, Pimenta, Lima, Johann and Costalonga2015c). So far, we have identified three types of relational properties: material, social and formal. Material relational properties arise from the interactions with physical objects. Social relational properties emerge from interactions among agents. And formal relational properties include cognitive simulations and conceptual operations handled through offline cognitive resources (Wilson Reference Wilson2002). Hence, formal relational properties are decoupled from agent–object and agent–agent synchronous interactions.
A longitudinal design study was implemented to determine how relational properties can be supported and how they impact the creative decision-making processes (Keller et al. Reference Keller, Timoney, Costalonga, Capasso, Tinajero, Lazzarini, Pimenta, Lima and Johann2014d). This ten-month design study targeted the observation of the creative processes of a video-artist, a sculptor and a composer, while working on the multimedia installation Palafito/Palafita/Home-on-stilts 1.0 (Capasso et al. Reference Capasso, Keller and Tinajero2012) (see Movie example 1, featuring an excerpt of the audiovisual material). Asynchronous, ubiquitous group activities were carried out by the three subjects through lightweight, off-the-shelf infrastructure. Data was extracted from a virtual forum and a file repository. The analysis of the creative exchange indicated cycles of activity alternating between reflection, exploratory action and product-oriented action. Technological support was incorporated through cycles of demand–trial–assessment, embracing a parsimonious approach to the adoption of new tools. Priority was given to repurposing of existing resources as opposed to development from scratch. Creative results included 19:30 minutes of sonic material and video footage, and three 5x8x3 metre raw-wood sculptures.
The procedures underlying the Palafito 1.0 study are summarised in Figure 3. Given the collective character of the endeavour, a common reference system becomes a requirement. Well-grounded local decisions can only be made if the stakeholders have access to the status of the other participants’ activities. Local data, in this case representations of sounds and footage, were shared through volatile resources – in the form of audiovisual (AV) sketches – to increase the flexibility of the exchange (Keller et al. Reference Keller, Timoney, Costalonga, Capasso, Tinajero, Lazzarini, Pimenta, Lima and Johann2014d; Keller, Miletto and Otero Reference Keller, Miletto and Otero2015a). The AV sketches provided a temporal reference system for the asynchronous decision-making process. Overall, the results indicated that social relational properties and material relational properties do not abide by the same mechanisms. The temporal investment on formal relational properties (related to reflective activities) was very high when compared to product-oriented activities. This points to a gap in musical interaction research. While most of the mainstream development has targeted musical instruments and musical performance systems (Wessel and Wright Reference Wessel and Wright2002; Tanaka Reference Tanaka2009), it seems that a better strategy to boost creative potentials involves the support of exploratory and reflective activities.
Asynchronous decision-making support in Palafito 1.0: audiovisual sketches.
Within the context of computationally assisted artistic practices, Bown (Reference Bown2014) discusses the implications of adopting complex tools for creative activities. He conducted a questionnaire-based study to probe creative coders behaviours. Participants were contacted via the coding community’s mailing lists, including users of the visual programming language Max/MSP and of the Processing language forum as well as members of the Computer Arts Society list. Thirty-nine respondents completed the survey. Bown’s analysis yielded four main topics: 1) time-demands of specific activities; 2) sources of knowledge and approaches to problem-solving; 3) approaches to exploration; and 4) information-seeking versus intuitive styles of working. The results of this study suggest that the usage of complex tools was centred around epistemic (exploratory) activities. Generative tools and abstract visualisation tools seem to be at the core of the support of exploratory actions. Given the users’ preference for batch processing, asynchronous support appears to be more accessible than synchronous exploration of data. These findings complement the conclusions of the Palafito 1.0 study: rather than focusing exclusively on performance-oriented tasks, creativity support systems should foster the increase of creative potentials through exploratory activities that target high-level conceptual relationships.
3.4. Ubimus ecosystems
The fourth component of the ubimus definition stems from the incorporation of creativity as a relevant target for technological support (through creativity support tools). Human–computer interaction research in this area was pioneered by Edmonds and Fischer (Edmonds et al. Reference Edmonds, Fischer, Mountford, Nake, Riecken and Spence1995). Creativity support has recently gained impetus through the emerging area of Information Technology Creative Practices (Mitchell, Inouye and Blumenthal Reference Mitchell, Inouye and Blumenthal2003; Shneiderman Reference Shneiderman2007). Shneiderman et al. (Reference Shneiderman, Fischer, Czerwinski, Myers and Resnick2005) have suggested that the development of creativity support tools can provide a new focus to human–computer interaction research. But tools as the central concepts of a research agenda on creativity may be problematic. The investment on isolated tools may prove to be effective when dealing with well-defined problems. This strategy suits the teleological techniques (Fantauzzacoffin and Rogers Reference Fantauzzacoffin and Rogers2013) in which the target of the procedure is defined before the start of the activity. In this scenario, the design and the choice of the tool do not impact the definition of the epistemic target. However, most artistic endeavours not only demand the solution of both undefined and clearly defined problems, but may also challenge established worldviews (Donald Reference Donald2006). For this reason, artistic activity has been described as self-reflective (Hamman Reference Hamman2002; Donald Reference Donald2006). In other words, the objectives of the activity are established during the act rather than a priori. The support of exploratory creative activity may demand more fluid methods than those provided by the teleological approach.
Another factor fostering a conceptual change in the fourth component of the ubimus definition involves the support infrastructure. Technological support for pervasive musical activities increases the difficulty of the design task on several fronts. Ubimus systems may enhance the users’ creative potential by providing access to previously unavailable material and social resources. But a more intensive usage of resources can introduce unintended complexities, narrowing the access to a small user base. Thus, one challenge faced by ubimus designers is to provide intuitive tools for diverse creative tasks. Furthermore, custom-made, special purpose hardware interfaces – such as those proposed by the tangible user interface design approaches (Fitzmaurice et al. Reference Fitzmaurice, Ishii and Buxton1995; Ishii, Mazalek and Lee Reference Ishii, Mazalek and Lee2001) – may fulfil the requirements of transparency and naturalness reducing the cognitive load of complex tasks. But they do not guarantee wide accessibility. In this case, the catch lies in the financial toll. Special purpose systems are difficult to distribute and maintain. As a consequence, the user base is narrowed by the increased costs of the hardware. A promising proposal to tackle this issue involves repurposing – that is, using existing technology for creative ends. Flores et al. (Reference Flores, Pimenta and Keller2010) target personal mobile devices as platforms for ubimus support. Lazzarini et al. (Reference Lazzarini, Yi, Timoney, Keller and Pimenta2012) introduce a sound synthesis platform for mobile devices based on the widely used musical programming language Csound. Despite the enhanced support for mobile devices, the ubimus initiative still faces a challenge related to the sustainability of its technological infra-structure. Software development makes sense when there is stable hardware base available for deployment. The fast renewal cycle of mobile devices makes it difficult to support sustainable musical practices. Hence, rather than targeting devices or instruments, ubimus design may require concepts that can handle permanent changes in the underlying technology.
Lazzarini and coauthors (2014b) propose the iterative implementation of ubimus ecosystems. Ubimus ecosystems function as technological hubs that support the integration of audio and interaction tools. These ecosystems can be reconfigured according to the needs of the users through rapid prototyping techniques. As a case study, Lazzarini et al. (Reference Lazzarini, Costello, Yi and ffitch2014a) report the development PNaCl Csound. PNaCl Csound provides an environment that can be employed for the development of a variety of ubimus applications on standard internet browsers. It is based on a well-understood, domain-specific language that features a wide-ranging variety of unit generators (over 1,500 Csound opcodes). Thus, while it supports the prototyping of reasonably complex audio processing applications, it can also benefit from a large library of code from the various pre-existing catalogues of Csound algorithms. The Csound PNaCl environment features a relatively low-latency performance and incorporates the know-how developed over thirty years of Csound usage, providing a path for the development of ubiquitous music ecosystems based on standard web browser technology. On the down side, support for the deployment of applications is not universal, as it is the case in all browser-based software development systems. This avenue of research emphasises the interrelated character of ubiquitous musical interactions featuring support both for audio processing and for rapid user interface prototyping. Hence, it may provide a methodological alternative to the tools designed for musical performance and based on simplistic acoustic-instrumental concepts.
An important dimension of the emerging ubimus ecosystems has flourished under the Maker Movement (Dougherty Reference Dougherty2012), the do-it-yourself (DIY) interventions. Much of the context under which ubimus has been considered, as far as the supporting hardware is concerned, is based on the use of existing manufactured technology (off-the-shelf devices), sometimes repurposed through software or hardware modifications for application-specific needs. This can be seen in a number of projects reported in the ubimus literature (Flores et al. Reference Flores, Pimenta and Keller2010; McGlynn, Lazzarini and Delap Reference McGlynn, Lazzarini and Delap2012; van Troyer Reference van Troyer2014). By bringing the dimension of DIY electronics platforms into the ubimus playground, designers are able to tap into a much wider source of technology solutions. In fact, many of the initial ideas that motivated ubicomp (Weiser Reference Weiser1995) become more tangible, as custom-made devices seamlessly fuse with our surroundings.
As we integrate multiple objects into the ubimus ecosystems, a key potential of embedded technologies becomes evident: we can take advantage of the internet infrastructure to offer connectivity over a wide area. Small computing units can be remotely controlled, to gather data and to interact with people and other objects in the environment. This brings the possibility of functional extensions through ubimus devices. Combinations of customised hardware and ready-made components can cooperate in systems where there is effectively no separation of computing devices and peripherals. Kevin Ashton has coined the term the Internet of Things (IoT) to describe this trend (von Kranenburg and Bassi Reference van Kranenburg and Bassi2012). ‘IoT will be composed of trillions of devices. While it is unlikely that all devices will be connected in a mesh [...] the number of interconnected object will outnumber by several orders of magnitude the current internet’ (von Kranenburg and Bassi Reference van Kranenburg and Bassi2012: 4). This massive, heterogeneous infrastructure furnishes new opportunities for creative action while it introduces several challenges for design. Increased energy usage (von Kranenburg and Bassi Reference van Kranenburg and Bassi2012) and potential increase of waste point to zero-entropy (or self-sufficient) devices act as design targets. Shared communication protocols will be required for full integration. Another issue that has to be addressed is a user-centred approach to privacy. Given the pervasive presence of the IoT, transparent privacy mechanisms will have to be supported on computationally limited devices.
Currently, with the appearance of small connectable units, such as the Arduino, Raspberry Pi, Galileo and Edison, we can build custom hardware systems based on components that interact with an unspecified number of other computational elements. The musical counterpart of the IoT points to the emergence of an Internet of Musical Things (IoMT), where a number of new possibilities of interaction with music-making devices may open up for both musicians and audiences. Lazzarini, Timoney and Byrne (Reference Lazzarini, Timoney and Byrne2015) demonstrate this potential by creating a fully featured system for musical signal processing on an embedded platform (the Intel Galileo). Although a prototype proof-of-concept is provided in terms of a conventional MIDI instrument, the authors describe the development of an open prototyping platform, whose applications target the Internet of Things (strictly speaking this is effectively an instance of the IoMT). As this ubimus ecosystem is based on both hardware and software interventions, creativity becomes a feature of the system design, its use and the emergent properties that arise from the multiple forms of interaction afforded by its components. Hence, previously perceived boundaries between design, artistic concept, technical realisation and deployment are dissolved into a continuum of creative activities.
A possible caveat of the IoMT infrastructure is the increased complexity of the systems, reducing the ability of the stakeholders to interact in meaningful ways. Miniaturisation of devices implies reduction or disappearance of surfaces for visual display or tactile interaction. This problem is already present in the design for mobile personal devices (Flores, Pimenta and Keller Reference Flores, Pimenta and Keller2014). Ubimus research points to the development of interaction patterns (Pimenta et al. Reference Pimenta, Keller, Flores, Lima and Lazzarini2014) and creativity support metaphors (Bernardo, Pestana and Martins Reference Bernardo, Pestana and Martins2015; Keller et al. Reference Keller, Flores, Pimenta, Capasso and Tinajero2011), striving to achieve ‘design once, deploy many times’ strategies (Lazzarini et al. Reference Lazzarini, Keller, Pimenta and Timoney2014b). Lack of integration among components is particularly problematic when dealing with time-sensitive activities, such as music-making. Given the heterogeneous profile of the IoMT, general-purpose protocols may not be sufficient for synchronous activities. The deployment of the Musical Instrument Digital Interface protocol – which predates the internet – could serve as an initial path for integration. But hybrid approaches, involving both tight synchronisation and loose aggregation, may be better suited to the dynamically changing demands of ubimus activities. Another problem is the anonymity of the sonic resources available through online sound databases. As pointed out by Truax (Reference Truax2015: 108–9), the lack of contextualisation of digitised sounds paves the way to their use as isolated objects, rather than as situated events. Despite the new opportunities furnished by the IoMT, the distributed nature of its resources may force the reduction of explicit control of its components. Hence, music-making as a hierarchical, centralised activity may not be feasible in some forms of IoMT. Community-based and shared decision making may be more amenable strategies for creativity activities based on the IoMT infra-structure.
We can also glimpse some new perspectives for ecologically grounded creative practices, providing a wider scope for a multimodal integration of various artistic formats, as well as for the delivery of artistically motivated educational projects. As an example of the artistic applications of the IoMT, the Memory Tree project (Ribeiro Netto et al. Reference Ribeiro Netto, Castheloge, Oliosi, Mateus, Costalonga and Coura2015) uses very simple recording and playback devices installed at a remote physical location, accessible through the internet. In this multimodal installation, users were able to record short audio extracts via a social-network environment. These snippets would be left for others to listen to at a tree holding playback devices. The technology probed the ranges from users’ mobile phones and used an existing internet service infrastructure (for the production and content uploading), featuring custom-made, DIY hardware (for the playback system itself). In such a IoMT scenario, location and environment are directly employed in and designed for the ubimus ecosystem. IoMT functional extensions can increase the geographical and the social significance of ubimus activities, fostering community engagement beyond co-located support. They also provide means of spreading the computational load of ubimus interventions over a heterogeneous collection of units. It is also important to point out that IoMT applications do not necessarily require the development of dedicated networking infrastructures, since ubimus systems can hitch a ride on the extant functionality of the internet. As observed in projects such as the Memory Tree (Ribeiro Netto et al. Reference Ribeiro Netto, Castheloge, Oliosi, Mateus, Costalonga and Coura2015), constructs pertaining to ecologically grounded creative phases, namely potential, deployment and outcome, can be explored deeply, expanding the use of cognitive, material and social resources, thus fostering both local and remote forms of social interaction without disrupting the ties to the ecological niche.
While revisiting the definition of ubiquitous music, we have reconsidered four components of the ubimus systems. Component 1, linked to their human-related aspects, finds theoretical scaffolding on the emerging views on evolutionary theory. When the system relies on social interaction, it can be construed as a behavioural ecology. Ubimus projects that make intensive use of environmental features to modulate the behaviours of the agents or to generate and organise the material resources emphasise the role of component 2 of the ubimus definition – the material resources. Given their heavy usage of environmental features and the exploration of the emergent qualities of behavioural patterns across modalities, these systems can be described as multimodal ecologies. The third component of the definition relies on interactions between components 1 and 2, encompassing relational properties. Material relational properties arise from activities with physical objects. Social relational properties emerge from exchanges among the agents. And formal relational properties are featured in cognitive simulations and conceptual operations that employ offline cognitive resources – that is, resources that are decoupled from the activity (Wilson Reference Wilson2002; Keller et al. Reference Keller, Otero, Lazzarini, Pimenta, Lima, Johann and Costalonga2014c). Broadly speaking, these formal operations engage what Evans and Stanovich (Reference Evans and Stanovich2013: 230) call the algorithmic mind. Thus, algorithmic meshworks can be defined as forms of material and cognitive arrangements that act on formal relational properties. While social and material relational properties are aligned with the well-established concepts of social and natural affordances (Keller et al. Reference Keller, Barreiro, Queiroz and Pimenta2010), formal relational properties have not yet been thoroughly explored from ecologically grounded creative perspectives. Aside from the hybrid forms of processing inspired on Von Foerster’s (Reference Von Foerster2003) concept of computing (Di Scipio Reference Di Scipio2014), these relational properties may play a significant role in both the planning and the conceptual stages of the ubimus design cycle.
Given the previous discussion, the fourth component of the ubimus definition – creativity support tools – does not seem to address the needs of ecologically grounded creative activity. Isolated tools – even when designed for creative action – may not be suited to the open, iterative and multi-tiered methods that are emerging in eco-based creative practices. A desirable quality of ubimus systems is self-reflectivity. Both Bown’s and the Palafito 1.0 study indicate that creative support systems should not only provide the means to achieve pre-defined objectives, they should also foster the exploration of new epistemic targets. Rather than dealing with well-defined problems, ubimus activities may involve open-ended handling of both material and social relational properties. A concept that may fulfil the requirements of the fourth component of the definition of ubiquitous music is the ubimus ecosystem. Handling material resources through ubimus ecosystems may open opportunities for the application of exploratory design strategies.
4. IMPLICATIONS OF THE ECOLOGICALLY GROUNDED UBIMUS FRAMEWORKS FOR CONTEXT-BASED COMPOSITION
Let us summarise what we have learnt so far. Ubimus ecologies provide a promising context for the expansion of ecologically grounded creative practice. These ecologies can be characterised as fixed or adaptive depending on the relationships between the local resources and the ecological niche. Relational properties emerge from interactions among material and social resources. Ubimus systems that make heavy usage of social interactions are labelled behavioural ecologies and systems scaffolded on environmental features are described as multimodal ecologies. The use of decoupled resources characterises algorithmic meshworks which yield formal relational properties. Theoretical and experimental treatment of this latter form of creativity support still needs to be tackled by the eco-compositional literature. Furthermore, the fourth component of the ubimus ecologies – initially labelled creativity support tools – needs to be expanded to handle the distributed and volatile qualities of ubimus socio-technological resources. Ubimus ecosystems may provide an alternative avenue for exploratory design strategies.
One of the objectives of this article is to explore the implications of ecologically grounded practices within the context of ubiquitous music-making. After having presented several examples of ubimus projects, we are in a better position to discuss the implications of the eco-based framework for context-based creative endeavours. Truax (Reference Truax2015: 109) defines context-based composition as ‘a powerful artistically and socially motivated means of re-engaging both the composer and the listener with real-world contexts’. The sonic outcomes of this practice are not just channelled through concert-style venues, they are also featured on radio, on the web, and on wearable computing devices. These location-oriented technologies are the focus of ubimus research. Truax suggests that musical practices enabled by ubiquitous technology may establish a trend to be followed over the next decade. Furthermore, he observes that ‘today’s job market […] seems to require everyone to carve out an innovative path, rather than fit into a standard pre-defined mould’ (Truax Reference Truax2015: 109). Hence, the proposal of designing for creativity – as stated in a recent ubimus manifesto (Pimenta et al. Reference Pimenta, Keller, Flores, Lima and Lazzarini2014) – seems to be aligned with the needs of context-based composition.
Despite contributing with the introduction of the place factor within creative musical practice, soundscape literature has not yet addressed the distributed nature of creative activity. We detected a similar shortcoming in one component of the definition of ubiquitous music. Creativity support tools, as proposed within the context of human–computer interaction research (Shneiderman Reference Shneiderman2007), do not seem to take into account the interrelated properties of material and behavioural ubimus resources. The proposed revised definition (mentioned at the beginning of section 3) incorporates creative ecologies as key elements of ubimus practice, featuring three layers: behavioural ecologies, multimodal ecologies and algorithmic meshworks. Algorithmic meshworks are forms of material and cognitive arrangements that act on formal relational properties. These meshworks may be relevant for the planning and conceptual activities involved in the early stages of creative practice. Multimodal ecologies target activities scaffolded on environmental features, highlighting the interactions among agents and material resources. Finally, behaviourial ecologies are based on social interactions targeting outcomes that shape new human behaviours.
The view of musical creativity as an individual endeavour carried out inside the composer’s head and involving the abstract manipulations of symbols may fit the demands of concert-oriented formats but does not fare well for everyday settings. The artistic projects discussed in this article challenge the anthropocentric and instrumentally biased perspectives on creative practice pointing to the need of more fluid creativity support strategies. Distributed forms of creativity such as those fostered by ecologically grounded creative practices imply sharing the responsibility of aesthetic decision making. As articulated in Keller et al.’s (Reference Keller, Lazzarini and Pimenta2014b) ubimus application of Glăveanu’s (Reference Glăveanu2013) general creativity model, both audience and agents become active participants of the creative process. Hence, the concept of musical performance – as a display of previously established choices – gives way to the deployment of sonic resources. On the one hand, the musical results do not depend purely on the compositional choices. On the other hand, distributed creativity engages the usage of local material resources and of socially shared knowledge, both fostering and limiting creative outcomes. This form of distributed agency has been labelled adaptive sonic ecology (Keller Reference Keller2012). Adaptive sonic ecologies occur in habitats where agents and objects interact, yielding creative by-products that make intensive usage of local material and behavioural resources. In contrast, fixed sonic ecologies enforce a strict separation of the decision-making processes and the ecological niche. This means that decisions do not necessarily depend on the local material resources. These two models provide useful guidelines of ubimus designs for context-based music-making. Support for adaptive sonic ecologies relies on volatile resources, thus it needs to ensure graceful degradation of system performance when the resources are not accessible (e.g. because of connectivity breakdown or because of stakeholders with limited musical expertise). Fixed sonic ecologies provide more robust support but the toll is reduced flexibility. The link between the resources and the eco-niche is severed, yielding sonic materials that can be freely manipulated but which are not necessarily ecologically valid (see the previous discussion of usage of online sonic databases).
Algorithmic meshworks emerge as interesting tools to deal with formal relational properties. Two studies reviewed in this article (Bown Reference Bown2014; Keller et al. Reference Keller, Timoney, Costalonga, Capasso, Tinajero, Lazzarini, Pimenta, Lima and Johann2014d) highlight the key role of exploratory (also called epistemic) activities in creativity-centred design. While Bown focused on the use of complex tools, Keller and coauthors adopted an opportunistic, ecologically light-footprint approach. Cognitive simulations and conceptual operations that are enabled by manipulations of material resources can also be defined ‘as self-assembling sets of interrelationships in which information is directly inscribed’ (see Introduction). The results of both studies indicate that creative activities demand a heavy usage of conceptual operations through asynchronous support. What contexts demand light-weight tools or complex generative strategies is still an open question. Design choices have several implications for context-based composition. Adaptive sonic ecologies are open to the influence of the habitat, so they usually demand synchronous support. And they are tightly constrained by the stakeholders’ cognitive resources. Fixed sonic ecologies are not bound by temporal constraints, hence they foster complex sonic manipulations. In this case, the barrier lies in the knowledge transfer mechanisms. When the decision-making process is distributed across time and space, the local cues are lost. These cues can be partially shared through creative surrogates (Keller et al. Reference Keller, Miletto and Otero2015a) as exemplified by the usage of audiovisual sketches (see section 3.3). But how much and what type of knowledge is necessary to ground informed aesthetic decision making is still unknown.
The footprint of context-based compositions can be significantly extended by exploring the creative potential of the IoMT. The IoMT can be understood both as a geographical and as a social design tool. With the embedding of musical devices in a variety of locations, the modes of interaction between the audience/public and the sonic resources are multiplied. While sharing local resources over a wide area, stakeholders can experience an expanded, enveloping environment. Actions and events in one location can have long-ranging effects over a number of interlinked resources. It is true that interactions between external, remote agents have already been used in web-based musical works. However, IoMT infrastructure based on minimal, customised, embedded and connected devices enhances the flexibility of creativity support systems to explore both the sonic and the social potentials of distributed ecosystems. As observed in IoT scenarios, design decisions occur both at local and at distributed levels, demanding a balance between bottom-up and top-down strategies.
The emerging trend of widely distributed and partially connected systems of heterogeneous devices points to increased access to volatile material resources, demanding a type of expertise that is not supported by current musical training (Brown Reference Brown2015). As we have seen, ubimus ecosystems function as technological hubs that support the integration of audio and interaction tools. These ecosystems can be reconfigured to fit the creative needs of stakeholders through rapid prototyping techniques. Future context-based compositional practices will probably demand stronger and more flexible support for distributed decision making. Given this set of requirements, ubimus research may furnish viable solutions to the challenges raised by context-based creative practices.
Supplementary material
To view supplementary material for this article, please visit https://doi.org/10.1017/S1355771816000340
