Introduction
Virtual reality and virtual acoustics (VR/VA) are powerful tools for the reconstruction of vanished buildings and the study of their characteristics and have also become important research and teaching tools. Digital models have proved very effective for studying construction and structure, as well as the thermal and acoustic behavior of such buildings. It is even possible to recreate virtual tours through the sensory experiences transmitted by the original spaces.
A restoration, if only virtual, reinstates part of the loss, and enables us to enjoy the spatial, visual, and acoustic sensations produced by the missing buildings. A reconstruction requires a historical, archaeological, and typological study of the buildings, indicating the geometry and its functional characteristics, as well as the location of each of the participants and social action inside the building.
Recent advances in virtual acoustics allow to recover the ancient sound in archaeological areas, obtaining realistic reconstructions of ancient buildings or recreate previous stages of actual ones. This way, users can be transferred to free moving space inside or around buildings that were lost or modified over time.
The contributions and innovative aspects of this work are related to the modularity of acoustic rendering elements applied to the VA reconstruction of ancient and lost buildings. This modality of VR allows the user to enjoy a sensory experience, not just seeing how the building was in different ages, but also “tasting” the sound in the environment. In particular, an application to an ancient and lost building devoted to the theatrical performance in Valencia (Spain) is analyzed in this work. Besides, a head mounted display (HMD) has been developed to acoustically render the model in an interactive manner, by using the relative position of the head as an input to select the first audio filter of the chain, based on head-related transfer functions (HRTFs).
This paper is structured as follows. Related works are reviewed in the section “Related work”, the section “Methodology & developments” describes the methodology and developments within the project-oriented to interaction. In that section, the development of the graphical and acoustical model of an ancient theatre in Valencia, and of the HMD with positioning sensors and headphones is explained. Then, the integration of the models for acoustical and graphical rendering in a common modular engine is explained. The section “Results” provides the results of a subjective study to evaluate the proper functioning and performance of the presented system. Finally, the section “Conclusions and future work” concludes the paper and provides some ideas for future work.
Related work
We can find many approaches that have been used for the acoustic reconstruction of buildings and events as examples of integration of acoustic environments. In Fazenda and Drumm (Reference Fazenda and Drumm2011), the authors study the acoustics of Stonehenge. In Collecchia et al. (Reference Collecchia, Kolar and Abel2012), the authors make a computational model of Chavín de Huántar in Peru to study an acoustical problem in the building. In Lubman (Reference Lubman2008), the author analyzes acoustic echoes at the Mayan temple of Kukulkan. In Azevedo et al. (Reference Azevedo, Markham and Wall2013), the authors developed a system for acoustically rendering the sermon for Gunpowder Day given by John Donne, on the 5 November 1622, in St. Paul's Churchyard outside St. Paul's Cathedral in London. Also, the rehabilitation of the sound of ancient objects and instruments is a matter of use. In Avanzini et al. (Reference Avanzini, Canazza, De Poli, Fantozzi, Pretto, Rodà, Angelini, Bettineschi, Deotto, Faresin, Menegazzi, Molin, Salemi and Zanovello2015), the virtual restoration of an ancient pan flute, probably of Greek origin in the Greek-Egyptian period, consisting of 14 reeds of different lengths held together by ropes and a natural binder, and originally coated with a resin layer (now partially missing) is done by means of a 3D scanning of the remains of an archaeological items recovered during a re-assessment of the Museum of Archaeological Sciences and Art in Padova. Other VR applications with haptics for historical devices in museums are analyzed in Gonizzi et al. (Reference Gonizzi, Caruso, Micoli, Covarrubias Rodriguez and Guidi2015).
A VA analysis of Seville Cathedral was performed in Alonso et al. (Reference Alonso, Sendra, Suárez and Zamarreño2014). Other European initiatives, such as the ERATO project, succeeded in recovering the famed acoustics of the ancient Greek and Roman theatres and odeia around Europe and the Eastern Orient (Rindel, Reference Rindel2011). In Berardi et al. (Reference Berardi, Iannace and Maffei2016), the authors develop a VA reconstruction of the Odeon of Pompeii and compare the results with other Roman odeia developed in the ERATO project. The CAHRISMA project studied various mosques and Byzantine churches to render Islamic and Byzantine religious chants (Micallef and Sammut, Reference Micallef and Sammut2002). In all these examples, auralizations helped to obtain a deeper understanding of the spaces being studied than that which could be obtained only through visual or numeric inspection of impulse responses, or other measured data (e.g. the evaluation of acoustic parameters). In addition, an example of the analysis of a lost church, including the integration of the graphical model and the acoustic model as a VA model, is presented in Sender et al. (Reference Sender, Planells, Perelló, Segura and Giménez2018).
Methodology & developments
The methodology for this project is based on the integration of a developed HMD, which combines a graphical and 3D audio engine to render a theatrical model, which has been also historically studied and specifically reconstructed.
Modeling the theatrical virtual environment
First theatrical performances in València (Spain) were made in an itinerant way, in outdoor spaces or improvized locations without a specific acoustical conditioning. The first building specifically designed to accommodate these performances was promoted by the General Hospital of València, an institution which got the monopoly on theatrical activities in the city to fund some expenditures of the hospital related to charity. It was built in 1584. The documentation about this original playhouse is very scarce, but the location is known. The old “Casa de la Olivera” was in operation for 34 years, until the Hospital decided to tear it down and build a new one, larger, and better adjusted to the functional requirements of the time. For this purpose, some adjoining properties were acquired and the new “Casa de la Olivera” was built (Juliá, Reference Juliá1950).
The new theatre was rebuilt in the same place in 1618. The building had a substantially circular geometry (nearly octagonal). The stalls had a width of 20.66 meters and a depth of 19 meters, allowing a maximum capacity of 1800 spectators, but only occasionally. It had a ground floor and two heights, and it was covered. The existing documentation includes detailed studies on the capacity (Mouyen, Reference Mouyen1991a) and a schematic plan of its layout in 1678. The ground floor had chairs where the nobility, para-nobility classes, and clergy were located, some benches behind these chairs and a few terraces in the background and the sides. The first floor was divided into 20 rooms that were the most expensive and luxurious seats of the theatre. Finally, the second floor, known as the “aposento de les dones” (“cazuela” as it was known in the Spanish “corrales” or playhouses) was reserved exclusively for women. Each of the zones had its own independent access. Figure 1 shows the distribution of chairs on the ground floor of the theatre. The stage was formed by a single rectangular platform, surrounded by the public on three sides. Behind, it was the costumes of the actors and over it, a balcony, which was accessed by a staircase at the hidden back of the stage, called “balcony of the apparitions” and used for performances on an upper level.
Schematic plan from 1678 of the “Casa de Comedias” in València (Source: General and Photographic Archive of the Diputación de València).
This “Casa de Comedias” remained in use until 1715 when, due to its poor condition, it was decided to carry out a significant remodeling that was maintained until 1748, when an earthquake caused serious damage to the building. At that time the archbishop Andrés Mayoral Alonso de Meya also forbade any kind of theatrical performance, so this caused the demolition of the theatre.
The graphical model was built from four basic studies: one by Sirera (Reference Sirera1980), other by Ros Andreu (Reference Ros Andreu1981), other by Mouyen (Reference Mouyen1981, Reference Mouyen1991a, Reference Mouyen1991b), and another one by Oleza (Reference Oleza2017), in combination with some old documents containing instructions to the builders. The study by Ros Andreu included the plan of three floors, two sections, and one axonometric. Figure 2 shows all these plans by Ros Andreu. This architectural study was contrasted by Prof. Joan Oleza with the original management documentation of the builders, conserved in the General Archive of the Diputació de València, and with the historical evolution of the urban topography at the zone. Then, the structure, the dimensions, and the architectural plans were reformulated. Following the indications of Prof. Joan Oleza, architect Ana Planells drew the new plans and Sebastià Mirasol build the 3D model from these plans using Sketchup (Tal, Reference Tal2009). Figure 3 shows two views of the model (also demo videos can be watched via the links in the footnoteFootnote 1).
Plans of the Olivera theatre by Ros Andreu (Reference Ros Andreu1981): (a) Ground floor, (b) first floor, (c) second floor, (d) façade, (e) section from the façade, and (f) axonometric section towards the stage.
Graphical Model of the Olivera: (a) view from the center of the stalls, (b) view from a lateral room in the second floor (“aposento de les dones”).
The acoustical model was obtained from a simplification of this graphical model and taking the acoustic characteristics of the materials in the original documentation of the builders. As documented, the six large pillars that supported the roof and the other smaller four that supported the “ochavo” (the octagonal galleries at the front side of the stage) were made of “Godella” stone, but in the first and second floors the pillars material changed to bricks covered with plaster. The walls were made of brick covered with plaster. The beams were made of wood and scaffold. The theatre had ceramic tiles flooring on all floors and iron railings on the upper floors. The grandstands where the spectators stayed were made of wood. The floor of the theatre was made of ceramic tiles and there were iron railings on the upper floorsFootnote 2.
In order to perform an acoustical study of the theatre, the acoustical properties of these materials were selected from databases, as the original materials were lost. The simulation was performed using ODEON software (Christensen and Koutsouris, Reference Christensen and Koutsouris2015) in order to obtain the impulse responses to perform the auralization of the sources within the virtual environment (Vorländer, Reference Vorländer2008).
Dry recordings and auralizations of different theatrical excerpts
Since this building was made in the 17th century of the Spanish literature, some scenes from various works by Lope de Vega have been selected to be auralized. These recordings were made in the dry room of the Signal Processing and Audio Technology group (SPAT) laboratory of the Escola Tècnica Superior d'Enginyeries of the Universitat de València, with various professional actors of classical theatres, such as Reyes Ruiz, Enric Benavent, and Toni Misò. For this purpose, a ZOOM H6 digital recorder with two microphones in XY format at 90° and two additional omnidirectional microphones were used to make the dry recordings for the auralizations.
From the acoustic simulations, the impulsive responses (IRs) are obtained in different positions. These locations were selected as a reference on each floor. These IRs are separated into early and late reverberation, taking their Early Decay Time (EDT) as the reference measure for this room. These pre-processed signals will later be used to filter the anechoic sound of the selected theatrical extracts.
Interaction development
In order to allow the graphical and acoustical rendering in an interactive manner, using HRTFs selection with the relative head position as an input and without relying on any commercial solution, a HMD has been developed.
The audio part is composed by SONY WH-CH500 headphones. The HMD graphical part is composed by: a 5.6″ LCD screen, a screen controller, two aspherical lenses, a MPU-9025 MEMS sensor (accelerometer, gyroscope, and magnetometer) (MPU, 2018), and a microcontroller. As the final application of the HMD is the visualization of realistic architectural models, the screen needs to have a high resolution and pixel density to allow the user a sharp and clear image. The selected screen has a resolution of 1280 × 800 pixels (with 16:10 ratio) and a density of 270 pixels per square inch. Each eye perceives a resolution of 640 × 800 pixels (with 4:5 ratio). In order to produce a depth effect, the images must be rendered with barrel distortion (see Fig. 4). This effect is similar to the one performed by the fish eye effect that applies a spherical deformation, allowing an increase of the angle of vision. After the image deformation, the aspherical lenses create a bearing effect that corrects the resulting spherical aberration in the image. In addition to the deformation of the image, the lenses have a magnification of 5×, so that it occupies most of the field of view.
Prototype screen for the HMD with barrel distortion.
Movement detection for HRTF selection
The motion detection consists of a microcontroller and a sensor. The objective of this component is to achieve a high sampling frequency with a low latency. This is necessary to provide a smooth motion sensation to the users, preventing from dizziness.
A diagram of the operation of the HMD can be seen in Figure 5. It shows the communication and interactions between components. The end of the development is marked by the production of a compact board containing the microcontroller, the sensor, the entire conditioning circuitry, and a USB port for sending serial data to Unity 2018 to be processed by two scripts connected to the audio engine. This connection is done by adding an emitter object in the Unity scene, disabling the audio source and specifying the script Studio Event Emitter. Also, in the receiver part, a First Person Controller is needed, then it is necessary to disable the audio source and specifying the script Studio Listener. Both scripts are provided by FMOD Studio (this process will be further explained later).
Operation diagram of HMD with sensors (MPU & MCU).
The prototype of the movement detection system has been implemented by using an Arduino microcontroller. We have used for this prototype an Arduino Nano microcontroller, a MPU-9025 sensor (gyroscope, magnetometer, and accelerometer), connection cables and a prototype board. In order to test the functioning of this sensor, a library for communicating with MPU-9025 was used. In addition, a Python program for data visualization by DebraFootnote 3 was used to check the data sent by the sensor. Figure 6 shows the positioning system with data visualization of the relative position.
Prototype board with Arduino Nano and MPU sensor for data visualization.
Since the sensor is very sensitive to small movements, it is necessary to apply an extra smoothing filter to reduce excessive movements. The problem with applying this intermediate stage to a microcontroller such as the Atmel 328P (ATMega328p, 2018), which uses the Arduino Nano, is that it slows down the frequency with which it is sampled, producing the opposite effect and causing a feeling of dizziness.
To reduce this inconvenience, it was decided to replace Arduino by a PSoC 3 (Programmable System on Chip) from Cypress Semiconductor (PSoC, 2018). Figure 7 shows a picture of the MPU mounted on a PSOC3 board. This environment is much more versatile, as it provides the option to reconfigure the hardware and make use only of what is necessary. This allows to optimize its use and get more frequent sensor readings. To do this, it has been necessary to create ad-hoc libraries to be able to operate the sensor correctly. After obtaining the simplest sensor readings through Serial Peripheral Interface communication (SPI), it has been possible to adjust the position of the device, which was mounted on the HMD.
MPU connected to PSOC3 by Cypress Semiconductor.
From the coordinates obtained at the exit of the sensor position, a mapping of the HRTFs, obtained from the CIPIC database (Algazi et al., Reference Algazi, Duda, Thompson and Avendano2001) with a KEMAR torso (a specific and standardized head and torso for binaural audio recording), has been carried out, and with this the first stage of binaural listening in the virtual environment can be configured.
Graphical and audio engines integration
The current version of Unity 2018 offers an Audio Spatializer Software Development Kit (SDK), but even though it has come a long way since the first time it was implemented, it is still quite limited in comparison with other available alternatives. FMOD (Lanham, Reference Lanham2017) is one of the alternatives, and it is frequently used together with Unity to achieve what Unity's Audio Spatializer SDK does not cover.
FMOD Studio is an audio content creator tool developed by Firelight Technologies for audio professionals and game developers. It provides a programming API compatible with most of the popular graphic engines, such as the one used in this paper or Unreal Engine. The FMOD tool provides the developer with an easy to use graphical interface that makes 3D audio design as easy as a drag and drop system of the desired sound. FMOD integrates several blocks for 3D spatialization (FMODStudio, 2018) which are very useful for creating sound in a virtual environment (see Fig. 8).
FMOD Studio interface.
First of all, we applied a 3D-panner that separates each channel of a stereo sound in a right and left mono track with a HRTF filter to render binaural both channels using the information coming from the sensors. This fact lets the use in real time with different pans and attenuation distances. This is what allows the final user hearing a 3D sound in relation with the distance and direction relative to the sound source.
As already mentioned, the interior of the building was acoustically modeled. This provides us with impulse responses of different interior parts. These responses could be applied directly to a convolutional reverb filter, which convolutes the sound with the impulse response on the fly. This makes any kind of noise sound as if it were physically in the point where the impulse response was taken.
All the changes made to the sounds are saved on FMOD's Master Bank. This is quite useful since in the future the user can access any of the sounds independently with all the modifications applied.
FMOD integration with Unity is quite simple. The tool is already available for the general public a plug-in for Unity that connects the master bank of any FMOD project to the graphics engine. The only adjustments needed is to make sure the FPSController has Unity's Audio Source disabled, and FMOD's Studio Listener script loaded as a component. Audio sources must do the same, but instead loading FMOD's Studio Event Emitter script loaded as a component. Then, the user can select in Unity the sounds included in the Master Bank as a source with the same filter chain applied in FMOD (see Fig. 9).
FMOD and Unity integration.
Results
Evaluation methodology
The assessment of the model and the interactive system for audio-visual rendering was done by designing and testing a short questionnaire using 5-point Likert rating scales (from 1 to 5, ranging from very negative/bad to very positive/well) in order to gather feedback (opinions, feelings) from the end-users testing the system. The test is based on a sound-walk within the model of the “Corral de Comedias de L'Olivera” by using the developed system. After the virtual experience, the participants had to fill in a questionnaire. It consisted of some questions about social information, three questions about the quality of different aspects in the model and finally, three questions about the place where the sound source was perceived (i.e. in the scene, in the first floor, and in the second floor). Also, a field with comments, where the respondents recorded their feelings about the whole test.
One of the objectives of the test was to determine the perceived media quality in the virtual environment. Another key objective was to determine whether the users navigating inside the building are able to identify where the sound sources are located (or come from).
With this test, we can demonstrate whether the spatialization of sounds works and whether simulated impulsive responses responds to the expectations of the users for this theatre.
Subjective assessment study
Table 1 shows the form used for the testers and Table 2 shows the answers from 20 users (40% men–60% women). The average age of all the users was 43.5 years. The test conditions were done in a dry environment to avoid room effects and using headphones, together with the HMD, in order to better appreciate the HRTFs performance.
Form used for the information collection
Answer of the users
On the whole, the responses from users have been very positive in terms of the perceived quality of the model, the ambience, and the sound, obtaining both a high average score, 4.6 (from “Good” to “Very good”). Despite the similarity, the quality of the lighting is also a highly rated feature, slightly higher than 4.5.
As for the hearing tests, all users have been successful in identifying the sound source on stage. The tests that caused the most confusion were those on the first and second floors because some of the users did not identify the origin of the sound, if not by indecision about whether it was on the second or third floor. This may be due to the fact that given the physiognomy of the human being, it is easier for us to identify sounds in a horizontal plane than in a vertical one.
Overall, the obtained results in terms of all evaluated aspects are very satisfactory and prove the ability of the presented system (HMD, and virtual reconstruction of the theatre) to provide a feeling of immersion and a pleasant Quality of Experience (QoE). This demonstrates its potential impact in the cultural heritage field, recovering old or non-existing buildings.
Conclusions and future work
The use of interactive 3D audio, together with visual rendering, allows increasing the feeling of immersion in virtual environments. In this work, the integration of different tools to work with any simulator or game that allows head movement independently of the body has been tested to develop a model of a lost building. The setup shows a lost 16th century theatre in València, which is focused on virtual archaeology environment to provide the user a sensorial experience of this building. The architectural model has been developed in Sketchup. It has been integrated in Unity to obtain visually and acoustically a first-person navigator. The integration with an audio engine, as FMOD Studio, has been studied by using a developed module to allow head tracking as an input to select HRTFs. In order to achieve high acoustic immersion, the acoustic model has also been developed based on the documentation of the archives and different studies, in order to determine the acoustic properties of the building materials.
The model has been built by taking into account the described geometry found in the ancient documentation and texturized based on existing buildings in the same century. The acoustical calibration from the experimental IRs was not possible, but the simulation is based on historical buildings of this type. For that reason, the auralizations and visualizations developed can be considered of general interest.
In this work, we have also considered an optional audio engine for 3D audio rendering apart from the usual 3D audio in Unity. We have designed a filter chain related to the use of the impulse responses of any room and integrated with the interactive selection of the correspondent pair of HRTFs according to the relative position of the head of the receiver towards the sound source. It was specifically applied to the integration of the graphical and acoustical model of the Olivera theatre. It considers the use of FMOD for auralizing anechoic audio in real-time. In our case, this tool combined with Unity allows a powerful engine for theatrical studies in the Spanish Early-Modern theatre. This is the first study for a future database of buildings devoted to the theatrical performance.
Also, further work to validate the accuracy of the system to represent the theatrical experiences will be done by using a 16th century existing theatre model, comparing the user experience with the real-life one, and to integrate specific types of smells in theatrical spaces will be included in the environment. These smells are already documented in the literature of that age. This will provide more immersive multi-sensory virtual experiences.
Author ORCIDs
Jaume Segura-Garcia, 0000-0002-9138-5465
Acknowledgments
This study has been partially supported by Ministry of Science, Innovation and Universities and FEDER funds in the grant “Juan de la Cierva - Incorporación”, in the research project Urbauramon with reference BIA2016-76957-c3 and in the Network of Excellence Consolider “Patrimonio teatral clásico español” with reference FFI.2015-71441-REDC.
Jaume Segura-García received the MSc and PhD degrees in physics from the University of Valencia, in 1998 and 2003, respectively. In 2008, he joined the Computer Science Department, University of Valencia, where he is currently Associate Professor. He has co-authored more than 90 technical papers in international journals and conferences. He is a member of the Spanish Acoustics Society and the European Acoustics Association.
Mario Montagud-Climen (https://sites.google.com/site/mamontor/) is a Senior Researcher at i2CAT Foundation (Barcelona, Spain) and a Part-Time Professor at the University of Valencia (Spain). He received a PhD degree in Telecommunications (Cum Laude Distinction) at the Polytechnic University of Valencia (UPV, Spain) in 2015. His topics of interest include Computer Networks, Interactive and Immersive Media, Synchronization and QoE (Quality of Experience). Mario is (co-) author of over 70 scientific and teaching publications and has contributed to standardization within the IETF (Internet Engineering Task Force). He is a member of the Editorial Board of international journals and has been a member of the Organization Committee of the many international workshops and conferences. He is also lead editor of “MediaSync: Handbook on Multimedia Synchronization” (Springer, 2018) and Communication Ambassador of ACM SIGCHI (Special Interest Group in Human–Computer Interaction). He is currently involved in three EU H2020 projects, being WP leader in two of them: VR-Together, ImAc, and 5G-Picture.
Sebastià Mirasol-Menacho finished his Degree in Telematics Engineering in 2018. He has co-authored several papers in national and international conferences and journals. Currently, he is working at Würth.
Joan Oleza-Simó is a Full Professor at the Dpt of Spanish Philology. He has been Dean of the Philology Faculty and was invited as a Professor in different international universities. In the field of Hispanicism, Joan Oleza stands out as the author of studies, essays, and articles on different aspects of Spanish literature, from the Golden Age to the latest narrative trends. His scientific contributions include the coordination of Max Aub's Complete Works and a Spanish Classical Theatrical Heritage project, a research macro-project on the Spanish theatre of the Golden Age with different research groups. In recent years, he has researched Lope de Vega's theatrical plays. In collaboration with doctoral students and professors from other universities, they have created a huge database on the works of Lope. As a novelist he has published several short stories and four novels in Spanish and Catalan: “La mansión roja” (1979), “Tots els jocs de tots els jugadors” (1981), a novel that won the “Premi de la Crítica del País Valencià”, “Cuerpo de transición” (1992) (free version, in Spanish, of the previous one) and “La vida infidel d'un Arlequí” (2009). He is the author of the version of La Estrella de Sevilla, by Lope de Vega produced by the Compañía Nacional de Teatro for the 1998–1999 season, which premiered at the Almagro International Festival in July 1998 and at the Teatro de la Comedia in Madrid on 1 October 1998.
