2. Proposed extended scope-dimension diagram

Mapping the current HBI research directions on the scope dimension diagram presented in Figure 1 shows less attention to the social direction of HBI developments, denoting a need to consider social interactions between occupants in providing a pleasant occupancy. Additionally, the scope-dimension framework presented in Figure 1 does not fully consider the inter-relationships between the three HBI directions, e.g., how social-physical dimensions should be combined to address occupant interaction needs while improving the energy efficiency of the buildings. As a result, we developed an extended HBI scope dimensions diagram to better understand HBI research’s current and future directions (Figure 3). To this end, firstly, we expand the scope of the HBI research presented in Figure 1 by considering different factors such as level and types of occupant-building interaction. The scope “People” denotes involvement of the occupants in HBI and can be collected through the participatory involvement of occupants or the use of sensors. In participatory involvement, occupants are contributing via surveys, questionnaires, and so forth (active engagement), whereas sensory involvement includes occupant data collection via sensing technologies (passive engagement). The scope “Built Environment” denotes the level of interaction that occupants have with the building. These interactions can be basic, e.g., the opening and closing of a door, or smart, e.g., the use of interfaces to collect data about perceivable comfort levels of occupants or regulating the Indoor Environment Quality (IEQ) parameters inside the building. The scope “Computing” denotes the level of analysis performed on the collected participatory and sensory data and can be operational or intelligent. Operational computing is supported by operating systems, communications, databases, ubiquitous computing, etc., while intelligent computing is supported by ML, NLP, Computer Vision, etc.

Figure 3.

Proposed HBI scope-dimension diagram.

Existing HBI literature, when mapped on the proposed diagram, reveals interesting insights. The mapped HBI research is shown as oval shapes on the proposed scope-dimension diagram shown in Figure 4. As previously mentioned, common HBI applications include improving energy efficiency, occupant comfort, and safety and space management. As can be observed in the Figure 4, energy-driven applications (to increase energy efficiency) lean towards physical aspect of physical-social interactions and are more widely studied to cover a range of interactions and applications. The comfort-driven studies lean more towards the social aspect of social-physical interactions representing those developing human behavior models to control energy consumption and thermal comfort. Safety-driven studies, e.g., indoor real-time location sensing (RTLS) for emergency navigation, cover the physical-spatial dimension. The studies in this category have a limited scope, using sensory data received from indoor RTLS, and not including participants’ subjective feedback through participatory data and intelligent computing. Finally, space-driven research, e.g., the use of RTLS to increase the spatial comfort of occupants, denotes the oval shape in the spatial-social dimension.

Figure 4.

Existing HBI applications mapped on the proposed scope-dimension diagram.

Mapping existing applications on the proposed scope-dimension diagram more specifically denotes the extensive focus of the literature on improving building efficiency as opposed to addressing occupant needs, and presents a pressing demand to investigate the social-spatial and the physical-spatial aspects of HBI. While current HBI design focuses on either the building processes or the pursuit of outcomes from the managerial side, it tends to forget that humans are not just another resource (Ehkirch & Matsumae Reference Ehkirch and Matsumae2024). As mentioned before, such social interactions are highly dependent on the type of building, which dictates the functionality of its spaces as well as the demographics of occupants and their interaction goals. The main focus of the current HBI research on specific-purpose buildings is again improving energy management in different types of buildings such as residential (Yu et al. Reference Yu, Haghighat, Fung, Morofsky and Yoshino2011; Chen et al. Reference Chen, Yang, Yoshino, Levine, Newhouse and Hinge2015; Tak et al. Reference Tak, Becerik-Gerber, Soibelman and Lucas2023), healthcare facilities (Mallak et al. Reference Mallak, Lyth, Olson, Ulshafer, Ulshafer and Sardone2003, Kalay & Schaumann Reference Kalay and Schaumann2021), offices and commercial buildings (e.g., Rijal, Humphreys & Nicol Reference Rijal, Humphreys and Nicol2009; Duarte, Den Wymelenberg & Rieger Reference Duarte, Van Den Wymelenberg and Rieger2013). Limited research also explored occupants’ specific needs in these building types; A study investigated the impact of ambient lighting on patients’ well-being and recovery rates in hospital environments. Other researchers proposed the use of interactive digital displays (Taher et al. Reference Taher, Cheverst, Harding and Fitton2009) and Smart Desks (Aryal et al. Reference Aryal, Becerik-Gerber, Anselmo, Roll and Lucas2019) and chairs (Labeodan et al. Reference Labeodan, Aduda, Zeiler and Hoving2016) in office environments to promote collaboration and productivity among employees.

3. Research objectives

As we delve into the complexities and possibilities of HBI, it becomes evident that understanding the dynamics of human-building interactions is not only about designing smarter buildings but about reimagining the relationship between occupants and their built surroundings. Understanding occupant needs is essential to designing smart buildings that meet the demands of their occupants and provide optimal comfort, safety, and functionality. Occupant needs in different types of buildings vary based on factors such as the building’s function, user demographics, and environmental conditions. While Human-Building Interaction (HBI) research has made significant strides in understanding and addressing the needs of building occupants, several limitations persist, including:

Limited understanding of user preferences: HBI research often relies on generalized assumptions about user preferences and behavior, which may not accurately reflect the diverse needs and preferences of building occupants;

Generalization of findings: HBI research often focuses on specific building contexts, leading to findings that may not generalize well across different types of buildings. For example, solutions that work well in office buildings may not be applicable or effective in educational or healthcare settings due to differences in user needs and building functions;

Complexity of occupant needs: different types of buildings serve diverse occupant populations with varying needs and preferences. HBI research does not adequately capture the complexity of these needs and design solutions that address them comprehensively.

As such, there is a pressing need for studying occupant demands in specific purpose buildings, including educational buildings, to tailor HBI strategies for each building type based on their occupants. Educational buildings are one of the important specific-purpose buildings that play a vital role in the community. Their architectural design and the environment that they create not only impact the students’ learning outcomes (Mulrooney & Kelly Reference Mulrooney and Kelly2020), but also can be an ideal location for community engagement (Cureton & Gravestock Reference Cureton and Gravestock2019). Current research overlooks the potential of HBI implementation in educational buildings in creating learning environments that are inclusive, engaging, and supportive of student success, thereby addressing the social dimension of HBI implementation for these buildings.

We hypothesize that addressing occupant-specific needs in buildings can have a positive impact on the experience of the primary occupants, including improvements in the learning experience of students in educational buildings. To address the aforementioned gaps, the objective of this research is to promote an occupant-centric approach towards designing HBI strategies for different types of buildings. We explore occupant needs and bidirectional dynamics of occupant-building interactions in educational buildings. First, we develop a novel HBI framework that combines HCI and AmI to enable bidirectional and multi-modal interactions between occupants and buildings. We then investigate the application of the proposed framework in enhancing the learning experience of students (as primary occupants) in educational buildings. A proof-of-concept qualitative study was conducted to explore the potential of the proposed approach in understanding and addressing the needs of occupants in educational buildings. Finally, we propose viable interventions to mitigate the identified occupant-specific needs in educational buildings.

4. Proposed extended human building interaction (HBI) framework

For a long time, HCI and AI were considered divergent fields of study because HCI encourages human involvement to make UIs more user-friendly whereas AI aims to construct human-like intelligence by using mathematical models (Cheng et al. Reference Cheng, Chiu, Hsieh, Yang and Chou2016). However, researchers from the HCI community and AI community are now coming together to improve the interaction between humans and AI systems by designing intelligent user interfaces (IUIs) (Chen et al. Reference Chen, Liu and Wu2018). Voice and text-operated assistants such as Apple’s Siri, Google Assistant, Amazon’s Alexa and ChatGPT are some examples of IUIs. Similarly, the HCI and AmI-based definitions of HBI can be combined. In this research, we propose a novel vision for an HBI Framework that makes use of this integration, aimed at creating IUIs for buildings, as shown in Figure 5.

Figure 5.

Proposed approach to Intelligent User Interfaces (IUIs) design for HBI.

The overlap between HCI and buildings represent augmented buildings with intelligent interfaces that are designed to facilitate interaction with occupants. The overlap between AI and buildings represents buildings that have AI-based systems to intelligently control building operations. Our vision of HBI lies at the intersection of HCI, AI and buildings, and focuses on improving the interactions between occupants and buildings through designing IUIs for buildings. This vision aims to facilitate real-time, bidirectional, and multi-modal interactions between the occupants and buildings. Figure 6 illustrates a model of the proposed HBI framework that is comprised of an intelligent layer to perform predictive analytics, and IUI to facilitate real-time, bidirectional, and multi-modal interactions between the occupants and the building. In this framework, data required for computational analysis is collected from occupants through both passive engagement (i.e., sensing) and active engagement (occupant participation). Participatory data ensure the active involvement of occupants in the HBI system. Similarly, data from the buildings is collected through real-time sensing as well as historical data available in the building databases, as opposed to conventional sensing as used in AmI systems.

Figure 6.

Left: Proposed Human Building Interaction (HBI) framework; Right: occupant-building data pool.

Building databases required for this purpose contain all essential data about the building that cannot be measured using sensing techniques, including as-built documentations such as plans, operation manuals, and so forth Similar to the AmI systems, the controller in the proposed HBI framework has an intelligent layer that uses AI to forecast occupant behavior and how predictions should be translated to actions in the building. The decisions made by the controller are translated into useful information and automatic actions. These automatic actions benefit both the occupants and buildings, as opposed to benefiting the building only, by providing context-aware information or guidance to occupants or actuating specific sensors in the building. The information is then conveyed to the occupants through multi-modal interactions such as audio, touch, gestures, smell, and so forth (Chen et al. Reference Chen, Liu and Wu2018). Such closed loop of interaction between buildings and their occupants increases the efficiency of the building, while ensuring enjoyable and productive occupancy by the users.

5. Application of the proposed HBI framework for educational buildings

The proposed HBI framework takes a human-centric approach in establishing bidirectional, and multi-modal interactions between occupants and buildings that are tailored to their needs. The proposed framework can be generally applied to different building types. The following steps map the approach to implement the proposed framework for each building type: (1) identify the building type and its primary occupants; (2) identify the primary occupants’ specific needs and preferences; (3) identify the data types required for addressing the identified needs and the respective data acquisition methods; (4) collect and analyze data and extract the required knowledge to make automated decisions based on the application; and lastly, (5) design HBI Intelligent User Interfaces (IUIs) to convey the decisions to the occupants.

In this section, the application of the proposed framework to enhance the learning experience of students, as primary occupants, in educational buildings, is presented through a case study that was carried out at Virginia Tech. To this end, first, the primary occupants and their needs were identified. The impact of the identified needs on the learning experience of students, identified as primary occupants, were then evaluated. The existing BAS in the selected building (Goodwin Hall) was also assessed. Finally, potential technologies to complement the current BAS to achieve the added benefits of the proposed HBI framework to improve the learning experience of students were evaluated and suggestions were provided.

6. Case study: enhancing the learning experience of students in Goodwin Hall

In this section, a step-by-step application of the proposed framework to enhance the learning experience of the students in Goodwin Hall, as a representative technology-enhanced building at Virginia Tech, is presented.

6.1 Goodwin Hall building type and its primary occupants

Goodwin Hall is an academic building at Virginia Tech with 160,000 sq ft of space. GH consists of 5 floors in an L-shaped design. It is the flagship smart building which houses 40 instructional and research labs, 8 classrooms, an auditorium, and 150 offices for several engineering departments making it the main home for the engineering students (mechanical and chemical). Some of the courses are open to other majors requiring large classrooms, which adds a transient population of students occupying the building daily. The first floor hosts the classrooms and the auditorium that are usually used for undergraduate engineering classes (Figure 7a). The upper floors have the same footprint and host laboratories and faculty/ staff offices (Figure 7b). Goodwin Hall has around 240 accelerometers attached to 136 sensor mounts throughout the building’s ceilings that can measure all vibrations made inside the building. It is also equipped with temperature and CO2 sensors to regulate the indoor climate.

Figure 7.

Goodwin Hall’s (a) first floor plan, and (b) 2–5th floor plans.

The primary purpose of any academic building like Goodwin Hall is to provide an adequate and efficient learning experience for the students. Primary occupants of Goodwin Hall include undergraduate and graduate students whereas the secondary occupants are faculty members and administrative staff. Considering the number and capacity of classrooms, laboratories and graduate offices and the average working hours for graduate students (~15–20/per week) as well as an average of 15 credit hours for undergraduate students, the students are considered the primary occupants of the building. Other occupants include faculty/staff, and temporary occupants including visitors participating in events held in the auditorium.

6.2 Occupant needs in Goodwin Hall

To identify the students’ needs and preferences while occupying the building, we conducted focus groups and semi-structured interviews. Qualitative analysis was conducted to identify important categories of needs. These categories were grouped using concept mapping to identify various themes and relationships among categories, codes, and subcodes.

6.2.1 Qualitative data collection

Focus groups and semi-structured interviews were conducted among 12 undergraduate and 7 graduate students. Although qualitative data collection methods (in this case study, in-depth focus groups and semi-structured interviews) are perceived as less rigorous compared to quantitative methods, they are known to offer unique strengths and advantages for exploring the complexity, and diversity of human experience and perspective (Denzin & Lincoln Reference Denzin and Lincoln2018). During focus groups and semi-structured interviews, the participants were questioned about their positive and negative experiences in the building, potential applications and advantages of building IUIs for improving their learning experience, as well as their preferred interaction modalities and potential data privacy concerns. All focus groups and semi-structured interviews were audio-recorded with consent from the participating students.

Designing a qualitative research plan and defining its structures and attributes (e.g., sample size) is impacted by various aspects such as research topic, cultural content, the scope of the research, and access to diverse participants (Marshall et al. Reference Marshall, Cardon, Poddar and Fontenot2013). While previous studies show that there is no fixed equation for calculating the right number of participants in a qualitative study, there are some measuring factors that can help the researchers define and justify their sample size. Research shows that in qualitative research, small sample sizes allow for in-depth analysis of individual cases, enabling researchers to explore complex phenomena with rich detail (Mason 2018). As such, the sample size can be made based on joint evaluation of epistemological and practical concerns that are derived from preliminary research and during the study preparation (Robinson Reference Robinson2014). “Data saturation” (Glaser & Strauss Reference Glaser and Strauss2017) is also a concept that identifies the point in a study where no additional data or new knowledge is being collected. In our case, the final three interviews demonstrated repeating data, where many shared experiences similar to the previous interview and focus group sessions were emphasized by the participants and no new knowledge was added.

6.2.2 Qualitative data analysis

The collected audio data was transcribed to text, and qualitative coding including In-vivo and Focused tools were used to identify the main occupant need categories, codes, and subcodes from the raw text data. Initially, audio-to-text transcription was done using web-based services that resulted in a total of 12 transcribed documents. These documents were manually checked to remove possible errors in transcription. In-vivo coding was used to assign labels to the data collected from the focus groups and semi-structured interviews. It uses words or short phrases from the participants’ sentences in the transcribed data as codes. Focused coding was further used to categorize the most frequent or significant codes and subcodes and develop categories. Qualitative coding was conducted using MAXQDA software. Two iterations were conducted to validate the reliability of the coding that resulted in a total of 580 In-vivo codes. In the first iteration, 228 In-vivo codes were identified and in the second iteration, the number of In-vivo codes was increased to 352. Both iterations were performed by the same researcher with a gap of 2 weeks to account for the potential bias in qualitative coding. Intra-coder reliability tests were performed on the In-vivo codes from the two iterations to check their validity. Percent agreement and Cohen’s Kappa statistics were conducted to check the intra-coder reliability of the In-vivo codes. The purpose of intra-coder reliability is to improve the quality of the identified In-vivo codes. The result of the intra-coder reliability test is shown in Figure 8.

Figure 8.

Percent agreement and Cohen’s Kappa statistic for intra-coder reliability test.

Focused coding was performed on the second iteration to group significant In-vivo codes into categories, codes and subcodes. Focused coding has its roots in grounded theory (Giles, de Lacey & Muir-Cochrane Reference Giles, de Lacey and Muir-Cochrane2016) and is used to identify the codes that are relevant to the research themes. Grounded theory comes into play where data collection and analysis happen simultaneously. This informs the collection process of what to collect and the analysis process of what is relevant and what needs further analysis (Flick Reference Flick2013). In our case, focused coding helped recognize the codes that were not relevant to further processes and thus 51 codes were dropped. As a result, 301 of the 352 In-vivo codes were considered for focused coding and the rest were discarded due to their insignificance to the research theme. The remaining 301codes went through another round of coding and were assigned weights according to their frequency of occurrence. All the categories, codes and subcodes were assigned a weight from 1–10 based on their frequency of occurrence and the emphasis given by the participants in the focus groups and semi-structured interviews (Table 1).

Table 1.

Frequency and weight of categories, codes and subcodes identified from focused coding

6.2.3 Results of qualitative data analysis

Concept maps were created to explore relationships among the identified codes and subcodes to assess their impact on the learning experience of students in the building (Figure 9). The key themes that emerged from the concept mapping exercise were “Navigation,” “Learning,” “Comfort,” “Stress Management” and “Safety.” The theme “Navigation” includes the codes “finding professor,” “finding parking,” “traffic management,” “indoor navigation,” and “finding study spaces.” “Learning” consists of codes “finding study spaces” and “study resources.” “Comfort” is concerned with “audio,” “visual” and “thermal” comfort levels of the occupants when conducting primary activities such as attending classes or studying. “Stress management” includes “receiving notification about the events happening in the building.” “Safety” includes the protection of occupants in case of an emergency such as fire or shooting inside the building. A summary of the themes with their cumulative weights is shown in Table 2. The theme with the highest cumulative weight is “Comfort,” followed by “Navigation” and “Learning.”

Figure 9.

Concept map showing code relationships of category “Occupant Needs.”

Table 2.

Themes and cumulative weights in category “Occupant Needs”

The codes “finding study spaces” and “study resources” were each found to have a direct impact on learning; Many undergraduate students highlighted that even though Goodwin Hall has a lot of open spaces, it is difficult for them to find an appropriate place to study in the building. Finding study spaces is a common problem that many undergraduate and graduate students usually face in educational buildings and libraries. Students also expressed the difficulties they faced when searching study resources such as proper furniture, power outlets, and whiteboards in the buildings. The most important code observed in Figure 10 is “comfort” and although it does not directly contribute to the learning experience, it has a significant indirect impact on enhancing the learning experience by providing students with a comfortable environment. “Comfort” was a primary concern for the graduate students who spent more time in Goodwin Hall. They complained about the lack of windows in their labs and how it makes them feel disconnected from the outside world and even claustrophobic. Lack of “audio comfort” was a problem for both undergraduate and graduate students, the undergrads preferred “study spaces” that were less noisy, and the graduate students complained about being disturbed by noises while working in their labs. “Thermal comfort” was also important for all students, they demanded more control over regulating the indoor temperature especially in offices. “Stress management” was a concern for graduate students because they spend a lot of time in the building and need some means to help them relax in between their studies. A possible solution for “stress management” was identified as “event notification,” to inform the students about the events happening in the building so they can attend in their free time. “Finding professors” was also a concern for the graduate students because most of them are working as graduate assistants with a professor and sometimes face difficulty to locate their respective professors. “Traffic management” is related to “finding parking” as well as receiving information regarding building occupancy at different times. Providing “safety” denotes protection in cases of possible emergencies such as fire and intruders. All participants were concerned about their data privacy and security but were willing to share their location and schedule data as long as the shared data remains anonymous and does not include any identifiers.

Figure 10.

Relative importance percentages of identified codes on learning experience.

6.3 Identify and collect “Data Types” from Goodwin Hall

The “occupant data types” and “building data types” identified from the focus groups with primary building occupants were then categorized based on the main need codes. The required sensing and interaction strategies to address the occupants’ needs regarding the theme “navigation” are shown in Table 3 as an example. It can be observed that most of the “building data types” are concerned with location, which can make use of Goodwin Hall’s accelerometers. A downside of accelerometers is that they can only determine the location of moving objects and not stationary objects. For the location of stationary objects, sensors such as RFID, BLE, and Wi-Fi can be used.

Table 3.

Data collection and interaction strategies for theme “navigation.”

Similarly, within the theme “Learning” “building data type” for the “Study Resources” and “Finding Study Space” which are the two identified codes with direct impact on “Learning Experience” include building layout, furniture locations and availability, power outlet and whiteboard locations, and so forth Currently there is no existing “building data type” or collection means available for any of the mentioned codes. “Building data types” for “Comfort” include luminance, noise level, and temperature and these are measured with light sensors, microphones, and temperature sensors, respectively. “Occupant data types” for “Comfort” include location and subjective feedback through surveys. These along with sensing techniques and interaction strategies for “Comfort” are shown in Table 4.

Table 4.

Data collection and interaction strategies for theme “Comfort.”

8. Conclusion and future works

This paper compiles existing research on “Human-Building Interaction (HBI)” to identify gaps in HBI theories and their applications. The results show existing gaps in taking occupant-centric approach towards designing HBI systems that consider the specific needs of the occupants in different types of buildings and enable bi-directional knowledge transition between the two. To address this need, we propose a novel vision to develop a comprehensive HBI framework that integrates the concepts of HCI and AmI with HBI to increase the intelligence of building and their interaction with occupants by identifying the type of the building and its primary occupants, investigating the needs of the identified occupants, and finally designing user interfaces that enables bi-directional interaction between buildings and occupants based on the identified needs of both. Such an approach facilitates bidirectional and multi-modal interactions between occupants and the built environment. A proof-of-concept study was conducted at Virginia Tech to evaluate if and how implementing the proposed occupant-centric approach can enhance occupants’ i.e., students, experience in an educational building. Focus groups and semi-structured interviews were conducted among undergraduate and graduate students to identify their needs in the studied technology-enhanced educational building, Goodwin Hall, and whether the currently available technology in the building is helping address their needs. The collected qualitative data was analyzed using qualitative coding and concept mapping. The results revealed that the HBI aspects emphasized by the occupants, i.e., students, in educational buildings were widely different from the standard factors considered in designing conventional HBI strategies. Even when promising to improve occupant comfort, the underlying goal of the conventional approaches is usually to improve the efficiency of the building operations. Based on the results of our study, the key occupant need themes were related to “Navigation,” “Learning,” “Comfort,” “Stress Management” and “Safety,” which are not usually considered when designing HBI systems for buildings. This emphasized the importance of considering the building type and the particular users in designing HBI strategies that benefit both the buildings and their occupants.

Qualitative data analysis also revealed that the code “study resources” has the highest direct impact and the code “comfort” has the highest indirect impact on the learning experience of students in the studied building. This result further highlights the requirement to consider both the building and occupant types in designing appropriate HBI systems and technologies. The existing sensing technologies, collected data types and interfaces were studied to create a data pool for Goodwin Hall. The results show that the existing data and data collection methods, i.e., accelerometers, temperature, humidity and CO2 sensors, are inadequate to address the identified needs of the occupants and additional sensors to measure comfort, stress, and safety are also needed. Based on the identified themes from qualitative data analysis, the features of the HBI system for Goodwin Hall should be selected to make indoor navigation easier, provide essential study resources, help in maintaining indoor environmental quality, and ensure a stress-free and safe social atmosphere. We then discuss how different UIs can be integrated into the building to address the identified needs. For example, various interaction principles including building affordances and advanced technologies such as voice assistants and AR can be used to provide the occupants with useful information and provide ubiquitous personalized services encompassing teaching/learning and psychological needs based on occupants’ contexts.

While the conducted user study was limited in terms of sample size, the results still yield valuable insights, generate hypotheses for further investigation, and inform the design and development of future research studies. Due to the unavailability of the required infrastructure, the proposed IUIs were not implemented in the targeted building and their potential impact was not investigated. In our future work, we will implement and assess the impact of the proposed IUIs to address the needs identified in this research. A more comprehensive study of implementing the proposed framework in buildings with different functionalities will also help further understand the impact of the building and occupant types on HBI design requirements.