Organization of paper

Based on the need for a more comprehensive understanding of diversity measurement in computational design, this paper consolidates existing information on the topic and offers new data concerning the link between quantitative diversity metrics and human perception of design diversity. The first section provides a literature review of diversity measurement and potential metrics for use in parametric design, before detailing the contributions of this paper. The next section describes the methodology of a survey-driven research study that tests how well these measurements match human judgments of design diversity and the relative performance of each measurement technique. The third section presents the results of the research study, while the fourth offers a discussion of the results and contributions, as well as an example application. Finally, the last section indicates future work and gives concluding remarks.

Background on diversity measurement

Diversity, especially when it is being measured, is also called variation, dissimilarity, or novelty in academic literature, although there can be slight differences between definitions. In comparison with other disciplines, architecture and related design fields have only recently become concerned with quantifying diversity, as parametric and other forms of computational design have begun using algorithms and processes to generate high-performing yet expressive geometries (Brown & Mueller, Reference Brown and Mueller2016). As such, many mathematical methods for measuring diversity are more widely used in other fields, such as biology, where measurement of population genetic diversity has been studied for much longer (Patil & Taillie, Reference Patil and Taillie1982). Nevertheless, many of the concepts created to compare species or phenetic differences within a species are mathematical or geometric in nature and can be applied to conceptual design, especially if design options are defined in a way that can be represented numerically.

However, a key difference between the measurements of biological diversity and parametric design diversity makes it difficult to use biological methods directly in design settings. When used in the context of biology or ecology, a diversity index is a quantitative measure of how many different types exist in a dataset, and how evenly the entities in the dataset are distributed across these types (Hill, Reference Hill1973). Sociologists can similarly use these existing metrics to compare types of people (Agresti & Agresti, Reference Agresti and Agresti1978), and information scientists can use them to distinguish between characters in a password, for example. Common diversity metrics in these situations include the Gini–Simpson Index (Rao, Reference Rao1981), which indicates the probability that two entities randomly have chosen from a set represent different types, the related Inverse Simpson Index, and the Shannon Entropy Index (Shannon & Weaver, Reference Shannon and Weaver1949), which quantifies the uncertainty of being able to correctly predict the type of the next entity in a given set (Jost, Reference Jost2006). Rényi's entropy (Rényi, Reference Rényi1961) is also used for similar purposes. Within the field of engineering design, Kan and Gero (Reference Kan and Gero2018) utilize the concept of entropy to quantify the creativity and productivity of a design session by scrutinizing linkographs from protocol analysis.

In contrast, the different designs being compared within a parametric definition often cannot clearly fit into distinct “types”, especially if there are many continuous variables. It is possible to manually group designs within a set, or use clustering algorithms to achieve categorization automatically, but parametric design spaces frequently transform too gradually and subtly for an approach relying on a first-step classification to yield meaningful results. Such sorting-based metrics could be used when comparing typologically different conceptual architectural sketches generated during a brainstorming process, but this comparison is less directly relevant for implementation in computational settings and is thus outside the scope of this paper. Although clustering is a promising idea and will be discussed later, this paper focuses on faster, single-step, geometric measurements that can be applied for early-stage conceptual design models. Nevertheless, despite the difficulty in using probabilistic biological measurements directly, many mathematical techniques related to their calculations can be applied to quantify diversity in parametric design.

It is possible to borrow concepts from other fields because most diversity metrics rely on distance measurement. Distance measurement is itself a distinct topic, and although it is used to create the diversity indexes mentioned above, it exists within the more fundamental fields of geometry and statistics. Since the question of diversity in parametric design is not necessarily how many different types are present, or how well distributed these distinct types are across a set, but rather involves characterizing how far apart different designs are in relation to all other designs, distance measurements can be especially useful for design settings. However, in most cases, a distance measurement must be combined with another geometric or statistical concept to create a meaningful metric relating to an entire set of designs. The difference between measuring raw distance, versus attempting to quantify the diversity of an entire design set based on combinations of distances or other geometric properties, is illustrated in Figure 1.

Fig. 1.

A diagram illustrating the difference between measuring distance and measuring the diversity of a set of designs. In the image on the left, the distance between two sets of parametric designs can represent their dissimilarity or variation, as closer designs look more similar than those far away; in this example, d ₁ is clearly greater than d ₂. However, if trying to quantify the diversity for the set on the right, judgments must be made about which geometric properties to use.

Relevant distance measures include Euclidean, City-block, Correlation, Geodesic, and when the distance is between groups rather than points, χ ² and Mahalanobis distance (McCune & Grace, Reference McCune and Grace2002). Distance metrics are widely studied and characterized in computer science, as the choice of method can have a significant effect on algorithmic performance in applications such as pattern recognition, data mining, segmentation, and signal analysis (Kokare et al., Reference Kokare, Chatterji and Biswas2003; Perlibakas, Reference Perlibakas2004; Wang et al., Reference Wang, Neskovic and Cooper2007). Although there are important differences to each, many of these metrics attempt to normalize for the size of the entire possibility space or adjust based on the relative importance of the axes being considered (in the case of ecology, these axes generally represent species). As such, they often collapse to Euclidean distance in simple cases when axes are equal and already unitized. However, each of these measurement techniques, although not necessarily directly applicable to measuring set diversity rather than pair similarity in parametric design, are worth considering because they have the potential to inspire modified versions that can be applied in architecture or structures. More sophisticated distance metrics, such as the Manhattan distance, may also be required to increase the accuracy or effectiveness of diversity measurement for some applications, including higher dimensional problems (Aggarwal et al., Reference Aggarwal, Hinneburg and Keim2001). This paper makes use of distance measurement in each diversity metric considered.

Creativity, novelty, variety, and related concepts in engineering design

In engineering design, there has been considerable research into idea generation during early-stage brainstorming, often called product ideation (Shah et al., Reference Shah, Kulkarni and Vargas-Hernandez2000). While attempting to improve ideation methods, researchers have created empirical laboratory or simulated environment studies that require characterizing and measuring ideation effectiveness (Shah et al., Reference Shah, Smith and Vargas-Hernandez2003). Common metrics for evaluating the outcomes of brainstorming sessions include creativity, novelty, and variety. Novelty is defined as being original, rare, unusual, or unexpected when compared with other ideas, while Variety is a characterization of the explored solution space. Measuring novelty often involves defining what is not novel, decomposing a problem into key functions and characteristics, assessing how an individual design fulfills these functions, and then grading its novelty with reference to all other possible designs. Although variety applies to a group of ideas rather than an individual idea, it is often measured by examining how different functions are satisfied and calculating differences in physical/working principals, followed by less relevant details such as dimensions. Creativity is defined in a variety of ways, including pure notions of novelty or originality (Runco & Charles, Reference Runco and Charles1993), or adding aspects such as quality, workability, and relevance (Dean et al., Reference Dean, Hender, Rodgers and Santanen2006), or importance and affect (Horn & Salvendy, Reference Horn and Salvendy2009). Others have proposed defining creativity in an entirely subjective manner based on observer response (Amabile, Reference Amabile1982).

Diversity in the context of architectural computational design aims to capture some combination of these inter-related concepts of novelty, creativity, and variety. However, within the parametric design, there is a direct need for numerical measurements that do not depend on expert judges or qualitative assessments and can thus be applied immediately in current design environments. Many architects and engineers working in building design are already relying on strategies such as creating catalogs of the design space or using multi-criteria optimization to obtain sets of designs for final selection (Balling, Reference Balling1999; Stump et al., Reference Stump, Yukish, Simpson and Harris2003). These designs are frequently generated rapidly using parametric scripts, as described in the next section. Although performance simulation is regularly attached to decisions between options (Tsigkari et al., Reference Tsigkari, Chronis, Joyce, Davis, Feng and Aish2013), experts do not necessarily sort designs into hard categories of feasibility, functionality, or other traditional engineering metrics. This is because it is common for parametric options in later conceptual design to be typologically or functionally consistent, yet contain subtle visual or geometric differences that are still relevant to the designer as he or she weighs the quantitative performance and qualitative experience of a future building. Thus, metrics for diversity measurement in this paper are valuable because they can be used to tune algorithms and processes in human–computer collaborative design, leading to creative, innovative, yet high-performance design outcomes.

Diversity in parametric formulations

It has become increasingly common in architectural and engineering design to define designs parametrically, using digital software to generate a geometry based on a set of variables (Burry, Reference Burry1996; Monedero, Reference Monedero2000; Holzer et al., Reference Holzer, Hough and Burry2008). This approach lends itself to both rapid design space exploration since many potential designs exist for a given parametric formulation, and optimization, since different design possibilities can be evaluated for performance and these variables can be adjusted accordingly using a guiding optimization algorithm (Oxman, Reference Oxman2008). Parametric software, such as Grasshopper for Rhinoceros (Robert McNeel & Associates, 2014) or Dynamo (Autodesk, 2013) enable parametric formulations as part of an integrated conceptual design process. Since parametric formulations at their root contain a numerical design vector, which denotes a point in space that is mathematically distinct from other vectors, the diversity of a parametric set of designs can be measured using mathematical or geometric techniques.

Parametric design also has limitations, which can relate to how diversity is quantified computationally. Although parametric design is more flexible than earlier digital design tools, which focused on documenting a single idea, it requires considerable input and knowledge by the designer to properly formulate the parametric logic and set appropriate variables and constraints. Parametric definitions are also commonly frozen to a specific typology, which can be limiting in architectural or structural design, and their results can be dependent on specific decisions made during the formulation of the design problem itself. In this way, a human could easily draw a suspension, arch, cable-stay, and truss bridge on a sheet of paper, and declare that this collection of designs is more diverse than any parametric set offered in this paper. Computers would not only have a difficult time generating each of these structures from a single definition, but also measuring diversity without a hierarchical rubric of function, aesthetics, or other properties devised by a human designer.

Nevertheless, despite still being reliant on human creativity and labor, parametric design is a useful tool for all types of designers looking to rapidly iterate, prototype, and explore. Consider Figure 2, which illustrates diverse design spaces in the physical world and a digital design context, both of which are relevant to architecture. Diversity during the design process can stimulate creativity and imagination, leading to a more diverse built environment. At the same time, it also allows designers to consider more ideas during brainstorming, which can improve design outcomes. The left image shows bridges over the Sumida River in Tokyo, where there are 26 bridges in 27 km. Despite each site having a similar span and loading, the bridge designers felt it was important to respond to a diverse range of creative structural solutions rather than one standard form. The image on the right shows a sampling of options generated by a parametric script for a hanging canopy structure. Despite limitations to parametric design, the visual diversity of the canopy solutions on the right would give designers substantial creative freedom to select an effective and expressive design. When such diversity is the desired outcome, the underlying mathematics behind design vectors in parametric modeling becomes an advantage, since it can be measured computationally using geometric concepts. The next section describes existing methods for this purpose. First, metrics that are evaluated comparatively in the survey portion of this paper will be presented, after which other related or composite methods will be described.

Fig. 2.

Diversity of architectural solutions for similar design conditions, in both the physical world and parametric design. The bridges in Japan demonstrate the creative instincts of architects and engineers during design, which leads them to consider larger design spaces. Although the designs on the right were generated with a parametric script, there are many visually different options to consider for the final solution. Images of the bridges are from (Google Maps, 2017).

Methods for measuring diversity in parametric modeling

This paper consolidates and synthesizes existing literature on diversity measurement to highlight the following metrics for use in parametric design. Some of these metrics have been directly proposed and used by others before, and are cited accordingly. Each metric measures the diversity of a set of designs, which is sometimes required to contain a minimum number of designs in order for the measurement to be possible. The study does not assign priority to design variables or classify designs into species or types, and the vectors are normalized to the unit square. The problems in this paper are two-dimensional (2D) – it is assumed that the proposed metrics would apply to early-stage parametric designs, which generally have low dimensionality so that designers can meaningfully interact with all variables that influence visual appearance and performance. As such, standard Euclidean distance was chosen as an intuitive, commonly accepted distance measurement for these applications. Euclidean distance between two points in space is defined as:

(1)$$\hbox{dist}(\,p\comma \,q) = \sqrt {{(\,p_1 - q_1)}^2 + {(\,p_2 - q_2)}^2 + \cdots + {(\,p_n - q_n)}^2}.$$

Distance from centroid to outlier

The first diversity metric under consideration is the maximum distance between the centroid of the group and each of the individual designs, effectively measuring the furthest outlier in the set:

(2)$$D = \max [\hbox{dist}(x_{{\rm centroid}}\comma \,x_i)].$$

This metric was proposed by Mueller and Ochsendorf (Reference Mueller and Ochsendorf2015) and is used in Brown et al. (Reference Brown, Tseranidis and Mueller2015). Although a single outlier can have a large effect on the value of the metric, the distance to this outlier is only considered once, rather than multiple times, as is the case with other methods.

Sparseness at center (centroid and median)

The second metric involves a similar distance measurement but considers each of the points in the set simultaneously. It does so by measuring the sparseness at the center of the design set, which is the average distance between a given point and each of the surrounding points. Thus, the diversity is equal to the measured sparseness:

(3)$$D = \displaystyle{1 \over k}\mathop \sum \limits_{i = 0}^k \hbox{dist}(x_{{\rm centroid}}\comma \,x_i).$$

This concept is taken from the novelty search algorithm developed in Risi et al. (Reference Risi, Vanderbleek, Hughes and Stanley2009) and Lehman and Stanley (Reference Lehman and Stanley2011). This algorithm, designed to overcome difficulties in finding intermediate, stepping stone solutions in fitness-driven evolutionary optimization, repeatedly measures sparseness and rewards points in sparser areas of the design space. This paper considers two different metrics involving sparseness to evaluate the diversity of a set of parametric designs. The first takes the average distance from the centroid, as described in Eq. (3) above. The second takes an average distance from the median value for each dimension of the design set:

(4)$$D = \displaystyle{1 \over k}\mathop \sum \limits_{i = 0}^k \hbox{dist}(x_{{\rm median}}\comma \,x_i).$$

Although these two measurements are often similar, they have different sensitivities to outliers and would yield drastically different results in the case where the centroid is not near the median.

Enclosing hypersphere

The next two diversity metrics involve geometric properties of the entire set rather than direct distance measurements. The first metric is the radius of the smallest hypersphere in n-dimensional space that contains each of the designs in the set:

(5)$$D = \hbox{min}(r_{{\rm hypersphere}}).$$

Finding the enclosing hypersphere, sometimes called the bounding or enclosing ball, is a common problem in computer graphics, computational geometry, and statistics. Consequently, various algorithms exist for this purpose, including Ritter (Reference Ritter and Glassner1990) and Welzl (Reference Welzl1991). These algorithms have different runtimes and degrees of accuracy, but for diversity measurement, faster approximations are likely acceptable. The idea for employing the enclosing ball to quantify diversity comes from Pavoine et al. (Reference Pavoine, Ollier and Pontier2005), which uses enclosing hypersphere radius while discussing quadratic entropy as a measurement for diversity when the entities being considered are divided among a fixed set of categories.

Convex hull

The final diversity metric studied in this paper is the volume of the convex hull formed by the designs in n-dimensional space:

(6)$$D = \hbox{volum}\hbox{e}_{{\rm convex\; hull}}.$$

The convex hull is often described in two dimensions as the shape a rubber band would take if stretched around all points in the set. Convex hulls are used by Cornwell et al. (Reference Cornwell, Schwilk and Ackerly2006) to look at habitat filtering, by Layman et al. (Reference Layman, Arrington, Montana and Post2007) to measure trophic diversity in food webs, and by Villéger et al. (Reference Villéger, Mason and Mouillot2008) to quantify functional richness of communities. A description of the suitability of convex hulls for these tasks, as well as their pros and cons in diversity measurement, is given in Podani (Reference Podani2009). As with the enclosing hypersphere, various algorithms exist for calculating its volume for a given set of points, including the “convhull” and “convhulln” functions in MATLAB.

Research contributions

After reviewing existing methods for measuring diversity within parametric formulations, it is clear that although a number of possible metrics exist, there is no consensus on the single best metric. Furthermore, when considering the human-centric context of architectural or structural design, there has been no rigorous investigation as to whether or not the given metrics accurately translate to a human perception of design diversity, and how the use of different diversity metrics might influence the conceptual design process. When design computation applications require diversity measurement, many researchers borrow an existing methodology, propose their own that is problem specific (Yousif et al., Reference Yousif, Yan and Culp2017), or suggest that designers themselves should choose one interactively (Erhan et al., Reference Erhan, Wang and Shireen2015), since the metric itself was not the focus of their research.

In response, this paper contributes to existing scholarship on the topic of diversity measurement and connects it to computational design in a number of ways. First, it consolidates methodologies proposed in different fields and provides an overview of available measurements for future use by both designers and researchers. Second, it demonstrates the extent to which existing diversity measurement metrics match up with human perception, which is essential to understanding how humans and computers can best collaborate while creatively brainstorming during the architectural or engineering conceptual design process. Finally, it provides a quantitative comparison between each of the methods as well as a qualitative discussion of their merits, which can assist tool developers and even designers in selecting the most appropriate diversity metric for a given application. The following sections describe the methodology and result leading to these last two contributions.

Qualitative comparison of metrics

In addition to the results of the study, there are clear advantages and disadvantages to some of the diversity metrics. For example, duplicates do not affect the geometric bounding techniques in the same way that they influence the metrics that measure distance for each point individually. Each metric is sensitive to extreme outliers in the set, although some may be affected more than others. Geometric measurements such as the convex hull and enclosing ball are dependent on the specific algorithm used to approach the problem, and may be less accurate or more computationally intensive, especially compared with the sparseness measurements. Of the tested metrics, the convex hull calculation also requires a minimum number of unique points, which can be problematic for design spaces of higher dimensions, a problem that is highlighted in the example shown in the next section. Especially in these situations, the convex hull volume can also yield results on different orders of magnitude, which may require its own special normalization. The methods tested in this paper are deterministic, but for other methods that include randomness, such as categorization by clustering, accuracy and reproducibility may be an issue. Each of these properties may be worth considering when selecting a diversity metric for measurement depending on its specific use.

Example application

To demonstrate potential applications of this research, diversity metrics were applied to a tower study, illustrated in Figure 10, that is more complex than the 2D shapes but is still of relatively low dimension. The parametric definition of the high-rise tower contains ten variables related to the floorplate shape, size, and twist of the building. For this example, it is assumed that the designers and developers have a target building volume, roughly corresponding to available square footage, but desire a signature, atypical geometric architectural solution for their site. First, an evolutionary solver was applied to the parametric design space to find a set of ten potential designs that are near to the desired volume. The entire design space was then randomly sampled at a resolution of 1000 designs, and the ten designs closest to the target volume were chosen as possibilities. Finally, the 50 designs closest to the target volume were attempted in ten different combinations. The diversity of these sets was measured using each metric in this paper except the convex hull, which requires more points to run for a problem of this dimension. The process then returned the most diverse set returned during the exercise. This crude diversity forcing technique does not require any additional performance evaluations from the sampling approach, but only a bit of extra computation for the diversity calculations. As can be observed visually, the process driven by diversity metrics sacrifices some in terms of performance, but exhibits considerably more variety and expressiveness than optimization or random sampling alone. Although this example is a rough approximation of more sophisticated algorithms, it nevertheless demonstrates the creative potential of diversity-based computational design workflows.

Fig. 10.

The results of an example parametric tower study, showing the novelty of a design set generated using a process based on diversity metrics in this paper.

Contributions toward diversity metrics for human-centric design

In summary, this paper has consolidated literature from different areas and extracted techniques that could be useful for designers in architecture, engineering, and related fields looking to consider performance and visual appearance simultaneously. It has demonstrated agreement between visual perception and digital calculation of diversity for sets of simple geometries that are often considered during conceptual design, although this agreement becomes unreliable when differences in diversity are not large enough. It also offers both a quantitative and qualitative comparison of five different metrics, which have various strengths and weaknesses, but all appear to be acceptable at the magnitudes in relative diversity that concern human designers. Finally, this paper has provided ideas and examples for researchers looking to integrate considerations of visual diversity and novelty into interactive, computational design processes.