I. INTRODUCTION
The implementation of computer coupling of phase diagrams and thermochemistry (CALPHAD) techniques early in alloy development efforts provides a critical opportunity to accelerate material transition. Efforts by the CALPHAD community to develop models and thermodynamic databases have modernized the traditional alloy development process. Now, vast compositional spaces can be quickly and economically explored using equilibria simulations as an initial screening step.Reference Miracle, Miller, Senkov, Woodward, Uchic and Tiley1–Reference Miracle, Majumdar, Wertz and Gorsse4 Recent development efforts are shifting towards characterizing compositionally complex, multiprincipal element alloys since conventional metallic systems in which a single element dominates the composition are not expected to meet demanding requirements for future applications.Reference Yeh, Chen, Lin, Gan, Chin, Shun, Tsau and Chang5–Reference Zhang, Zhang, Chen, Zhu, Cao and Kattner8 Exploration of the vast composition space benefits from CALPHAD tools to assist in alloy discovery; however, it is unclear how well currently available thermodynamic databases can reliably predict phase equilibria in nontraditional alloys that fall outside the bounds of well-described compositional space.
Many detailed resources outline the history, theory, and practice of CALPHAD.Reference Kattner9–Reference Kaufman and Bernstein11 In short, a high-order polynomial free-energy model is constructed for each relevant phase in a multicomponent system and the system equilibria response is predicted by determining the minimum global Gibbs energy. The calculations employ material descriptions containing thermodynamic properties with model parameters obtained by fitting experimentally measured and/or atomistically derived phase equilibria. CALPHAD relies on a hierarchical scheme to provide equilibria predictions for multicomponent systems. A higher-order system can be modeled via free energy extrapolation from relevant lower-order systems.Reference Chou and Chang12 For example, the Gibbs energy G φ for phase φ in an A–B–C ternary system can be defined as a function of the Gibbs energy contributions from (i) the three pure elements, (ii) the three constituent binaries, and (iii) the ternary itself [Eq. (1)].
$${G^\varphi } = f \left(\sum\limits_{i = {\rm{A}},{\rm{B}},{\rm{C}}} {\left( {x_i^\varphi G_i^\varphi } \right)}, \,\,\sum\limits_{i,j = {\rm{A}},{\rm{B}},{\rm{C}}} {\left( {x_{ij}^\varphi G_{ij}^\varphi } \right)}, \,\,G_{ijk}^\varphi\right)$$Here, the Gibbs energy for the φ phase in ternary composition space, G φ, is modeled using compositionally dependent weighted contributions of the φ phase Gibbs free energy from each of the constitutive elemental and binary systems, 
$x_i^\varphi G_i^\varphi$ and 
$x_{ij}^\varphi G_{ij}^\varphi$, respectively. A ternary interaction term 
$G_{ijk}^\varphi$ is sometimes necessary, particularly if the phase is not contained within constituent systems. Ideally, if the model parameters are fully assessed for each term, then the calculated Gibbs energy should reflect the experimentally observed thermodynamic properties and equilibria states. Incorrect energy descriptions or missing terms, such as an unassessed or partially assessed binary, may lead to inaccurate representations of the Gibbs energy for phase φ and potentially change the predicted stability of the phase.
Gibbs energy descriptions are contained in “databases”—most of which have developed organically. Experimental development of phase diagrams has focused on providing thermochemical equilibria for common-use engineering alloy families (i.e., steels, titanium alloys, and nickel superalloys), so it follows naturally that CALPHAD maturation has focused on such principal component systems. CALPHAD databases are usually dedicated to a specific alloy system and consequently contain thermochemical inputs and Gibbs energy model parameters for sets of significant components [e.g., a primary element (PE) and important alloying elements]. The CALPHAD methodology has been proven in a variety of systems, primarily by interpolating within compositional bounds of existing engineering alloys for which these dedicated databases have been constructed and validated.Reference Senkov, Senkova and Woodward13–Reference Senkov, Zhang and Miller15 It is unclear to what extent CALPHAD can inform novel alloy spaces beyond the defined compositional bounds. Four central concerns exist in using the CALPHAD methodology/databases outside established compositions: (i) incomplete coverage of lower-order (binary, ternary) systems; (ii) lack of higher-order (quaternary and above) definitions; (iii) extrapolation, rather than interpolation, between calculations; and (iv) different thermodynamic descriptions for the same system across databases. These concerns are discussed here.
From Eq. (1), a complete thermodynamic description can only be achieved when the Gibbs energy contributions from all of the appropriate interaction terms are known. For example, the free energy description for a quaternary alloy requires a thermodynamic description for four elements, six binary systems, four ternary systems, and one quaternary system. Alloys with more elements (even as minor alloying additions) need even larger numbers of subcomponent systems. A six-component alloy requires thermodynamic descriptions for 15 binary and 20 ternary systems, and a superalloy with 10 elements needs 45 binary and 120 ternary thermodynamic descriptions. Yet, multicomponent databases usually do not contain all binary or ternary descriptions simply due to the sheer number of the subsystem descriptions needed. A complete thermodynamic database for 15 elements would require 105 binary alloy system and 455 ternary alloy system descriptions, in addition to higher-order definitions. This makes alloy exploration efforts difficult, as all of the constituent systems must be well defined within the same database to calculate the free energy of novel higher-component alloy systems. Furthermore, while thermodynamic descriptions are available for essentially all important metallic binary systems, far fewer ternary diagrams are available and relatively few have been reduced to thermodynamic models. Models for quaternary and higher-order systems are essentially absent in CALPHAD databases. There is some evidence that contributions of quaternary or higher-order interaction terms may have negligible influence on the free energy model.Reference Kattner9,Reference Chang, Chen, Zhang, Yan, Xie, Schmid-Fetzer and Oates16 Nevertheless, this remains a concern of uncertain impact.
A major value of the CALPHAD methodology is the ability to predict phases and phase relationships in composition and temperature regimes, where data are not explicitly available. This requires either interpolation or extrapolation from the thermodynamic models. Interpolation is usually more reliable, and small extrapolations to compositions and temperatures just outside the bounds used to construct databases might be used with some confidence. However, large extrapolations—such as those used when considering novel alloys—should be approached with caution. Nevertheless, such extrapolations are being considered to accelerate exploration of the central regions of complex phase space.Reference Senkov, Miller, Miracle and Woodward2,Reference Senkov, Miller, Miracle and Woodward17 Methods to improve the reliability of such extrapolations have been devised but their effectiveness has not yet been determined.
CALPHAD databases contain a large body of thermodynamic data not readily available elsewhere; however, energy model parameters are often optimized for the principal element highlighted by a database. For instance, aluminum is a key alloying element in titanium alloys but titanium is not a major alloying element for aluminum alloys. As a result, CALPHAD model parameters for the Al–Ti binary phase diagram in the titanium alloys database may emphasize accuracy at the titanium-rich end at the expense of accuracy in the aluminum-rich regime. The aluminum alloy-focused database may use slightly different model parameters for the Al–Ti binary system to better match aluminum-rich compositions. This tailoring, along with updating of existing descriptions as new data become available, can lead to differences in system descriptions between databases. These discrepancies add additional uncertainty when evaluating novel near-equimolar systems.
It has long been understood by the CALPHAD community that a prediction’s accuracy depends on the quantity and quality of the thermodynamic descriptions used to construct the database. However, an effort to quantitatively assess the uncertainty in predictions has yet to be completed, even within the composition and temperature bounds for which databases have been developed. This issue is particularly important for CALPHAD predictions of novel alloy spaces that are beyond the compositional bounds used to construct databases since this represents a major new opportunity for alloy discovery. The objective of the present work is to confirm which a priori factors are important to minimize risk when evaluating CALPHAD calculations in the context of novel composition exploration.
Creating a comprehensive catalog of validated experimental or theoretical data for comparison to the equilibria calculations would be an intractable task. Therefore, this manuscript focuses on evaluation of the CALPHAD algorithms as a function of the thermodynamic information already contained within the proprietary databases. Databases for metallurgical systems rarely contain thermodynamic data beyond ternary systems, and so ternary systems are the most complex considered here. The effect of binary descriptions on ternary predictions and the extent to which extrapolation from elemental and binary descriptions affects ternary prediction uncertainties will be evaluated. Within this article, “uncertainty” refers to the self-consistency and repeatability of predictions and not the degree to which a prediction reflects experimental data. While proprietary databases are used, the methods conceived and developed here are expected to have general applicability and the results are expected to show general trends.
II. METHODOLOGY
To evaluate the ability of commercially available thermodynamic databases to predict ternary phase diagrams in the composition space beyond the database limits, eight databases developed by CompuTherm LLC® were used: PanAl2013, PanCo2013, PanFe2013, PanMo2013, PanNb2013, PanNi2013, PanSol2013, and PanTi2013. CompuTherm Pandat™ 8.2 software was used to perform all the calculations. The number of elemental, binary, and ternary descriptions differs in each database, as does the total number of possible binary and ternary system definitions (Table I). PanSol differs fundamentally from the other databases in that only elemental and binary descriptions are included, and there is no emphasis placed on a particular region of the binary diagram when adjusting model parameters to reflect experimental data. Moreover, the fully assessed binary descriptions in PanSol are sparsely distributed across the database (e.g., of the 3003 binary combinations possible from the 78 elemental components, only 785 are defined).
Summary of unary, binary, and ternary thermodynamic descriptions contained in the Pandat 2013 databases and the subset of system descriptions considered in the present analysis.
Binary and ternary system descriptions in the CompuTherm databases are categorized as fully assessed or partially assessed. Fully assessed systems contain thermodynamic parameters describing all known phases for all compositions and temperatures. Partially assessed systems contain limited data—only the major phases across the relevant temperature range have been identified or select temperature and/or compositional ranges have been fully described. In either case, a free-energy model for the unary/binary/ternary/higher-order system has been explicitly included within the database. By contrast, systems not described within the database are extrapolated from lower-order constituent system definitions.
Beyond these categories, two additional metrics developed in previous publications were assigned to each system evaluated.Reference Senkov, Miller, Miracle and Woodward2,Reference Senkov, Miller, Miracle and Woodward17 The fraction of assessed binaries (FABs) and fraction of assessed ternaries (FATs) quantify the fraction of binary/ternary descriptions assessed within a database for a given alloy of interest. A ternary alloy has three constituent binary systems (three binary interaction terms in the free-energy model), so the possible FAB levels are 0.0 (0/3 binaries assessed), 0.33 (1/3 binaries assessed), 0.67 (2/3 binaries assessed), or 1.0 (3/3 binaries assessed). A FAB level of 0.0 implies that the ternary is predicted exclusively from elemental data. The FATs are defined in a similar way using ternary alloys as the basis. For example, a quaternary alloy has four possible ternary systems so the FAT assumes values of 0.0, 0.25, 0.50, 0.75, or 1.0 for databases that include full thermodynamic descriptions for 0, 1, 2, 3, or 4 of the constituent ternary systems, respectively. The same alloy can have different FAB or FAT values in different CALPHAD databases since different binary and ternary systems may be defined. Similarly, the same database can provide different FAB or FAT values when considering different alloy systems. A metric for the fraction of assessed unaries is not needed, as the CALPHAD calculation is not completed if an element does not exist within the database.
The databases used in the present work include a number of fully assessed ternary phase descriptions (Table I). For such definitions, FAB = 1.0 and FAT = 1.0. These ternaries are considered “reference systems” in the present study, as they have the most extensive thermodynamic descriptions and are assumed to be the most reliable ternary calculations available for the databases that contain them. This does not imply that these systems are 100% accurate relative to known experimental data. Rather, these ternaries have been validated from available experimental phase diagram data over the full compositional range and have not been extrapolated from lower component systems. Calculations within a given ternary system will have FAT = 1.0 when the database used for calculations contains the full thermodynamic description of the ternary system; calculations within the same ternary system, but in a different database, will have FAT = 0.0 if the ternary system is not assessed. No partially assessed ternary descriptions were considered in this analysis.
The present work compares equilibrium calculations for specific ternary compositions in databases that do not contain the ternary description (FAT = 0.0) with the results from databases where the full ternary description is available (FAT = 1.0). Comparisons are further made between calculations with databases that give different FAB values relative to the reference system calculations. In this way, the trends associated with using CALPHAD to predict unknown compositional space with limited/varying extent of known thermodynamic data can be explored.
For each fully assessed ternary system, line calculations were conducted in 10 °C increments from 500 to 1500 °C for 409 evenly spaced compositions throughout the ternary diagram (∼2.5 at.% compositional steps). These are referred to as FAT = 1.0 ternary calculations. Next, calculations were performed for the same alloy compositions in the same ternary systems but using databases that do not contain a full thermodynamic description of the ternary phase. These are referred to as FAT = 0.0 ternary calculations. A FAT of 0.0 implies that ternary calculations are based on binary and elemental thermodynamic data. Not every database could be used to calculate every reference system since not all databases contain descriptions for the three elements defining the ternary system. Additionally, any calculation predicting the solidus temperature below 500 °C was eliminated from consideration. Comparing FAT = 1.0 and FAT = 0.0 ternary system calculations produced approximately 29,000 unique compositions (78,850 distinct composition-database calculations) that quantified differences in CALPHAD predictions relative to a reference system. In instances where multiple FAT = 0.0 or FAT = 1.0 calculations were available for the same composition, all results were included in the final data set.
The solidus temperature (T solidus), first and last phase transformation temperatures, and phase information at T solidus and 1000 °C were recorded from the thermodynamic predictions. Seven comparison metrics, aligned with an alloy screening approach previously proposed,Reference Miracle, Miller, Senkov, Woodward, Uchic and Tiley1 were drawn from these data (Table I). The FAT = 0.0 calculations were evaluated against all seven criteria, where each criterion was classified as either “matching” or “not matching” the FAT = 1.0 prediction. For phase identification, text results were compared and assigned as “matching” if the output text matched (regardless of case sensitivity) or null if a different phase was predicted. Comparisons of integer values, such as the number of phases, had to be exactly equivalent to be considered “matching.” Temperature comparisons were classified as “matching” if the predictions fell within a predefined temperature range of ±150 °C.
The definitions above for FAB and FAT assume that thermodynamic data for the complete binary (FAB) or ternary (FAT) system are included in the database—no credit is given for partial thermodynamic assessments unless explicitly stated. This may degrade correlations between FAB and FAT terms and prediction credibility since useful (albeit partial) thermodynamic data are included in a database but are not reflected in the credibility scores. To address this bias, selected analyses in the present work evaluated the effects of including partially assessed binary descriptions. In these cases, partially assessed binary descriptions were rated the same as a fully assessed system, giving an opposite bias by inflating the FAB score. No distinction was made concerning whether the partial description covered the appropriate compositional or temperature range for a particular prediction.
Care was taken to track binary systems that were not assessed within a database but that show complete solid solubility in published binary phase diagrams.18 We propose that ternary free-energy models may be more accurately extrapolated from such binary systems, useful when considering novel, unexplored compositional space. This gives another potential bias by improving calculation credibility without increasing the FAB values for those calculations. These systems were tracked separately to confirm this hypothesis. For this subset analysis, the binaries were counted as fully assessed in the credibility calculations.
Predictions of microstructural stability near phase boundaries (PBs) are known to be sensitive to small deviations in the free energy calculations—the closer a composition is to a PB, the more likely it becomes that a small error incorrectly influences the calculation. As a result, a mismatch becomes more likely when comparing predictions from different databases for nodes that are compositionally close to a PB. To address this concern, analyses were conducted in two ways: (i) the first approach counted every compositional node explicitly as predicted, independent of proximity to PBs and (ii) the second approach eliminated compositional nodes near PBs from comparison. The term “PB sensitivity” is used to distinguish evaluations that disregard nodes neighboring PBs. PBs were identified for each ternary system–database combination by calculating the number of phases for each compositional node and its six nearest neighbors. Any node that did not have the same number of phases as the six neighboring nodes was treated as a PB and removed from consideration. The disregarded bands are approximately ±5–7 at.% wide, which matches inconsistencies observed when comparing multiple experimental phase diagrams of the same alloy system.18
When appropriate, contour plots were developed to examine the spatial relationship of the prediction quality as a function of compositional distance from known constituent subsystems. Symmetries exist when considering the “known” and “unknown” binary system combinations that may bound a ternary diagram. A ternary diagram with FAB = 1.0 displays three mirror symmetry planes; ternary diagrams with FAB = 0.67 have a single mirror plane that bisects the “unknown” binary; while ternary plots with FAB = 0.33 have a single mirror plan that bisects the “known” binary. These symmetries allow results to be compared relative to the known and unknown binary subsystems, regardless of the elements present in the binary and ternary systems. Likewise, generic contour plots were analyzed based on plotting symmetry about the PE of a given database, rather than evaluating contour plots of specific elements.
This methodology executed in this manuscript is naturally limited in scope. In order for a calculation to be included in the analysis, a full ternary system description must be contained within one of the CALPHAD databases and have, at a minimum, all three constituent elements defined within a second database. Only predominant commercially or academically important engineering alloys likely meet these conditions. Furthermore, the ternary systems are not evenly distributed across FAB credibility levels or databases (Table I). To maintain the statistical integrity as much as possible, certain analyses could not be conducted. For example, PanAl2013 only contained one FAT = 0.0, FAB = 1.0 ternary system that could be compared to a reference system from another database. Likewise, only one FAT = 0.0, a FAB = 0.0 system met the criteria for inclusion in the analysis. These cases, and other similarly sparse assessments, were not included in the manuscript.
III. RESULTS
By comparing the results of CALPHAD calculations where ternary phase space is extrapolated from binary descriptions to those derived when the ternary system descriptions are explicitly defined, we hope to identify those factors important for thermodynamic predictions of novel compositions. The following sections first provide a demonstrative example of the analysis described in the methodology, followed by an evaluation comparing fully assessed ternary descriptions across multiple databases. These results will provide a framework for further evaluation of ternary prediction capability as a function of three factors: (i) the number and quality of binary descriptions present for extrapolation, (ii) specific database variations, and (iii) relative distance from a defined subsystem. The interactions between these three primary factors will also be highlighted.
A. Demonstrative example of analysis
To establish a basis for results in Sec. III.B through III.F, an example analysis is first highlighted. Thermodynamic calculations for 66.7Al–10.4Ti–22.9V from the PanAl2013 and PanTi2013 databases are compared (Table II). In PanAl2013, the Al–Ti–V ternary system has FAT = 0.0 (the ternary is not described) and FAB = 0.67 (Al–Ti and Al–V binaries are fully assessed). The same Al–Ti–V system and all three of the binary subsystems are fully assessed in PanTi2013 resulting in FAT = 1.0 and FAB = 1.0. Thus the PanTi2013 results are the reference state, for the Al–Ti–V system. Equilibria predictions from PanAl2013 and PanTi2013 give matching results for Criteria 1 and 2 (number and identification of solid-solution phases at T solidus, Table II), as both databases predict one BCC_A2 solid-solution phase. However, the results do not agree when considering the remaining criteria. For instance, PanAl2013 predicts three phases at T solidus (a disordered BCC_A2 phase and two ordered phases), while PanTi2013 predicts only BCC_A2.
Comparison of results from Pandat calculations for 66.7Al–10.4Ti–22.9V (at.%) using PanAl2013 and PanTi2013. The results for Criteria 1 and 2 match, but none of the remaining criteria agree.
Poor overall agreement between the databases is observed for this composition. However, a single observation does not demonstrate the capacity of the PanAl database to accurately predict the entire Al–Ti–V system, supporting the approach in this paper of conducting analyses across all nodes of a ternary diagram. To further this illustration, an expanded comparison of the calculations for the entire Al–Ti–V alloy system using the PanAl2013 and PanTi2013 databases is shown in Fig. 1 for Criterion 3. The number of phases present and the location of multiphase regions are clearly different between the two calculations. Nevertheless, 76% of the PanAl calculations match the PanTi reference predictions for Criterion 3 [Fig. 1(c)]. As the prediction of a single alloy composition is insufficient to describe the accuracy of a particular database, in the same way the credibility of a database cannot be evaluated by considering a single ternary system. Hence, the analysis is expanded to consider all PanAl FAT = 0.0 ternary systems that have FAT = 1.0 in another database, regardless of the FAB level. This much broader evaluation of the PanAl database shows that 64% of the calculations agree with results from a FAT = 1.0 reference prediction for Criterion 3.
Total number of phases at T solidus (Criterion 3) calculated using (a) PanAl2013 and (b) PanTi2013. (c) The percentage of matching nodes is 76%; a node-by-node comparison of results in (a) and (b). The composition discussed in Sec. III.A (66.7Al–10.4Ti–22.9V) is circled on each of the plots.
Expanding the analysis to include all FAB = 1.0 calculations for ternary alloy systems with FAT = 0.0 in one database and FAT = 1.0 in at least one other database shows that 80% of the nodes match reference predictions when considering the number of equilibrium phases at T solidus (Criterion 3). Thus, ternary predictions based on thermodynamic data extrapolated from binary systems appear to be rather reliable for this single criterion. However, other criteria in this study may be more difficult to satisfy.
B. Phase boundary sensitivity assessment
Seven fully assessed ternary systems occur in more than one Pandat 2013 database: Al–Co–Ni, Co–Fe–Ni, Co–Ni–W, Cr–Fe–Ni, Fe–Mo–Ni, Fe–Ni–W, and Mo–Si–Ti. For reasons discussed in Sec. I, the thermodynamic descriptions may be slightly different for the same ternary system in different databases, introducing some uncertainty in defining a reference system when comparing with FAT = 0.0 systems. To assess the uncertainty introduced from these small differences, comparisons were made between predictions by different databases for a given, fully assessed ternary system (Table III). Comparisons involving the first three criteria—those describing the phases present at T solidus—are the most reliable and match in 95% of the calculations when considered individually. This drops to 91% when the first three criteria are cumulatively totaled (Criterion 1 match AND Criterion 2 match AND Criterion 3 match). The agreement is slightly lower for Criteria 4 and 5 (the number and type of solid-solution phases at 1000 °C) and drops to 90% for Criterion 6 (the total number of phases at 1000 °C). When considered cumulatively, there is 82% agreement for Criteria 1–6.
Fraction of matching nodes when comparing Pandat calculations across all compositions and databases. Includes comparisons for systems where FAT = 1.0 in more than one database (with and without PB sensitivity) and comparison of FAT = 0.0 calculations to FAT = 1.0 results as a function of FAB credibility.
A significant number of nonmatching calculations occurred near PBs, and agreement improved for all Criteria as PB sensitivity was considered (Table III). The number of solid-solution phases and solid-solution phase identification at T solidus matched perfectly, and the cumulative agreement for Criteria 1–6 increased to 91% when the nodes near PBs were disregarded from both databases. A significant number of calculations (44.5%) were removed from analysis when accounting for PB sensitivity. The location of these boundaries would not be known a priori when evaluating novel composition space; therefore, despite the importance of PB sensitivity when comparing calculations, PB sensitivity was not taken into account in the remaining analyses.
C. Effect of fraction of assessed binary systems on ternary predictions
On aggregate, increased FAB credibility improved the agreement between the FAT = 0.0 results and predictions from the FAT = 1.0 reference calculations when examining all compositions across all databases for each of the criteria assessed (Table III). This improvement from increased credibility trend is also maintained when examining the criteria cumulatively (Fig. 2). The likelihood of a given node matching all seven of the criteria identified in Table III was 60% when all of the constituent binaries were fully assessed in the same database (FAB = 1.0). This cumulative success rate drops under 40% when FAB = 0.67 and down to 12% when FAB = 0.33—emphasizing the importance of fully assessed binary descriptions for the free-energy model.
Cumulative agreement of FAT = 0.0 and reference (FAT = 1.0) calculations, when considering all compositions and all databases. Criteria are sorted in a manner to sustain the highest cumulative success rate, thereby providing an indication for which criterion are more successfully predicted.
Higher credibility does not necessarily guarantee accurate replication of a given alloy system phase diagram. For example, consider the accuracy of three different alloy systems: Al–Cu–Mg, Al–Cu–Fe, and Al–Fe–Mg. Each of these systems is rated FAT = 0.0, FAB = 1.0 within the PanFe2013 database, yet significant differences are observed between the system uncertainty when compared to reference calculations from PanAl2013 (Fig. 3). Accuracy of a given FAT = 0.0 system is highly dependent on the alloy system and database.
Comparisons between FAT = 0.0 and FAT = 1.0 predictions for select ternaries calculated using the PanFe2013 database. For all instances, the binary credibility is high (FAB = 1.0). The Al–Cu–Mg and Al–Cu–Fe systems have an average node match of 47–48% compared to the 90% match of Al–Fe–Mg.
Additional agreement between FAT = 0.0 and FAT = 1.0 reference systems is also observed when partially assessed descriptions are available for otherwise unassessed constituent binary systems. As expected, a partial description does not offer the same magnitude of increase in the number of matching nodes as a fully assessed description (Table III). For ternary calculations where FAB < 0.67, a partial description was found to average 75% as many matching nodes as a fully assessed description when considered cumulatively across all seven criteria evaluated in this analysis. Interestingly, for alloy calculations where FAB = 0.67, any description of the third binary (whether full or partial) results in similar levels of agreement between the FAT = 0.0 and FAT = 1.0 calculations.
“Crediting” soluble binary systems, when the subsystem is otherwise unassessed, improved the agreement between FAT = 0.0 and the FAT = 1.0 reference frames as much or more than when partially assessed binary descriptions were considered, an example of which is shown in Fig. 4. Presumably this is because there are no unique intermetallic phases that must be added to the system-wide free energy calculation, whereas partially assessed descriptions may miss binary phase descriptions.
Effect of binary credibility on agreement of FAT 5 0.0 and reference calculations for phase prediction (cumulative Criteria 1–6). Definitions for FAB credibility were evaluated using fully assessed binary descriptions with/without inflated credit for either partially assessed systems or unassessed solid-solution systems.
D. Individual database assessment
When comparing the results of FAT = 0.0, FAB = 1.0 calculations to those of reference systems per CALPHAD database, each of the databases performed relatively similarly (Fig. 5). The standard deviation of the “success rate” (fraction of FAT = 0.0 results that match the FAT = 1.0 reference calculation) between individual databases was under five percent for each of the individual criterion and a maximum of 7.6% when considering cumulative Criteria 1–7. Suggesting that on average, agreement between a FAT = 0.0 result and a reference calculation was more sensitive to binary credibility than to the database used for the calculation. No trend was observed between the accuracy of a given database and either (i) the FABs relative to the total possible per the database constituent list or (ii) the relative number of solid-solution binaries contained within a given database. The most significant difference between the databases was the ability to correctly predict intermetallic phase formation at T solidus and 1000 °C. Databases that predicted these criteria well (e.g., PanCo2013 and PanNi2013) were then noticeably more accurate for cumulative Criteria 1–7 as well. Despite the lack of ternary phase descriptions, PanSol2013 had near median accuracy when considering the success rate of all seven criteria cumulatively.
Evaluation of Pandat databases for each criterion considering only FAB = 5 1.0 calculations.
E. Ternary compositional space: Distance from known edge and other considerations
When a ternary system description is not explicitly included within the database, the free-energy model is extrapolated from the available constituent binary or unary descriptions [Eq. (1)]. It was confirmed in Sec. III.C that increasing the number of subsystem descriptions improved the model on average; here, we address the differences observed between nodes compositionally near versus compositionally far from assessed binary descriptions. To this end, a series of element-agnostic contour plots were constructed to visualize the spatial relationship—the symmetry of which is defined by the FAB credibility of the ternary system [Fig. 6(a)]. When considering FAB = 1.0 predictions across all databases [Fig. 6(b)], the nodes located the furthest “distance” from the known subsystem were less accurate than those compositionally closer to a binary, independent of the criteria examined. All of the nodes within the ternary space have the same FAB credibility, but differences were observed in the fraction of FAT = 0.0 results matching the FAT = 1.0 reference calculations. For cumulative Criteria 1–7, the results ranged from 38 to 94% matching the reference predictions. In other words, a three-component equimolar alloy extrapolated from fully assessed binary systems had less than 40% likelihood of matching all seven criteria when compared to the respective reference calculations. The compositional distance from a known subsystem, even when all subsystem descriptions are available and fully assessed, is a significant factor in the accuracy of a thermodynamic prediction. CALPHAD cannot predict novel phases from subsystem descriptions,Reference Kattner9–Reference Kaufman and Bernstein11 so a major consideration when extrapolating thermodynamic data into unknown composition space is the ability of subsystems to represent higher-order compositional space. To determine the fraction of ternary systems that have additional phases not observed in any of their constituent binary systems, the list of predicted phases were compared for each of the FAT = 1.0 reference systems and the three FAB = 1.0 binary descriptions within the same database. At T solidus, 23% of the ternary systems had at least one extra phase that was not present in any of the respective binary descriptions. This value dropped to 14% for the calculations recorded at 500 and 1000 °C. All additional phases were intermetallic compounds; any generic phase names such as “sigma” or “mu” were disregarded since the stoichiometry of the individual phases could not be confirmed across the bulk calculations. No ordered phase formation was observed for the five unique FAT = 1.0 ternary systems bounded by binaries consisting exclusively of solid-solution phases: Al–Si–Zn, Co–Fe–Ni, Hf–Nb–Ti, Mo–Nb–Ti, and Nb–Ti–Zr. The predictions for these five systems contained exclusively solid-solution phases across all of the nodes at all temperatures above 500 °C.
Composition-agnostic contour maps depicting spatial sensitivity of ternary system calculations relative to reference predictions (FAT = 1.0) for multiple criteria and FAB credibility.
F. Interactions between the three primary factors
The previous sections outlined the response of three factors that influence ternary CALPHAD predictions: (i) number and quality of binary system descriptions, (ii) individual database variations, and (iii) compositional distance from a known subsystem. Valuable trends can also be established by examining interplay between these factors, where possible.
The relationship between FAB credibility and the distance from a known binary was examined via the composition-agnostic contour plots [Fig. 6(c)]. As seen before, regions of compositional space closer to an assessed subsystem matched the reference results more often than regions further from known edges or corners. The spatial variation was particularly pronounced with lower FAB credibility [Fig. 6(c)]; higher FAB consistently resulted in higher success rates over a larger fraction of the ternary space. Similarly, comparison across multiple FAB levels of regions neighboring a known binary description indicated the penetration depth of these highly matching regions also increases with FAB credibility. The compositional locations of least agreement between the FAT = 0.0 and the reference calculations depended on the subsystem credibility level. For systems where all three binaries were known, the greatest mismatch occurred at the centroid of the ternary triangle; whereas when only two of the binaries were known, the area of more uncertainty shifted towards the unknown binary [Fig. 6(b)]. Corners where two unassessed binaries meet, such as with FAB = 0.33 calculations, were dominated by extrapolation of the elemental data. Interestingly, the response in corners of the ternary compositional space bounded by one known and one unknown binary description appeared to vary with FAB. Examination of a known–unknown corner for the cumulative Criteria 1–7 contour plots with FAB = 0.33 and FAB = 0.67 revealed the unary description has greater influence on the free-energy model of near-elemental compositions when system credibility is high.
Although the dataset used for this analysis did not have statistical integrity to identify a relationship between individual databases and credibility or compositional distance, conclusions could be drawn for PE-based versus nontailored databases and a second factor. Three categories were considered when comparing FAT = 0.0, FAB = 1.0 results with the FAT = 1.0 reference calculations: whether the composition (i) contained the PE namesake of the PE designed database, (ii) did not include the PE but was calculated in a PE-designed database, or (iii) was calculated in PanSol2013, a nontailored database (Fig. 7). Calculations that were conducted in tailored databases without the PE namesake as a component in the composition exhibited the expected response: the fraction of FAT = 0.0 calculations matching reference results was high near known subsystem descriptions but decayed symmetrically in the near-equimolar region. However, when the PE was contained within the composition, the region of high matching closest to the PE expanded at the expense of accuracy in other portions of the compositional range. The region of lowest matching was then observed opposite the PE corner, near the binary system which does not contain the PE. This trend was consistent across each of the individual criteria, and the difference in the minimum accuracy was nontrivial: for Criterion 3 alone (total phases at T solidus, Fig. 7), a 17% difference in the success rate was measured between the nodes with the lowest fraction of FAT = 0.0 results matching the reference. Calculations from the PanSol2013 database performed worse than a tailored database calculation without a PE component when all binary systems were fully assessed but did not experience the localized fluctuations observed when the PE component was included in the composition.
Example of spatial-sensitivity differences observed when considering calculations containing PE that matches database namesake (Criterion 3).
IV. DISCUSSION
CALPHAD simulations offer a unique capability for conducting extensive compositional exploration early in a multiprincipal component alloy discovery effort. CALPHAD thermodynamic databases are designed to predict phase equilibria within specific compositional ranges declared by the developer. Characterizing the uncertainties associated with extending this toolset beyond its conventional intent is necessary to minimize risks in subsequent, more costly development steps. The approach presented here begins to confirm and quantitatively define trends known by the CALPHAD community: namely, that the quality of a ternary system free-energy model is highly dependent on the completeness of the thermodynamic description of the respective binary system number and that results are particularly sensitive near PBs. The discussion will highlight those conditions which influence the relative accuracy of CALPHAD calculations within the same software framework, under the context of informed screening for novel alloy development. As mentioned previously, the conclusions drawn here are not anticipated to be specific to the proprietary thermodynamic databases used in this analysis but rather indicative of the CALPHAD methodology. Variations between commercial and/or free-source databases are to be expected—the rigor by which a given database has been developed and the extent by which it has been validated will depend greatly on the practices of individual groups. The results of this work should not be considered as a judgment of current databases but rather as the first step towards understanding how far CALPHAD calculations can be extended beyond the compositional range limits of the current databases.
One challenge in using CALPHAD for ternary system phase prediction is the uncertainty that exists even in the “best case” simulations where the exact ternary description is available and fully assessed. Differences were observed in the phase predictions between the same FAT = 1.0 ternary system but different databases, even when the nodes along PBs were disregarded: only 91% of the calculations agreed for the six nontemperature criteria. There are two possible explanations for the discrepancies: (i) out-of-date descriptions and/or (ii) selective optimization of the free-energy models in PE-designed databases. When a database is constructed, the best thermodynamic descriptions available at the time are used to create the free-energy models. It takes a monumental effort to update old descriptions when new experimental or computational results are available, and implementation of new data into contemporary databases can sometimes be staggered between software releases. Tailoring of the free-energy model parameters in PE database was also shown to adversely affect the prediction capability of some nodes in fully assessed FAT = 1.0 systems. The extent of these effects is not known a priori by the end-user and this uncertainty exists in addition to the absolute error that occurs between a prediction and its corresponding experimental/atomistic observations—an error which cannot easily be systematically defined. Only through comparison of the same fully assessed descriptions across databases were the contradictions apparent; stressing the importance of having internally consistent, updated databases for all CALPHAD calculations. Given that the database discrepancies cannot be reduced without significant effort from the community, the attention then focuses on how best to minimize the relative risk of individual composition calculations when considering expansive calculations across multiple databases in the same software version.
High subsystem credibility was the most vital factor for reducing risk in CALPHAD calculations. Each fully assessed binary definition increased the fraction of ternary system nodes matching the reference results by approximately 20% on average for cumulative Criteria 1–7 (Table III). The credibility metric holds even when a full description for one of the constituent subsystems is not known. Partially assessed definitions increased the average success rate of ternary calculations matching the FAT = 1.0 reference results but not to the same extent. Partial descriptions added 75% of the accuracy of a fully assessed description when considering all seven criteria cumulatively; therefore, each partially assessed binary definition could reasonably be given a fractional credibility of 0.75 × 0.33 = 0.25 for this dataset. Since binary systems that contained exclusively solid-solution phases had similar improvement in average accuracy and can be identified prior to conducting CALPHAD calculations, the binary definition credibility could also be artificially inflated to a similar extent when no other definitions for the subsystem exist.
The other major consideration when assessing the relative uncertainty of a given ternary alloy calculation is the compositional distance from an assessed subsystem. CALPHAD simulations for near-equimolar compositions contained more risk than a discrete composition “closer” to a known subsystem, leading to an additional challenge when exploring multiprincipal alloy space. Although the fraction of FAB = 1.0, FAT = 0.0 calculations matching the reference results (FAB = 1.0, FAT = 1.0) was 60% when considering the average response of all nodes across the seven criteria; the success rate varied from 38 to 90% when accounting for spatial position. Furthermore, this spatial sensitivity is dependent on credibility, with overall lower accuracy and expanded regions of poor productive capability for systems with lower FAB levels. Databases designed for PE alloy systems were found to have a modest influence on the symmetry of these low accuracy domains when the PE was a component in the composition-of-interest. Therefore, for nodes in the primary component-rich region of the phase space, preference should be given to calculations conducted in the namesake database. Regions where the PE is a minor constituent (<25 at.%) other databases should be favored.
Specific database selection was not as significant as the either subsystem credibility or spatial location when considering ternary CALPHAD predictions with full binary credibility. The most noticeable difference between well-developed databases was in the capability to correctly predict the total number of phases at T solidus (Criterion 3) and at 1000 °C (Criterion 6). Discrepancies between individual databases and the intermetallic phase descriptions that they contain may arise from the relative complexity of phase diagrams in specific compositional domains. Insight into these differences could likely be gleaned from conducting a statistical analysis relating the nature of phases present in ternary systems to those contained in their respective binary systems. A thorough investigation examining the compositional trends across the family tree of constitutive elements, binaries, and ternaries may provide additional insight into metallurgical features that could be captured within the CALPHAD databases to improve the free-energy models in novel alloy space. Meanwhile, when comparing CALPHAD results across databases, calculations with higher subsystem credibility should be given priority.
The factors examined in this manuscript are insufficient to create a rigorous empirical model for estimating the error of a specific ternary calculation. However, a series of guidelines can be established to maximize the likelihood of the correctly predicting ternary phase space when the ternary description is not explicitly defined. Unsurprisingly, to fully minimize the risk in novel three-component equilibria CALPHAD calculations, all binary descriptions should be present, fully assessed, and contain all suspected phases. For compositions where this pedigree is not possible, the following guidelines provide a 50% likelihood of predicting the number of total phases at solidus and 1000 °C, the solid-solution phase structures, and the ternary solidus temperature within ±150 °C (all seven criteria cumulatively) for a novel composition:
(i) Two of the three binary descriptions are either fully or partially assessed (FAB > 0.5) and
(ii) The discrete composition falls within 25 at.% of an assessed binary.
These guidelines provide general guidance for risk reduction when employing the CALPHAD method for ternary exploration but do not address the challenges associated with expansion into multicomponent alloys. It is likely that the credibility for an n-component system will differ from the minimum credibility recommended here for ternary compositions; however, it is unclear what level will be required to reach acceptable risk. A similar analysis to understand the ability to predict multicomponent alloys is limited by the absence of well-defined higher-order descriptions to act as suitable reference systems. As CALPHAD lacks the ability to predict the formation of novel phases, a thermodynamic model must be generated for any new phase not contained within any of the constituent subsystems. The results discussed in Sec. III.E indicate 14% of the ternary systems assessed in this work possess intermetallic phases that are not captured by the relevant binary isotherms at 1000 °C. These systems contain phases that are not predicted when the three-component phase space is extrapolated exclusively from the constituent binary descriptions. It has been speculated that the free energy contributions from systems more compositionally complex than ternaries are relatively weakReference Senkov, Senkova and Woodward13,Reference Senkov, Senkova, Woodward and Miracle14 and that very few unique phases are expected beyond those represented in ternary descriptions.Reference Mackay19 This suggests that it is probable that quaternary and higher systems will not require full-assessment of the four-component phase space to provide valid predictions (unlike with ternaries). More likely, a high fraction of binary and ternary descriptions will be sufficient; however, the number of relevant subsystems increases significantly with each added component. An n-component system will have 
$C_n^2$ constituent binaries and 
$C_n^3$ constituent ternaries [Eq. (2)].
$$C_n^i = {{n!} \over {i!\left( {n - i} \right)!}}\quad .$$In the absence of suitable experimental data or assessed CALPHAD systems for higher-order systems, density functional theory could be utilized to develop a full thermal and compositional description of quaternaries, although at significant computational expense. For multiprincipal component investigations, the development of one global database would be ideal. However, the magnitude of this endeavor makes this goal unlikely using conventional database development techniques. The number of binary and ternary definitions required increases dramatically with the number of elemental constituents contained within a database. A more realistic approach for high-fidelity multiprincipal element alloy development using CALPHAD is to limit the number of components under investigation and create a well-assessed database.Reference Zhang, Zhang, Chen, Zhu, Cao and Kattner8 Various non-CALPHAD–based physical and phenomenological criteria have been proposed to assist in palette selection and prescreening.Reference Miracle, Miller, Senkov, Woodward, Uchic and Tiley1,Reference Troparevsky, Morris, Kent, Lupini and Stocks6,Reference Senkov and Miracle20,Reference Poletti and Battezzati21 There is already precedence for this approach—both CompuTherm LLC and Thermo-Calc Software® have recently released high entropy alloy specific databases with complete binary descriptions (100% of the possible binaries are described within the database) and a large number of ternary descriptions.22,23 Ongoing efforts by the community to reduce the verification and implementation time of new equilibria data are also helpful.
Although there are limitations to using CALPHAD to predict multiprincipal component systems, it remains one of the best methods to quickly and efficiently explore the equilibrium conditions for a large number of complex alloy systems. Even when considering the least accurate compositional point at the equimolar composition for all FAB = 1.0 alloy systems, CALPHAD has been shown to have a 38% chance of matching all seven criteria. Compared to an uninformed bilateral choice, the chance of correctly predicting the same seven metrics cumulatively would be 1/(27) = 0.7%, highlighting the risk reduction that can be obtained by using CALPHAD in the early stages of novel alloy exploration. Practitioners should understand the limitations that exist in today’s tools and set rational thresholds when evaluating each of the criteria when screening or developing new alloys, using the anticipated uncertainty to intelligently guide computational and experimental resources in subsequent alloy development efforts.
V. CONCLUSION
The importance of strong, credible databases for CALPHAD calculations has been demonstrated through comparison of ternary system descriptions across multiple thermodynamic databases. In total, the analysis accounted for more than 29,000 unique compositional evaluations for seven predicted thermodynamic criteria. In addition to quantitatively confirming many known trends regarding uncertainty within the CALPHAD methodology, a few new findings have become apparent:
(1) Use of the CALPHAD methodology for novel alloy exploration is a viable technique, particularly considering the ease by which large numbers of discrete novel compositions can be evaluated. Preliminary metrics have been established by which simulations processed using the same software may be compared. This information can be used to guide subsequent experimental or computational discovery efforts.
(2) When assessing novel compositional space, both the credibility of the constituent subsystem descriptions and the compositional “distance” from an assessed subsystem are important metrics for quantifying the relative success of a given prediction.
(3) On average, partial subsystem descriptions and undescribed continuous solid-solution subsystems add approximately 75% capability of a fully assessed binary description and may be included to expand element CALPHAD database coverage during novel alloy exploration.
(4) Databases designed for PE alloy systems predict accurately near the PE-of-interest at the expense of compositional regions containing the PE as a minor constituent.
(5) Of the 72 unique ternary systems explored in this analysis, 14% contained phases specific to the three-component system that were not observed in the constituent binary systems. Extrapolation of this finding to higher-order multicomponent systems was not possible in this analysis.
ACKNOWLEDGMENTS
The authors would like to express their gratitude to Dan Miracle and Chris Woodward of AFRL and Fan Zhang of CompuTherm, LLC for the thorough discussions regarding multiple aspects of this effort. The authors would also like to thank Christopher Halley of Breakthrough Performance Group, LLC for preliminary statistical evaluations that served to guide the analyses and results presented in this paper. ONS acknowledges the financial support through the Air Force on-site contract FA8650-15-D-5230 managed by UES, Inc., Dayton, OH.
