Data limitations and the use of features/descriptors

One of the historic limitations to applying ML to the materials science domain has been obtaining data sets large and diverse enough to robustly train and validate ML models. Even the Materials Project database is not large enough to use “big data” approaches that include tens of millions of training data points. One way to improve the performance of ML for smaller data sets typically encountered in materials science is to extract features (or descriptors) from the input data. In this context, a set of features (or descriptors) refers to data derived from the original raw inputs that encompass domain knowledge and is expected to simplify the ML problem (Figure 1). Depending on the complexity of the problem and the scarcity of available data, the choice of appropriate descriptors can have a large impact on final performance of a ML model.^{Reference Young, Yuan and Zhu13}

Figure 1.

Example of machine-learning (ML) workflow, including the generation of descriptors. Each raw input is transformed into a set of descriptors. Here, all raw inputs are chemical compositions; they might instead be a crystal structure, experimental spectrum, or microstructural image. The descriptors represent a set of numbers that encapsulate the raw inputs in a compact and physically meaningful way. For example, one descriptor of a composition might be the variation in electronegativity of its component elements, which would indicate the degree to which the compound might be metallic, covalent, or ionic. The set of descriptors (sometimes in conjunction with the raw inputs) and known outputs are then used to efficiently train the ML model.

Early efforts at feature extraction in the materials science domain were concentrated on the chemical composition. The raw string format of a composition (e.g., “LiFePO₄”) is not amenable to efficient learning because machine-learning algorithms are unaware that this text represents a chemical composition with specific physical characteristics. Studies have instead employed numerical descriptors that represent physical aspects of the composition such as sums of covalent radii, differences in electronegativity, or the average melting point of the component elements in the composition.^{Reference Seko, Togo and Tanaka14,Reference Ward, Agrawal, Choudhary and Wolverton15} This descriptor generation has been a crucial preprocessing step to machine learning for a wide array of properties, including melting points,^{Reference Seko, Maekawa, Tsuda and Tanaka16} thermoelectric figures of merit,^{Reference Carrete, Mingo, Wang and Curtarolo17} thermal conductivity,^{Reference Carrete, Li, Mingo, Wang and Curtarolo9} solute diffusion barriers in face-centered-cubic metals,^{Reference Wu, Lorenson, Anderson, Witteman, Wu, Meredig and Morgan18} and elastic properties,^{Reference de Jong, Chen, Notestine, Persson, Ceder, Jain, Asta and Gamst19} to name a few.

One shortcoming of compositional descriptors is the inability to distinguish between entries sharing a chemical formula but possessing different crystal structures (e.g., diamond from graphite or carbon nanotubes). Compositional descriptors alone have had the most success in problems where the crystal structure has been held constant or was the intended variable to predict.^{Reference Legrain, Carrete, van Roekeghem, Curtarolo and Mingo20,Reference Gaultois, Oliynyk, Mar, Sparks, Mulholland and Meredig21} One active research topic is how to best extract descriptors from a crystal structure which should be invariant to symmetry operations of the crystal. For example, translating all the atoms, rotating the entire crystal, changing the “order” in which atoms are listed, and creating a supercell are all operations that should not change the value of a well-formed crystal structure descriptor. Recently, researchers have devised many such formulations,^{Reference Isayev, Oses, Curtarolo and Tropsha22–Reference Pham, Kino, Terakura, Miyake, Takigawa, Tsuda and Dam29} including several variants of a descriptor for the complex network of coulomb interactions within a structure. Faber et al. recently tested this using a data set of 3938 formation energies retrieved from the Materials Project database.^{Reference Faber, Lindmaa, von Lilienfeld and Armiento28}

It is also possible to generate features by mixing together combinations of the original features according to various equations,^{Reference Ghiringhelli, Vybiral, Levchenko, Draxl and Scheffler30,Reference Ramprasad, Batra, Pilania, Mannodi-Kanakkithodi and Kim31} (e.g., as the sum of two features divided by a third). Although it is possible to generate many thousands of features or more, selecting the most relevant compound features is still an area of active research. The presence of redundant or uninformative features can reduce the overall performance of machine learning, making it prone to overfitting (i.e., reproducing patterns present in the known data that will not accurately reflect new data). Various feature selection methods devised by the ML community^{Reference Chandrashekar and Sahin32} and certain ML algorithms such as kernel ridge regression employ regularization parameters that can be used to penalize the use of too many features.

Although most descriptors are constructed manually using domain knowledge or some mathematical basis, it is also possible to automatically construct descriptors with sufficient data. Representation learning^{Reference Chandrashekar and Sahin32} tackles the issue of how to best represent raw ML inputs. One approach is an autoencoder, which is a neural network architecture designed to transform the original input into a reduced set (or a latent representation) that can faithfully be transformed back into the full input. This method can be used to generate representations on a large unlabeled data set and transfer this representation model to a smaller but similar data set. For example, Google trained an autoencoder model on 10 million image thumbnails to uncover features that corresponded to “face descriptors” or “cat descriptors” from unlabeled data.^{Reference Le33} These automatically generated features, which did not require domain knowledge to devise, were used to boost the performance of an image classifier by more than 70% over the existing state-of-the-art models on the well-known Imagenet^{Reference Deng, Dong, Socher, Li, Li and Fei-Fei34} data set. This type of approach is termed “transfer learning” and variants of this approach have recently been applied to the materials science domain as well.^{Reference Hutchinson, Antono, Gibbons, Paradiso, Ling and Meredig35} Such research hints that it might be increasingly possible in the future to “auto-learn” descriptors given large and sufficiently diverse data sets.

ML and accelerated discovery

An attractive feature of ML models is their speed, especially compared to obtaining new materials property data using either experiments or computations. In this section, we highlight several examples of applying ML models to computed materials data. As an example, phase stability presents a ubiquitous property of interest in materials science. Faber and co-workers^{Reference Faber, Lindmaa, Von Lilienfeld and Armiento36} applied kernel ridge regression to predict the formation energies of two million elpasolite (ABC₂D₆) structures with a mean average error of approximately 100 meV/atom. Considering that approximately 90% of the crystals in the ICSD have energy above the hull (E _hull ) < 70 meV/atom,^{Reference Sun, Dacek, Ong, Hautier, Jain, Richards, Gamst, Persson and Ceder37} and the errors of density functional theory (DFT)-calculated formation energies of ternary oxides from binary oxides relative to experiments are ∼24 meV/atom,^{Reference Hautier, Ong, Jain, Moore and Ceder38} there is still significant room for improvement in fit-to-DFT stability predictions. The choice of the target metric in particular may not be the most appropriate for ML applications. Both, E _hull and the formation energy (E _f) (referenced to elemental compounds or binary oxides) are not ideal learning targets due to the variation in reference states for each species across different structures, which introduces additional DFT errors that can bias the learning and discovery process.^{Reference Wang, Maxisch and Ceder39} Faber et al. identified 128 new expected stable elpasolite structures, 90 of which were unique stoichiometries and could not have been identified in the same timeframe using direct DFT computation, despite the lack of accuracy.

Bandgap is another important material property commonly estimated via first-principles calculations. The 54,000 band structures in the Materials Project are calculated by DFT with the generalized gradient approximation (GGA), which underestimates the bandgap due to the approximation in exchange-correlation functionals, the self-interaction error, and the missing derivative discontinuity. Common solutions to this problem rely on more accurate, but more computationally expensive, methods such as the modified Becke–Johnson (mBJ) functional,^{Reference Tran and Blaha40} the delta self-consistent-field (∆SCF) method,^{Reference Chan and Ceder41} hybrid functionals (HSE06)^{Reference Heyd, Scuseria and Ernzerhof42} or GW calculations based on many-body perturbation theory.^{Reference Fuchs, Furthmüller, Bechstedt, Shishkin and Kresse43} Lee and co-workers,^{Reference Lee, Seko, Shitara, Nakayama and Tanaka44} attempted to circumvent this by learning from the cheaper PBE bandgap (E _g [PBE]) or mBJ bandgap (E _g [mBJ]), with an additional 18 descriptors to predict the more accurate G₀W₀ bandgap (E _g (G₀W₀)). The best model trained by nonlinear support vector regression (SVR) yielded a root-mean-square (rms) error of 0.24 eV (Figure 2).

Figure 2.

Comparison of the E _g (G₀W₀) method and E _g (predicted). (a, c) The ordinary least-squares regression model with E _g (PBE) as a single predictor. (b, d) The support vector regression model with the 18-predictor set. A randomly chosen result from 100 trials is shown. Reprinted with permission from Reference Lee, Seko, Shitara, Nakayama and TanakaReference 44. © 2016 IOP Publishing.

Another important set of materials properties is mechanical behavior. The elastic bulk and shear moduli (K and G, respectively) govern the stress–strain relations of isotropic materials, and are highly correlated to properties such as hardness^{Reference Pugh45} and thermal conductivity.^{Reference Snyder and Toberer46} Recently, 1940 computed elastic constants from the Materials Project were utilized as a training set for a gradient boosting machine local polynomial regression (GBM-Locfit) model to predict K and G for k-nary inorganic polycrystalline compounds by de Jong.^{Reference de Jong, Chen, Notestine, Persson, Ceder, Jain, Asta and Gamst19} After careful optimization of hyper-parameters through tenfold cross-validation, the rms error of K and G were 0.0750 and 0.1378 (log(GPa)), respectively (Figure 3). The authors found that for both K and G, the top four most important descriptors were V (volume per atom), R _n (elemental row number), E _c (cohesive energy), and X (mean of elemental electronegativity). The model was therefore used to screen for hard materials among 30,000 compounds from the Materials Project, with Mg(B₆C)₂, Sc₂CrB₆, and Mg₂B₂₄C discovered as promising candidates.^{Reference de Jong, Chen, Notestine, Persson, Ceder, Jain, Asta and Gamst19} This work is a demonstration of how data from the Materials Project can be directly used to extract useful information and inspire new high-throughput screening.

Figure 3.

Comparison of density functional theory (DFT) training data with GBM-Locfit predictions for K (a) and G (b). The training set consists of 65 unary, 1091 binary, 776 ternary, and eight quaternary compounds.^{Reference de Jong, Chen, Notestine, Persson, Ceder, Jain, Asta and Gamst19}

Besides predicting basic materials properties, ML can also assist in developing an understanding of the fundamental physics of crystals by constructing force fields. Force-field development allows for the simulation of systems that contain thousands of atoms and over time scales of nanoseconds to microseconds. Chen and co-workers^{Reference Chen, Deng, Tran, Tang, Chu and Ong47} applied ML techniques to build a spectral neighbor analysis potential (SNAP) model for Mo, which outperforms existing Mo potentials in accuracy over energies, stress tensors, elastic properties, melting point, and surface and grain-boundary energies using the data and software tools developed by the Materials Project. The data set was further screened by a principal component analysis, which ensures distinct local environments among data. A differential evolution algorithm was employed to simultaneously optimize the weights of the ML model and spectrum parameters used to calculate features. This work, besides presenting a superior force field for Mo, showcases inspiring strategies in data preparation and global optimization.

It is also worth noting that the fruit of ML can in return empower the Materials Project. Simultaneously introduced with XASdb, a database containing ∼800,000 K-edge XANES spectra, a novel ensemble-learned spectrum identification algorithm was proposed to enable instant matching between experimental XAS spectra and the references in XASdb.^{Reference Zheng, Mathew, Chen, Chen, Tang, Dozier, Kas, Vila, Rehr, Piper, Persson and Ong48} The algorithm classifies an input spectrum according to chemistry and oxidation state, using a group of 33 weakly correlated algorithms in an ensemble akin to a random forest, to span the whole space of materials chemistry and oxidation states available in the Materials Project. This divide-and-conquer approach correctly identified the oxidation state and coordination environment with 84.2% accuracy in a test of 19 high-quality experimental spectra and is currently implemented in the Materials Project for rapid classification of measured XAS spectra. The combination of the database and the designed algorithm significantly accelerates materials characterization using XAS and XANES spectra, and represents a powerful addition to the Materials Project.

Finally, a recent development is the application of deep learning to materials science. Xie and coworkers^{Reference Xie and Grossman49} developed convolutional neural networks (CNN) to learn from the “crystal graph,” which represents the crystal structure using graph nodes and edges that represent atoms and bonds. A Materials Project data set containing 46,744 materials, covering 87 elements, seven lattice systems, and 216 space groups, with seven properties (DFT calculated formation energy, absolute energy, bandgap, Fermi energy, bulk moduli, shear moduli, and the Poisson’s ratio) was used for training. The results show that with careful design of crystal representation, CNN models can be generalized to predict a wide range of properties with a level of accuracy comparable to DFT. The effort of generalizing both the structure and target space was greatly emphasized in this work and should be acknowledged. Another merit in the developed CNN model is the interpretability, meaning insights can be drawn from analyzing the contribution to the target from different local environments of atoms, which were encoded in the input vectors.

Outlook

The discussed examples showcase the power of combining robust and diverse materials data with ML techniques. Besides directly feeding existing data or initial structures for model training, the Materials Project database and its related infrastructures for structural analysis^{Reference Ong, Richards, Jain, Hautier, Kocher, Cholia, Gunter, Chevrier, Persson and Ceder12} and high-throughput calculations^{Reference Jain, Ong, Chen, Medasani, Qu, Kocher, Brafman, Petretto, Rignanese, Hautier, Gunter and Persson50} are also enabling efficient generation of new data^{Reference Mathew, Montoya, Faghaninia, Dwaraknath, Aykol, Tang, Chu, Smidt, Bocklund, Horton, Dagdelen, Wood, Liu, Neaton, Ong, Persson and Jain51} for specific learning tasks. We emphasize that while increasing accuracy is always a goal of any ML study, efforts should focus on increasing the scale, diversity and quality of data. For example, while the few hundred data points, as used in Lee’s work^{Reference Lee, Seko, Shitara, Nakayama and Tanaka44} on bandgap prediction, as well as Medasani’s work^{Reference Medasani, Gamst, Ding, Chen, Persson, Asta, Canning and Haranczyk52} on classifying defect types in B2 intermetallics were used to extract trends, such sets are insufficient for the automatic training of robust ML models. Furthermore, most models are only applicable within constrained structure space,^{Reference Faber, Lindmaa, Von Lilienfeld and Armiento36,Reference Schmidt, Shi, Borlido, Chen, Botti and Marques53,Reference Pilania, Mannodi-Kanakkithodi, Uberuaga, Ramprasad, Gubernatis and Lookman54} but a truly robust model across most structure types is still awaited. Hence, until enough consistent and reliable materials data is available across structure and chemical spaces, the key may lie in the development of better compact yet descriptive structure descriptors.

The forward approach adopted by the Materials Project is to support both avenues, by (1) continuously generating and making available ever-growing data sets of fundamental materials properties across diverse chemistries and structures, and (2) developing algorithms to generate descriptor functions that can be used in ML methods for smaller data sets. The Materials Project has been continually releasing new types of large data sets that can serve as the basis for future machine learning studies.^{Reference de Jong, Chen, Geerlings, Asta and Persson7,Reference de Jong, Chen, Angsten, Jain, Notestine, Gamst, Sluiter, Ande, van Der Zwaag, Plata, Toher, Curtarolo, Ceder, Persson and Asta8,Reference Petousis, Mrdjenovich, Ballouz, Liu, Winston, Chen, Graf, Schladt, Persson and Prinz55,Reference Petretto, Dwaraknath, Miranda, Winston, Giantomassi, Van Setten, Gonze, Persson, Hautier and Rignanese56} Further, the Materials Project is collaborating on the development of the matminer (www.github.com/hackingmaterials/matminer) software platform, which provides open-source implementations in the Python programming language for a variety of descriptors of all types that have been proposed in the literature. For example, the Materials Project has recently added order parameter functions that are highly sensitive to the local environment (e.g., a “tetrahedral” versus “square planar” arrangement of atoms around a central site^{Reference Zimmermann, Horton, Jain and Haranczyk57}), and can be used to assess structure similarity or improve structure prediction performance. Our hope is that an abundant user base, a large well-curated database of properties and descriptors, combined with a community that is enthusiastic to employ machine learning in materials science will foster the next generation of advances in structure–property relations and materials discovery at an ever-increasing pace.