Introduction
Chemistry and materials science are entering a new data-driven age,[1–3] in which planning algorithms select experiments to be conducted by humans or performed autonomously using laboratory robotics.[Reference Henson, Gromski and Cronin4–Reference Correa-Baena, Hippalgaonkar, van Duren, Jaffer, Chandrasekhar, Stevanovic, Wadia, Guha and Buonassisi6] Laboratory automation has been an ongoing endeavor for nearly a quarter century with seminal demonstrations of high-throughput materials research performed by Xiang et al. in 1995.[Reference Xiang, Sun, Briceño, Lou, Wang, Chang, Wallace-Freedman, Chen and Schultz7] Subsequent research, predominantly in combinatorial chemistry, focused on the development of high-throughput techniques targeting new material syntheses[Reference Schultz and Xiang8] and methods of characterization[Reference Koinuma and Takeuchi9] that have been the topic of several comprehensive reviews.[Reference Takeuchi, van Dover and Koinuma10–Reference Maier, Stöwe and Sieg17] Recent advances in machine learning and artificial intelligence allow for extracting further physical insights latent within these results.[Reference Butler, Davies, Cartwright, Isayev and Walsh18,Reference Sanchez-Lengeling and Aspuru-Guzik19] Important themes include comprehensive capture and analysis of “successful” and “failed” experiments,[Reference Raccuglia, Elbert, Adler, Falk, Wenny, Mollo, Zeller, Friedler, Schrier and Norquist20–Reference Xu, Olshansky, Adler, Huang, Smith, Zeller, Schrier and Norquist23] adaptively modifying the experiment plans as data are collected,[Reference Henson, Gromski and Cronin4,Reference Duros, Grizou, Xuan, Hosni, Long, Miras and Cronin24–Reference Celse, Rebours, Gay, Coste, Bourgeois, Zammit and Lebacque29] machine-learned characterization of experimental outcomes,[Reference Bai, Xue, Bjorck, Le Bras, Rappazzo, Bernstein, Suram, Van Dover, Gregoire and Gomes30] ensuring sufficient sampling of relevant experimental variables by avoiding human biases,[Reference Duros, Grizou, Xuan, Hosni, Long, Miras and Cronin24,Reference Cao, Adutwum, Oliynyk, Luber, Olsen, Mar and Buriak31] and integration with chemical informatics features[Reference Stanev, Oses, Kusne, Rodriguez, Paglione, Curtarolo and Takeuchi32] and physical simulation outputs as machine learning model inputs.[Reference Yan, Yu, Suram, Zhou, Shinde, Newhouse, Chen, Li, Persson, Gregoire and Neaton33–Reference Shinde, Suram, Yan, Zhou, Singh, Yu, Persson, Neaton and Gregoire35]
The types of chemistry, experimental processes, level of automation, and throughput of modern materials research vary tremendously. Representative examples of the current state-of-the-art range from real-time control of single batches of carbon nanotubes[Reference Nikolaev, Hooper, Webber, Rao, Decker, Krein, Poleski, Barto and Maruyama27] and continuous flow organic synthesis,[Reference Bédard, Adamo, Aroh, Russell, Bedermann, Torosian, Yue, Jensen and Jamison26] to tens of hydrothermal syntheses performed with the help of humans,[Reference Raccuglia, Elbert, Adler, Falk, Wenny, Mollo, Zeller, Friedler, Schrier and Norquist20,Reference Xu, Olshansky, Adler, Huang, Smith, Zeller, Schrier and Norquist23,Reference Duros, Grizou, Xuan, Hosni, Long, Miras and Cronin24] to hundreds or thousands of reactions performed in microwell plates[Reference Ahneman, Estrada, Lin, Dreher and Doyle21] and in droplets,[Reference Lin, Dikler, Blincoe, Ferguson, Sheridan, Peng, Conway, Zawatzky, Wang, Cernak, Davies, DiRocco, Sheng, Welch and Dreher22,Reference Zhou, Li and Zare25] up to national-level synchrotron facilities.[Reference Green, Choi, Hattrick-Simpers, Joshi, Takeuchi, Barron, Campo, Chiang, Empedocles, Gregoire, Kusne, Martin, Mehta, Persson, Trautt, Van Duren and Zakutayev36]
The software implemented to generate experiments and collect data for machine learning are similarly diverse. Software packages for materials chemistry have focused on closed-loop operation with specific experiments, robotic hardware, and algorithms, such as ARES to study single-walled carbon nanotubes,[Reference Nikolaev, Hooper, Webber, Rao, Decker, Krein, Poleski, Barto and Maruyama27] NREL-HTEM for physical vapor deposition of inorganic thin films,[Reference Zakutayev, Wunder, Schwarting, Perkins, White, Munch, Tumas and Phillips37] AIR-Chem targeting inorganic perovskite quantum dots,[Reference Li, Lu, Xu, Liu, Tu, Ye, Liu, Xie, Qian and Zhu38] as well as a number of solutions for polymer chemistry.[Reference Adams and Schubert39,Reference Adams and Schubert40] Like these, ChemOS, a recently released modular software environment for autonomous laboratory operation, focuses primarily on optimization rather than comprehensive data capture.[Reference Roch, Häse, Kreisbeck, Tamayo-Mendoza, Yunker, Hein and Aspuru-Guzik41] (We note in passing software libraries for managing closed-loop computational organic,[Reference Hachmann, Afzal, Haghighatlari and Pal42] materials,[Reference Potyrailo, Rajan, Stoewe, Takeuchi, Chisholm and Lam15,Reference Baumes, Jimenez and Corma43] and heterogeneous catalyst[Reference Tran, Palizhati, Back and Ulissi44] discovery workflows.)
Generalizable automation and data capture in biology is more mature and includes variants such as Wet Lab Accelerator,[Reference Bates, Berliner, Lachoff, Jaschke and Groban45] Autoprotocol,[46] laboratory automation such as Par-Par[Reference Linshiz, Stawski, Poust, Bi, Keasling and Hillson47] and Roboliq,[Reference Whitehead, Rudolf, Kaltenbach and Stelling48] full experimental workflow and design with integrated LIMS such as Aquarium,[Reference Keller, Vrana, Miller, Newman and Klavins49] and services offered by companies such as Emerald Cloud Lab[50] and Transcriptic.[Reference Miles and Lee51,52]
Despite the advances in related fields described above, these technologies have experienced limited integration into the chemistry and materials science communities. Furthermore, current technology environments provide limited support for human–robot interfacing, do not support the organization of unstructured or unprocessed data, and often are developed without intention for distribution of curated datasets to non-domain expert computer scientists (i.e., machine learning experts). The development of an extensible framework for materials science data collection should thereby include the following components: First, a generalizable mechanism for specifying machine-readable experiment plans that enable operations via a web-based application programming interaction (API). Second, the facilitation of hybrid human–robot laboratory operations that enable the recording of data and optimization of experiments. Third, a comprehensive capture of data and metadata for a complete description of an experiment, involving both human operations and associated machine-generated files. Finally, processing the data into reports that facilitate subsequent use in machine learning algorithms operated by both domain expert chemists and non-domain expert computer scientists.
We describe here a general data model and associated software pipeline code—ESCALATE (Experiment Specification, Capture and Laboratory Automation Technology)—that addresses each of the needs discussed above. As an illustrative specific example, we describe the ESCALATE implementation for high-throughput exploratory synthesis of single-crystal metal halide perovskites, a class of materials that has attracted a great deal of both fundamental[Reference Mitzi and Karlin53–Reference Li, Zhang, Song, Xie, Meng and Xu55] and technologic interest for application in photovoltaics,[Reference Snaith56,Reference Ansari, Qurashi and Nazeeruddin57] sensors,[Reference Yao, Gui, Zhang and Lin58] lighting,[Reference Lozano59,Reference Smith and Karunadasa60] and batteries.[Reference Ahmad, George, Beesley, Baumberg and De Volder61] The experimental hardware, chemical process details, and scientific outcomes of this “robot-ready perovskite” synthesis method will be described elsewhere. This example is used to showcase how the ESCALATE data model addresses challenges related to capturing the distinction between experimental intent and reality, hybrid human–machine laboratory operations, data capture for multi-step multi-component reagent preparation, handling increasingly complex datasets for evolving chemistry workflows, and curating datasets useable by non-domain experts for machine learning.
Lifecycle of a metal halide perovskite experiment
A robot-ready metal halide perovskite synthesis provides a concrete example of our ESCALATE framework, and illustrates a number of challenges associated with experiment specification and capture that are solved by ESCALATE. Reagent stock solutions consisting of one or more organic, inorganic, and solvent chemicals must be prepared by a human operator, in a process that includes manual weighing, followed by heating and mixing. The intended compositions of these reagent stock solutions cannot exceed the solubility limits of the species. The resulting reagent solutions are then moved to a robotic liquid handler capable of pipetting specified volumes into a set of 96 individually-addressed glass vials (denoted as a “well”). In addition to the order and timing of the reagent additions specific to each individual well, time-varying heating and shaking conditions can be applied globally to the entire collection of vials (denoted as a “plate”). The total volume of all species in a valid experiment cannot exceed the volume limits of the vial in which the experiment is contained. At the conclusion of the synthesis, each vial is photographed from several angles, and scored for the formation of crystals. Each well is an individual experimental entity, described by its own template, model, and object information, but sharing common descriptions of reagent stock solutions and plate-wide conditions.
The lifecycle of an experiment starts as a general “template”, which is a structured, fill-in-the-blank form that provides a general pattern of information needed to specify an experiment and any relevant experimental limits (see the left side of Fig. 1). Specifying the prerequisite portions of the experimental template generates a “model,” which contains the intended laboratory actions and materials. Executing the notional plan described by a model results in an “object” representing the particular physical laboratory instantiation. Each object is characterized by its particular type, material, action, observation, and outcome data, see Table I, as well as the intended plan (model) and general experimental constraints (template). Treating entities in this fashion allows us to track the experimental intent, reproduce the particular experimental execution, and clarify the relationship between the nominal experiment and the empirical observations.
Lifecycle of a metal halide perovskite experimental object starting from the specification of a template experiment through the execution of each experiment object.
Chemical ontology for data collection outlining categories of data collection throughout the pipeline.
Each experimental entity proceeds through four states, as shown in the right side of Fig. 1. The entity begins as a set of templates describing the general process and experimental constraints that must be satisfied. Chemicals are specified as raw ingredients, described in terms of standardized identifiers (e.g., InChIKey) with a set of known attributes (e.g., molecular formula, molecular weight, density). A reagent-type (R) template asks the user to specify specific chemicals and preparation parameters (e.g., mixing time and temperature) and enforces composition constraints (e.g., solubility constraints as a maximum concentration). An experiment-type (E) template asks the user to specify both well- (e.g., statistical distributions of the different reagent volumes to use) and plate-level intent (e.g., addition times, temperature, and mixing conditions) and enforces experimental constraints (e.g., minimum and maximum well-volume limits and operating temperatures). Encoding the experimental constraints into the template allows this to be generalized to other experiment types.
The operator provides the required inputs to the templates using an ESCALATE executable to formally start the generation of models. ESCALATE returns the generated models to the operator. Reagent models include nominal preparation instructions including actions, and materials necessary to prepare the model. Experiment models contain associations with the relevant reagent models as well as the necessary information to perform the experiment in the laboratory.
“Objects” describe the physical reagents and experiments conducted in the laboratory according to the instructions in the model. Data collected about objects are distributed across the type, material, action, observation, and outcome categories as the experiment proceeds (see Table I). For metal halide perovskite experiments, the empirical observations of reagents and experiments include data such as solution volumes, measured temperatures, and concentrations. ESCALATE provides forms for the operator to record observations regarding reagent and experiment objects. The operator aims to capture variance between the nominal specification of the model and the empirical observations associated with a given object. These data are especially important for accurately capturing the properties which exhibit non-ideality during object preparation (i.e., concentration variance resulting from non-ideal solution mixtures).
For metal halide perovskite experiments, the operator records two data points in the final step: (i) scoring based on optical microscopy and (ii) photographic images of each well of the robotic plate (Fig. 2). Optical microscopy data are recorded as outcomes on a scale from 1 to 4, where 1–3 indicates a “failure” to produce large single significant crystals, and 4 indicates the “success” of producing large single crystals.
Operational overview of ESCALATE detailing data collection forms and distribution of data into ontologic components leading to a final curated data frame.
The “type” category describes general information about the nature of the entity, who interacted with the entity, where the entity was created, as well as any associated entities and relevant relationships. Data grouped by type record metadata relating to an entity or to group similar entities together. The “material” category describes the physical components that are part of the entity, such as chemicals, reagents, equipment, and instruments. The “action” category describes operations performed by or to the entity. An “observation” consists of empirical data captured during the course of an experiment; this can include both structured machine-generated data files as well as unstructured general comments from the human operator. An “outcome” is a subcategory of observation describing a (intermediate) product; flagging these as possible dependent variables facilitates use by data scientists. Additional discussion of categories can be found in the Supplementary Information (Fig. S1).
Capture
Subsequent sections of this article detail the underlying software process of ESCALATE for metal halide perovskite data capture and reporting. There are five key data capture steps positioned throughout the pipeline. Each section of data capture has been designed with extensibility, flexibility, and with a regime for reporting in mind. The points discussed in the following sections should provide sufficient background to extend ESCALATE to other materials investigations. ESCALATE is designed as a lightweight way to collect comprehensive data from existing and new experiments, and simplify the data capture, reporting, and transition to “closed loop” operation. ESCALATE is equally suited to any combination of human- and automated processes, unlike comprehensive self-driving laboratory software designed for fully automated operation,[Reference Roch, Häse, Kreisbeck, Tamayo-Mendoza, Yunker, Hein and Aspuru-Guzik41,Reference Häse, Roch and Aspuru-Guzik62] and well-tailored high-throughput infrastructures for specific materials problems.[Reference Nikolaev, Hooper, Webber, Rao, Decker, Krein, Poleski, Barto and Maruyama27] Rather than providing a single implementation strategy, ESCALATE facilitates data reporting and experiment specification through a human- and machine-interpretable file infrastructure compatible with most machine learning programming languages. The key forms, as well as a brief description of the operation of the pipeline software, are outlined in Fig. 2.
Specify
The first point of data collection involves capturing user-provided information to initialize the software (Fig. 2). These data are descriptions of the type of experiment, including what is generally referred to as the metadata of the entity. ESCALATE uses an executable script to capture initial user specifications. ESCALATE parses the provided constraints which include chemicals, reagent, and custom constraints on the experimental entity. An example of the data captured in the specification step of ESCALATE is provided in Table II.
Examples of the structure of data collection necessary to successfully generate perovskite experimental template file.
The experimental template constrains the experimental system and imposes requirements on data needed to define the subsequent models, specifically: a unique identifier (UID), version control for software and experimental process chemistry, and placeholders for the input reagents and chemical identifiers, and laboratory equipment/instrumentation needed. Requiring these data as part of specifying a template ensures that a common baseline of data exists for similar templates.
Chemicals and reagents have locally-resolved UIDs. For example, a URL accessible tabulated data entry form (such as those provided by an online spreadsheet) can store the list of the chemical components of a given reagent model along with an associated UID. In our ontology, “chemicals” describe pure compounds. We use InChIKeys; other identifiers (e.g., SMILES, InChI strings, CAS numbers, PubChem ID) can be used by ESCALATE, but in practice these have problems with canonical order and copyright. In principle, chemical provenance could be further tracked to specific laboratory inventory items, as is done in Aquarium,[Reference Keller, Vrana, Miller, Newman and Klavins49] but this adds too much complexity to the early stages of data capture. “Reagents” describe a set of chemicals along with preparation instructions expressed as an extension to Autoprotocol[46] (see Table II). Each reagent is assigned a UID, which allows for comparison across experiments. Competing approaches for assigning UIDs for reagents to UIDs are also available, such as Synbiohub for biologic components,[Reference McLaughlin, Myers, Zundel, Mısırlı, Zhang, Ofiteru, Goñi-Moreno and Wipat63] or proposed IUPAC systems, such as RInChI or MInChI for descriptions of reactions and mixtures,[Reference Grethe, Blanke, Kraut and Goodman64] respectively, but these only track composition and do not specify preparation conditions such as temperature and mixing parameters.
Equipment and instrumentation provide the last pieces of information needed to constrain the generation of models from a template. In the metal halide perovskite chemistry, the selection of the equipment determines the number of parallel reactions (i.e., how many wells), as well as setting baselines for nominal reaction conditions in the model. For instance, a different type of reaction plate, one being an aluminum block and the other being steel, would exhibit different thermal properties across the plate. This, in turn, would affect the nominal temperature of each reaction well, and thereby alter the nominal temperature proposed for a given experiment at a particular location on the plate. Assigning an equipment UID provides another layer of necessary information for accurately capturing the intention of the experiment and generating accurate nominal models.
Depending on the experiment, other metadata could be tied to the template selection and initial model generation. For instance, the location where the experiment is performed, the chemist executing the code, and a notes section are reasonable data to capture when implementing a template for the generation of models.
Generate
Valid experiments must obey experiment-specific technical constraints (e.g., the total volume of reagents added cannot exceed the container volume) and composition-dependent physical constraints (e.g., the final concentration of a species cannot be greater than the most concentrated reagent solution). ESCALATE enforces these limits using the experiment template and reagent model description. For a given template, the chemical space is bounded hierarchically by the following constraints: (1) the material limits (i.e., the size and mass constraints of the equipment); (2) the type of reagent, and thereby the concentration of chemicals in the reagents; and (3) user-defined constraints (i.e., user-defined minimum concentration of a given chemical). Each additional constraint shrinks the potential search space, which can be quantified through the volume of the convex hull encapsulating possible points in chemical space. For instance, the lower concentration boundary for the organic component of a metal halide perovskite experiment is defined not by the binary mixture containing the organic and solvent, but by the ternary stock solution of inorganic (PbI2), organic (e.g., ethyl ammonium iodide needed to solubilize the inorganic), and solvent (e.g., γ butyrolactone). To satisfy the dependency of the organic concentration originating from two reagents, ESCALATE samples the reagent volumes starting from the reagent with the most chemical components with conflicts or exceptions resolved by user specification. In this case, the volume of the ternary inorganic mixture is sampled first, followed by the binary organic mixture containing solvent and the organic, with the solvent volume sampled last. ESCALATE incorporates limits from each of the entities provided by the template, analyzes the boundaries through the described hierarchy [Fig. 3(b)], and samples from the maximum accessible convex hull to generate the desired number of experimental models.
(a) Single Variable Modulation and (b) quasi-random sampling of metal halide perovskite chemical space in three dimensions. Constraints are illustrated for quasi-random sampling.
ESCALATE provides two modes of experiment generation: (1) quasi-random sampling of a limited number of experiments within the user-specified constraints [Fig. 3(b)] and (2) exhaustive generation of the possible experimental “state space” suitable for use by external experiment recommendation programs (Fig. 4).
Visual representation of a reduced density state space for PbI2, EtNH3I, and formic acid metal halide perovskite system.
Quasi-random experiments generation
Efficient exploration of the multidimensional composition space is a challenging problem. The naive human strategy is to alter a single variable at a time [Fig. 3(a)], but this is inefficient and does not capture multivariate changes.[Reference Duros, Grizou, Xuan, Hosni, Long, Miras and Cronin24,Reference Cao, Adutwum, Oliynyk, Luber, Olsen, Mar and Buriak31] Full or partial factorial grid searches are commonly employed in the statistical design of the experiments, but have the disadvantage of requiring prior knowledge of grid spacing and variable interaction orders. During the first exploration of a new chemical space, an expedient strategy is to generate a quasi-random distribution of compositions [Fig. 3(b)], which unlike pseudorandom sequences, minimize the discrepancy between subintervals.[Reference Press, Flannery, Teukolsky and Vetterling65] No prior knowledge of the relevant spacings is needed and each well composition is generated independently. This has the twofold advantage of (i) avoiding systematic spurious correlations between the position of the well (e.g., inconsistent heating or shaking) and the well composition and (ii) allowing an arbitrary number of experiments to be generated that adequately sample the accessible composition space.
Exhaustive experimental state space generation
A fully sampled convex hull defines the complete experimental “state space.” A file of this “state-set” provides non-domain expert collaborators with the full set of feasible experiments. ESCALATE generates a state space file containing the full reaction description (e.g., concentrations, physicochemical properties, etc.) to the operational specifications needed to define the experiment model (e.g., masses and volumes of chemicals needed). This enables machine learning algorithms to work with the full description of the experiment as an input, without having to carry out these calculations separately. An example of the state space for the PbI2, EtNH3I, and formic acid metal halide perovskite system is shown in Fig. 4.
The state-set CSV file for PbI2, EtNH3I, and formic acid system (three stock solution volume choices) is generated such that each point in the “state-set” refers to unique experiment. The estimated liquid handler dispensing precision is ±5 µL resulting in 10 µL spacing between points along a given reagent axis in the state space.[66] In addition to the three reagent concentrations, the values of 67 reaction descriptors are included for each experiment in the “state-set.” The “state-set” generation requires approximately 15 s on a modern desktop computer and generates an approximately 200 megabyte file. At larger numbers of variables, the increase in computational resources required to generate and store the “state-set” becomes prohibitive; a solution to this scaling problem is on-demand validation of the proposed experiment plans, and is currently under development.
The state-set, along with curated and normalized data from previous experiments (i.e., training set), provides a means for outside collaborators, using their own preferred machine-learning strategies, to recommend possible reactions to the experimental process. Using a state-set as an intermediary enables participation without requiring a collaborator to understand specific experiment constraints or develop a mapping from reagent volume to chemical concentration. A non-expert user or artificial intelligence program only needs to recommend an experiment model from the state space to ensure that the constraints of the experimental template are met and that the object will be incorporated into the pipeline.
Prepare
After performing the experiment generation, described above, ESCALATE has defined a nominal model for each of the entities (reagent, chemicals, and experiments as in Fig. 1) which can be fully characterized by the type, material, and actions categories. The generated experimental models include all information necessary to fully describe the nominal experiments, including reaction temperatures, mixing times, equipment descriptions, specific locations on the robotic plate, and reagent dispensing data.
ESCALATE next sums the target dispense volume of each reagent for all generated experiments occurring on a single 96-well plate and calculates the total nominal amount of each chemical needed to prepare a sufficient quantity of each reagent. Chemical information required to calculate unit conversions for presentation to the operator (e.g., molecular weight, density) is centrally stored in a human interfaceable web-hosted spreadsheet.
The reagent models along with the nominal preparation instructions are provided to the operator as an interactive form with specific handling instructions (temperature, stir rate, duration, etc.) along with nominal amounts of each chemical necessary to prepare the reagents. More generally, interactive forms provide a means to expose intermediary data during ESCALATE operation. The reagent model/object interface provides a single repository for both the reagent data calculated during experimental model generation as well as observations about the reagent objects prepared in the laboratory. In particular, the shared form minimizes human error during entry by automatically prefilling available sections and highlighting to the operator the sections for recording materials, observations, and notes.
A simplified example of the interactive data-entry form is shown in Fig. 5. A full example of the metal halide perovskite interactive form file is discussed in the Supplementary Information. A more detailed illustration is also included as Fig. S2. The completed form is converted to JSON format,[67] a human readable and computer interpretable format, for later data workup and post-processing.
Representation of the simplified reagent model/object interface for capturing metal halide perovskite reagent preparation. Highlighted sections are observations required for executing subsequent steps of the pipeline, all other cells are filled by ESCALATE.
Execute
Execution of experiment objects generated by ESCALATE has been predominantly performed by a Hamilton Microlab NIMBUS4 robot. The current configuration of the Hamilton software uses an XLS formatted control file to instruct the volume dispensed into each of the 96 wells on a plate. For our particular purposes, ESCALATE has been programmed to output experimental models as a spreadsheet, but the final output of the experimental object file is flexible. A subset of the experimental model file is provided in Table III and a specific example file is discussed in the Supplementary Information (Experiment-Model_RobotInput.xls).
Example of selected entries taken from the experimental models’ file (robot input file) including the nominal values for entities (vial site, labware), materials (reagents), actions (temperature, mixing).
Note that the experimental model file includes type information, relevant for describing the particular location on the plate an experiment is being performed. Nominal reagent dispensing information is dictated to the robot on a well-by-well basis and can vary from one experiment to the next. Other properties such as reaction time, mixing conditions, and temperature are nominal values that apply to the reaction plate as a whole; the robot does not have the ability to mix individual wells at different rates.
ESCALATE has also been used in combination with human operators for reagent dispensing with little alteration of the code. For instance, manually dispensing highly varied volumes to specific locations on a plate is tedious. To ease implementation for human interaction, manual modification of the nominal reagent volumes after generation of the experimental model is fully supported. Thereby, an operator can alter the experimental model file if necessary to accurately capture the intention of the experiments.
Observe
Observations collected during the perovskite experiments include ambient temperature and humidity, thermal images which map the actual temperature across the heating block, operator notes, liquid handler operational log files, instrument configuration and setting files, images of the products, and the final experimental outcome. Instrument output files are copied directly to the online repository for later use and human observations are recorded by the operator during the experiment. Metadata capture and management solutions for instrumentation, such as Allotrope Standard,[68] can be included as files in the ESCALATE data lake.
Other files collected during ESCALATE operation, including parsable outputs from instruments, images, and log files, are attributed to the relevant experiments. The advantage of a data lake architecture is that no initial effort needs to be invested in parsing them before they can be collected. However, by storing them all, should scientific interest arise, a future processing pipeline can retroactively extract the relevant data. A full example of the data collected during typical ESCALATE operation is included as and discussed in the Supplementary Information.
Report
The second portion of code processes the gathered data into a structured format that can be used for analysis (e.g., data visualization, machine learning), as depicted in Fig. 6. Categories of data (i.e., material, action, etc.) can be individually targeted by coding operations designed for converting grouped data into the final report. For example, the final concentration of a particular chemical in an experiment is dependent upon empirical measurements contained in the reagent model and the experimental model, two large repositories of data. However, the material category contains all of the information necessary to characterize the concentration of a specific chemical in the final experiment. More importantly, the organization of the data into the material category provides a target staging platform for the development of automated data processing pipelines. The categories are thereby effective not only from the perspective of ensuring accurate data capture, but also for providing an extensible platform to apply ESCALATE to new classes of experiments.
ESCALATE process for generating the final report data frame.
The report format depends upon the recipient and the type of experiment. For shallow datasets with a limited total number of experiments, as is common in materials chemistry research, a two-dimensional data array file (e.g., CSV) is readily imported by data processing packages. Data presented in two dimensions are often wide (i.e., large numbers of columns) which encumber human inspection and analysis without the aid of additional software. Employing systematic naming schemes can aid in the processing of 2d data. The implementation used in ESCALATE is outlined in Table IV.
Overview of the namespace for dataset normalization.
* is a wildcard indicating any combination of characters, phrases, or abbreviations are acceptable at the indicated position.
The key features of the naming syntax include (1) keyword-oriented titles; (2) informative, machine-parsable prefixes and suffixes; (3) avoiding spaces. Keywords provide a general description of what is contained in a particular column of data. If a user or program needs to operate on all “chemical” and “InChIKey” information, then the column headers must be similar enough to be generally referenced. Further specification by the user could include a numerical argument, “reagent_1_chemical_1_InChIKey” providing only the InChIKey from a specific chemical in “reagent 1.”
Data presentation generated by ESCALATE uses prefixes to help parse different general categories of data. “Raw” refers to unprocessed data that are often unsuitable for consumption by non-domain experts. An example is extensive quantities such as mass that specify a particular experiment, even though the intensive property of concentration is more relevant. “Rxn” includes both processed and unprocessed data experimental specifications that are suitable for machine learning and data analysis. “Feat” and “calc” describe the physicochemical properties of the chemicals and reagent mixtures, respectively, and are calculated using ChemAxon[69] and RDKit[Reference Landrum70] in the current implementation. “Out” indicates suitable dependent variables to be predicted by machine learning applications (e.g., metal halide perovskite crystal score). Suffixes distinguish “nominal” (i.e., experimental intent) and “actual” (i.e., empirical observation) values. Using an underscore or hyphen delineated division between prefix, description, and suffix avoids common problems with parsing as some programming languages handle spaces poorly.
A simplified example of an ESCALATE rendered report in the format of a 2d data frame is presented in Table V, a full example from a single plate of reactions is included as FinalReport_2d_Curated.csv with the Supplementary Information. Each row of the final data frame represents a complete description of the experimental intent and the performance of the nominal experiment. The namespace, “RunID_Vial,” is a concatenation of the well site in which the reaction was performed in combination with the previously described “Run ID.” The namespace represents the UID of the experiment; no duplicates should exist in the final dataset. The headers contain the relevant prefixes and suffixes to describe the data and are organized in such a way to group similar data.
Final data-frame structured for distribution.
* is a wildcard indicating any combination of characters, phrases, or abbreviations are acceptable at the indicated position.
ESCALATE curates experimental data to support “best-practices” such as the F.A.I.R. data principles,[Reference Wilkinson, Dumontier, Aalbersberg, Appleton, Axton, Baak, Blomberg, Boiten, da Silva Santos, Bourne, Bouwman, Brookes, Clark, Crosas, Dillo, Dumon, Edmunds, Evelo, Finkers, Gonzalez-Beltran, Gray, Groth, Goble, Grethe, Heringa, ‘t Hoen, Hooft, Kuhn, Kok, Kok, Lusher, Martone, Mons, Packer, Persson, Rocca-Serra, Roos, van Schaik, Sansone, Schultes, Sengstag, Slater, Strawn, Swertz, Thompson, van der Lei, van Mulligen, Velterop, Waagmeester, Wittenburg, Wolstencroft, Zhao and Mons71] which calls for making the data Findable, Accessible, Interoperable, and Reusable. ESCALATE makes the data findable by assigning each experiment a unique human- and machine-readable identifier to maintain origin, laboratory, operator, and date provenance to each entry. ESCALATE is not a publication/dissemination platform, but broader accessibility can be achieved by upload to public repositories such as NREL's HTEM DB[Reference Zakutayev, Wunder, Schwarting, Perkins, White, Munch, Tumas and Phillips37] or Citrine.[72] Interoperability is provided by the report generation which compiles the experimental data and metadata into a CSV format, which can be imported into a wide variety of visualization, statistics, and machine learning packages. The report also incorporates calculations of physicochemical properties of the reagents, stoichiometry, concentration, etc., that facilitate the use of these data by non-domain experts. The data are reusable at two levels. The output reports described above are reusable for various analyses. Furthermore, the metadata persistently associated with each item in the data lake allows for retroactive retrieval and analysis of other experimental details should a future need arise.
Closed-loop operation
The current infrastructure for closed-loop operation uses two CSV files in a shared directory to provide the report of past experimental results that can be used to train the model and the state-set of possible experiments. The state-set includes a complete description of each nominal model experiment, including all of the computed material physicochemical features and stoichiometric calculations that are included in the report. By providing all of the necessary domain-specific calculations, the report and state-set files allow non-domain experts to easily import the existing metal halide perovskite dataset, train a model using existing data, and make recommendations for upcoming experiments. Experiment recommendations are communicated by writing a file containing a list of desired state-set indices into a shared directory. An ESCALATE command line program reads these files and generates the complete experiment description (experiment template, experiment model, reagent model, etc.) to be performed by the operator. ChemOS uses a similar shared-filed based scheme for closed-loop operation.[Reference Roch, Häse, Kreisbeck, Tamayo-Mendoza, Yunker, Hein and Aspuru-Guzik41] An experiment submission and validation API is currently under development.
Results
ESCALATE has been operating for 10 months as of April 2019. In that time, the data pipeline has operated a total of 55 times, capturing a total of 4336 single-crystal metal halide perovskite reactions. The throughput volume of data capture using this software is illustrated in Fig. 7(a).
(a) Total experimental throughput of curated experiments captured by ESCALATE present in the final dataset by month; (b) number of experiments performed for different organic cations (ammonium iodide salt).
The throughput of data collection has only been limited by experimental capacity; ESCALATE has facilitated the maximum bandwidth of experiments possible. These data represent one of the largest metal halide perovskite crystallization datasets ever curated. The comprehensive data capture pipeline results in a dataset containing all of the “dark reactions”—otherwise unreported “failures” and marginal success—that are crucial for the development of machine learning models in materials science research.[Reference Raccuglia, Elbert, Adler, Falk, Wenny, Mollo, Zeller, Friedler, Schrier and Norquist20] The template-driven experiment generation results in an unbiased sampling of chemical space. The ontology and data structure developed for ESCALATE has also enabled collaboration with non-domain experts. Four research groups—at Lawrence Berkeley National Laboratory, the Broad Institute, Haverford College, and Netrias—comprised of a mixture of chemists, materials scientists, computer scientists, and mathematicians—have recommended more than 96 metal halide perovskite experiments per week generated by a variety of machine learning algorithms trained on ESCALATE generated report CSV files. These recommended experiments are conducted at the Lawrence Berkeley National Laboratory's Molecular Foundry, resulting in a total of 16 iterations of a DARPA sponsored interactive campaign for materials discovery. This campaign is ongoing, and results will be reported in future articles.
Various experimental workflows have been developed in the laboratory and implemented using ESCALATE for data capture. The most successful workflow has covered eight different single-crystal metal halide perovskite experiments using lead iodide, an ammonium iodide, and formic acid in GBL [Fig. 7(b)]. Work is underway to further broaden ESCALATE to aid in the exploration of metal halide perovskite chemical space. Specifically, we aim to incorporate additional halides, amines, and inorganics and various crystallization regimes into this dataset in the near future.
Current limitations
Limitations of the current version of ESCALATE reflect the experimental needs of the perovskite experiments discussed above, in which up to seven human-prepared solutions of reagents (comprised of up to three chemicals) are dispensed as solutions either by a human operator or robot in batches of no more than 96 experiments. (The underlying data handling supports an unlimited number of reagents and chemicals, but the current ESCALATE generated graphical user interface is limited to up to three chemicals per reagent for the human operator instruction, and up to seven reagents per experiment for the liquid handler control file. The other core functionalities of ESCALATE—specification, generation, preparation, observation, and reporting—support input of any number of chemicals per reagent composition, number of reagent solutions, and number of experiments.) In the example described in this article, ESCALATE generates the experimental model file as an XLS in the format used by a Hamilton Microlab NIMBUS4 robot. The experimental model file used to operate the NIMBUS liquid handler robot also includes the robotic execution control file specific to this instrument and experiment type. These instrument- and vendor-specific programs can be modified by the users for the desired application and instrumentation available; a copy is retained in the data lake for each experiment.
The data handling is performed using pandas dataframes which have a variety of native output formats[Reference McKinney73] that allow the experimental model, reagent model, and final report files to be rendered as a CSV, textfile, XLSX, or other standard formats via minor modifications of the code; exceptions to the pandas data handling occurs only in intermediary file storage mechanisms (e.g., Autoprotocol formatted JSON). Closed-loop execution is initiated by executing a command line program that reads the recommended experiments from the state-set CSV, similar to the operation of ChemOS.[Reference Roch, Häse, Kreisbeck, Tamayo-Mendoza, Yunker, Hein and Aspuru-Guzik41]
Concluding remarks
ESCALATE is both an ontologic framework for describing experimental entities in a computer-friendly format and a software package for facilitating experiment specification, data capture, and reporting. The initial development of infrastructure can be a daunting task for scientists unfamiliar with the process. Often the simplest course of action is to start by organizing the relevant types, materials, actions, observations, and outcomes needed to describe the target system. This article describes examples of data belonging to these categories and demonstrates how to assemble meaningful datasets from typical laboratory processes, organized around familiar chemical concepts and structured to facilitate both human and machine interpretability. We offer a specific demonstration to single-crystal metal halide perovskite synthesis, which illustrates laboratory manipulations, high-throughput applications, and interfacing challenges common to many research endeavors. The combination of this ontology with the software package makes it easy for experimentalists to start collecting valuable, comprehensive datasets in automated and semi-automated laboratory environments without a large initial investment in database design. Additionally, a non-expert user or artificial intelligence program need only recommend an experiment generated by ESCALATE to ensure that the constraints of the chemical system are met and can be performed in the laboratory. Captured files are stored in a way that facilitates transparent access by users, even those without programming background. At the same time, the structure imposed by the ontology facilitates future data extraction. Future work will be aimed at improving data workflow, generalizing ESCALATE to additional experimental systems, and closed-loop autonomous experimental design.
Code availability
The code used for this project can be found at the following links: https://github.com/darkreactions/ESCALATE_Capture and https://github.com/darkreactions/ESCALATE_report and is released under an MIT license. Discussion about code implementation and a comprehensive overview of a single iteration of ESCALATE along with the associated files can be found in the Supplementary Information.
Supplementary material
The supplementary material for this article can be found at https://doi.org/10.1557/mrc.2019.72.
Acknowledgments
We thank Alex Cristofaro (MIT Broad Institute) and Scott Novotney (Two-Six Labs) for helpful feedback in the development of the ESCALATE state space and file reporting mechanisms. This material is based upon the work supported by the Defense Advanced Research Projects Agency (DARPA) under Contract No. HR001118C0036. Any opinions, findings, and conclusions or recommendations expressed in this material are those of the authors and do not necessarily reflect the views of DARPA. This material is DISTRIBUTION A. Approved for public release: distribution unlimited. Work at the Molecular Foundry was supported by the Office of Science, Office of Basic Energy Sciences, of the US Department of Energy under Contract No. DE-AC02-05CH11231. J.S. acknowledges the Henry Dreyfus Teacher-Scholar Award (TH-14-010).
