Participants

The study cohort consisted of 163 patients (92 female; 162 Caucasian). All patients were evaluated at The Memory Assessment Program, Rowan University, New Jersey Institute for Successful Aging; and at the Department of Neurology, Drexel University, Pennsylvania. Rowan University and Drexel University institutional review boards approved this investigation and the current research complied with the Declaration of Helsinki. All data were collected between 2005 and 2018. The work-up for MCI and dementia was identical between research sites and included comprehensive neuropsychological assessment, evaluations provided by a board-certified geriatric psychiatrist or neurologist, a brain MRI/CT scan, and serum blood tests. An interdisciplinary team consisting of a neuropsychologist, psychiatrist/or neurologist, and social worker determined clinical diagnosis. Exclusion criteria included any lifetime history of head injury, substance abuse, major/medical psychiatric disorders (e.g., major depression and epilepsy), B12, folate, or thyroid deficiency. The presence of these issues was determined on the basis of information provided by both patients and their families and was deemed positive if any of these issues interfered with work or other aspects of daily life. Any changes in functional independence in connection with a blow or strike to the head were considered indicative of history of head injury. All neuropsychological evaluations were obtained by one of us (DJL, a PhD level, broad-certified neuropsychologist). A knowledgeable family member provided information regarding functional status. Demographic and clinical characteristics including age, education, and Mini-Mental State Exam Performance (MMSE; Folstein, Folstein, & McHugh, Reference Folstein, Folstein and McHugh1975) are displayed in Table 1.

Table 1.

Demographic information (means and standard deviations)

Note: With the exception of one, all study participants were Caucasian.

Neuropsychological Assessment

The neuropsychological protocol used to clinically diagnose non-MCI and MCI patients is identical as described by Emrani et al., (Reference Emrani, Libon, Lamar, Price, Jefferson, Gifford, Hohman, Nation, Delano-Wood, Jak, Bangen, Bondi, Brickman, Manly, Swenson and Au2018). Three domains of neuropsychological functioning were assessed including executive control, naming/lexical access, and verbal episodic memory. Nine neuropsychological parameters, three from each neurocognitive domain, were employed to classify patients as presenting with non-MCI or MCI subtypes as described below. Tests were expressed as z-scores derived from normative data as listed below.

Diagnostic Determination

The Digital Clock Drawing Test

The instructions used to administer the dCDT are consistent with traditional CDT administration and included both a command and copy test conditions. In the command condition, patients were asked, “draw the face of a clock, put in all numbers, and set the hands for 10 after 11.” Upon completion of the command test condition, the copy test condition was administered whereby patients were asked to copy a model of a clock with hands set for 10 after 11. The dCDT (Davis et al., Reference Davis, Penney, Pittman, Libon, Swenson and Kaplan2010; Penney, Davis, et al., Reference Penney, Davis, Libon, Lamar, Price, Swenson, Garrett, Freedland, Weninger, Scala, Giovannetti and Kaplan2010) was developed by the Lahey Clinic and Massachusetts Institute of Technology in collaboration with the ClockSketch Consortium and uses digital pen technology developed by Anoto, Inc., Westborough, MA, USA. The pen works as an ordinary ball point pen while capturing pen position 80 times/sec. + .002. All data are time-stamped, allowing the pen to capture the final drawing for more accurate classification than would be possible without this technology. The Clocksketch program can label and calculate latencies for the length and number of strokes associated with individual clock components such as digits and hands; the dimensions and orientation of the clock face, digits, and hands; the time elapsed during and between the drawing of individual clock features; and the deviation of clock features from “ideal” placement on the command clock stimulus.

Data Preprocessing

Prior to formal analysis, all dCDT data underwent preprocessing to remove duplicate features, noisy features, and extremely sparse features, that is, those that are only available for a very small subset of the patients (such as clock drawing center dot and related features). All feature values were standardized by subtracting the mean and dividing with the SD of that feature obtained from the total sample. Standardization allows the down-stream processing to be independent of scaling and amplitude.

Information theoretic approaches for feature selection evaluate mutual information (MI) among classifier variables to identify and reduce the number of features with shared predictive contributions. These approaches were based on discrete outcome probabilities, that is, to augment the selection of predictive features, continuous values were converted to discrete, categorical values in order to better isolate MI contributed by these features. All continuous valued features were discretized by binning data into 10 bins using the discretization function in Gavin Brown’s FEAST toolbox for Matlab® (Brown, Reference Brown2009). The process of binning divides continuous values into N = 10 categorical values, where all N bins have equal ranges.

For some patients, features of interest were missing such as the absence of a clock hand or digit within the clock face. To resolve issues revolving around missing features, a k-nearest neighbor (kNN) algorithm was used to impute the missing values. For any missing feature, kNN searches the k nearest neighbors of available features with respect to Euclidean distance and averages them to fill in for the missing data (Beretta & Santaniello, Reference Beretta and Santaniello2016). To find kNNs, Euclidean distance was calculated for all relevant features. For example, if a patient did not draw the number “3,” a search was performed on all patients of the same diagnosis who did draw the “3.” Then, the k patients who were most similar with respect to the Euclidean distance to the patient in question were identified. The features associated with the number “3” were averaged from these k patients and were then incorporated as the missing values. To account for patients who did not draw an entire digit and/or clock hand(s), binary variables were created and coded such that a one indicated the value was originally missing and a zero indicated it was available. These additional features preserved the information that the value was imputed for the corresponding patient.

The last preprocessing step involved data partitioning with respect to diagnostic labels. As stated above, the overall goal of this project was to classify non-MCI, MCI subtypes, and AD patients into their respective groups. A singe four-group classification and two combinations of three-group classifications were calculated. However, because of modest sample size among our four patient groups, we were particularly interested in how well machine learning feature selection was able to classify patients into any combination of two groups. Hence, six 2-class problems were evaluated (i.e., all pairwise combinations of two diagnostic groups): non-MCI versus aMCI; non-MCI versus mixed/dys MCI; non-MCI versus AD; aMCI versus mixed/dys MCI; aMCI versus AD; and mixed/dys MCI versus AD. In performing these experiments, information theoretic feature selection algorithms were applied to the problem and then fit to an appropriate feedforward neural network classifier (see Table 2). In an additional, follow-up analysis, aMCI and mixed/dys MCI groups were combined to enable an analysis of classification accuracy for non-MCI versus MCI.

Table 2.

Information theoretic final results

FS criterion = feature selection criterion; mxMCI = mixed/dysexecutive mild cognitive impairment.

Information Theory-Based Feature Selection

The classification of patients through machine learning-based analysis of the dCDT was based on two processes: information theory-based feature selection to identify the most relevant constellation of clock features for distinguishing between groups and successive training of a neural network classifier to identify group membership based on training data for which the group was known.

As noted above, 350 command and copy dCDT features were obtained from the entire sample of 163 patients. Information theoretic-based feature selection approaches were used to determine the most informative features (Brown, Reference Brown2009). Information theoretic methods evaluate different forms of entropy-based MI between diagnoses and features.

Several information theoretic measures are available for computing the amount of information carried by a set of attributes. In this study, we evaluated MI, minimum redundancy maximum relevancy (MRMR), joint mutual information (JMI), and conditional mutual information maximization (CMIM), as described in more detail below (Brown, Reference Brown2009). All of these metrics are based on the central measure of information, defined by entropy, which measures the amount of uncertainty in a random variable, that is, the higher the uncertainty, the higher the information. Given a random variable X (e.g., a specific feature), the entropy of X is given by Equation 1.

(1)$$H\left( X \right) = - \mathop \sum \nolimits_{x \in X} p\left( x \right)\log p\left( x \right)$$

where $p\left( x \right)$ is the probability that the random variable X assumes the specific value x. For example, if the specific value of a feature is always the same (i.e., there is no uncertainty), the $p\left( x \right)\;$ is either 0 or 1, depending on the value of x.

Shannon’s mutual information I (X,Y)

This is defined as the dependence between two random variables X and Y as shown in Equation 2.

(2)$$I(X;Y) = \sum\nolimits_{x\in X} {} \sum\nolimits_{y\in Y} p (x,y)\log \left( {{{p(x,y)} \over {p(x)p(y)}}} \right)$$

where p(x, y) is the joint probability that the random variable X assumes a particular value x, and the random variable Y assumes a particular value y. For a given feature represented by the random variable X and corresponding diagnosis represented with the random variable Y, the MI between X and Y is a measure of the change in entropy (drop in uncertainty) of Y due to the presence of X. Therefore, high MI between a feature and a diagnosis indicates that the feature is a good predictor of its respective diagnosis.

Minimum redundancy maximum relevancy

This was defined below in Equation 3 and includes the consideration of redundancy in addition to maximizing the MI of a feature and a diagnostic label.

(3)$${J_{mrmr}}\left( {{X_n}} \right) = {\rm{I}}\left( {{X_n};Y} \right) - {{1}\over{{n - 1}}}\mathop \sum \nolimits_{k = 1}^{n - 1} I\left( {{X_n};{X_k}} \right)$$

Specifically, the MI between two features ${X_n}$ and ${X_k}$ (i.e., the redundancy in knowing both ${X_n}$ and ${X_k}$), averaged over all features $k = 1, \ldots ,\;n - 1$, is subtracted from the MI between the feature of interest ${X_n}$ and the label Y. The purpose of MRMR is to eliminate redundant features, while still selecting a set of features that enable an accurate prediction of the class label.

Joint mutual information

As shown in Equation 4, this measure includes an additional positive term, the conditional redundancy, compared to MRMR.

(4)$${J_{jmi}} = I\left( {{X_n};Y} \right) - {{1}\over{{n - 1}}}\mathop \sum \nolimits_{k = 1}^{n - 1} \left[ {I\left( {{X_n};{X_k}} \right) - I\left( {{X_n};{X_k}{\rm{|}}Y} \right)} \right]$$

The conditional redundancy is the redundancy between two features ${X_n}$ and ${X_k}$ given the diagnostic label Y. This term is an indication of first-order interaction between those two features that, when used together, is beneficial in the prediction of the diagnostic group.

Finally, CMIM is similar to JMI; however, CMIM takes a more pessimistic approach than JMI (Equation 5).

(5)$${J_{cmim}} = I\left( {{X_n};Y} \right) - ma{x_k}\left[ {I\left( {{X_n};{X_k}} \right) - I\left( {{X_n};{X_k}{\rm{|}}Y} \right)} \right]$$

Instead of taking the average of all first-order interactions (i.e., difference between redundancy and conditional redundancy), CMIM only considers the specific pair $\left( {{X_n}\;;\;{X_k}} \right)$ whose first-order interaction outputs the maximum score. For this reason, CMIM selects the feature ${X_k}$ based on the interaction with an already selected feature ${X_n}$ that gives the lowest score J. Therefore, if ${X_k}$ has high redundancy and low conditional redundancy with only one other feature, then ${X_k}$ will have a low score regardless of the interactions of ${X_k}$ with other features.

Using these metrics as a feature selection approach involves computing the relevant information theoretic metric for each of the features followed by rank ordering the features in descending order according to the scores they receive. Then, a subset of these features with the highest scores is selected as the most informative feature subset.

Balancing the Data

In the current data set, the largest group (i.e., patients with AD) had 59 patients and the smallest group (aMCI) has 26 patients. The adverse impact of imbalanced datasets, creating a bias in the classification decisions of most machine learning algorithms, is well known. To address the imbalanced data concern, we used the well-established Synthetic Minority Oversampling Technique (SMOTE) to rebalance the classes (Chawla, Bowyer, Hall, & Kegelmeyer, Reference Chawla, Bowyer, Hall and Kegelmeyer2002). This approach adds carefully constructed synthetic representative data points to underrepresented classes near the vicinity of existing instances. SMOTE does this by first determining the kNNs of each existing minority instance (k is set to five by default). The algorithm then randomly chooses an instance of a minority class and randomly selects one of its kNNs. Synthetic instances are then created through interpolation along the vector space between the randomly selected instance and its kNN. This process is repeated for additional instances of each minority class until all the minority classes are the same size as the majority class. It is important to note that SMOTE was only used to rebalance training datasets with which the classification models were trained. All diagnostic performances listed in the Results section were computed on a test/ evaluation dataset that strictly consisted of real patient data only.

Classifier Training

Group classification of patients based on dCDT feature selection was accomplished using feed-forward neural networks, due to their previously reported success in classifying medical data for diagnostic purposes (Amato et al., Reference Amato, López, Peña-Méndez, Vaňhara, Hampl and Havel2013) and because of the ability to modify the number of hidden layers and the number of nodes per hidden layer, which allows the network to adjust its learning capacity and approximate decision boundaries (Autio, Juhola, & Laurikkala, Reference Autio, Juhola and Laurikkala2007). To address over-fitting, we implemented a relatively “small” network with fewer adjustable weights. A gradient descent-based stochastic optimization approach was the optimizer used, with early stopping as a further protection against over-fitting (Kingma & Ba, Reference Kingma and Ba2014).

Three different neural network architectures were compared to identify the best predictive configuration for accurately identifying group membership based on dCDT features alone: (1) one hidden layer of 50 nodes, (2) two hidden layers of 10 nodes each, and (3) two hidden layers with 20 and 10 nodes. Each of these structures was trained for features selected with each of the four information theoretic criteria in order to determine the highest performer of each test case (see Supplemental Tables 1–9). We further implemented a 10-fold cross-validation to obtain a better prediction of the true generalization performance of the diagnostic classifier, as well as a confidence interval of the estimated performance. In all cases, the diagnostic performance was computed on test datasets that had been set aside and not used for training, and strictly consisting of real patient data only.

There is, of course, empirical interest in describing the contributions of individual features to accurate classification. Unfortunately, feed-forward machine learning analysis does not yield this kind of information, because it does not analyze data using a General Linear Model (GLM)/regression-based model. While magnitude is considered, there is no directionality assigned to features or the score that ranks features using MI analysis. In addition, any weight that the analysis assigns to a given feature is masked by the hidden layers of the neural network architecture. It would be inaccurate to think of the relationship between a given feature and a classification label as linear, that is, possessing an easily interpreted directional value, because all features are being considered holistically within the model to arrive at a “best fit” classification. While the features themselves and their rank order allow for speculation about what value individual features contribute to the accuracy of classification, much of what machine learning analyses extract from a given feature is hidden within the analysis, thus limiting anything that can be said beyond that a given feature is relevant to distinguishing between two or more groups.