2.1 CD for dichotomized items

In CDMs, the cognitive skills assessed by an assessment are referred to as attributes. Assuming all attributes are binary and exhibit no hierarchical structure, if the assessment is used to measure K attributes, $2^K$ possible latent classes can be formed. Each latent class $C_m$ is defined by a K-dimensional attribute profile $\boldsymbol {\alpha }_m = (\alpha _{m1}, \ldots , \alpha _{mK})^\prime $ , where $m=1,\ldots ,2^K$ . Each entry $\alpha _{mk}$ in $\boldsymbol {\alpha }_m$ takes the value of 1 if the kth attribute has been mastered and 0 otherwise. For examinee i belonging to latent class $C_m$ , his/her attribute profile, $\boldsymbol {\alpha }_i$ , can also be denoted as $\boldsymbol {\alpha }_m$ .

For all CD analyses, the Q-matrix is an integral component, used to indicate relations between the test items and their required attributes. Suppose a test of J MC items measures at most K attributes. When the MC items are dichotomized to fit a binary CDM, only the q-vector of each key is specified and thus, the Q-matrix, denoted as $\boldsymbol {Q}_{\text {B}}$ , has the dimensions $J \times K$ . Similar to an examinee’s attribute profile $\boldsymbol {\alpha }_i$ , the q-vector of item j, denoted as $\boldsymbol {q}_j$ , is K-dimensional, with $q_{jk} = 1$ indicating that attribute k is required by item j and 0 otherwise.

2.1.1 The DINA model

The DINA model is the most extensively researched dichotomous CDM. Let $\boldsymbol {Y}_i = (Y_{i1}, \ldots , Y_{iJ})^\prime $ be the observed response vector of examinee i. The item response function (IRF) of the DINA model is defined as

$$ \begin{align*} P(Y_{ij}=1|\boldsymbol{\alpha}_i)=(1-s_j)^{\eta_{ij}}g_j^{1-\eta_{ij}}, \end{align*} $$

where $s_j$ and $g_j$ represent the slipping and guessing parameters of item j, respectively, subject to the constraint $0 < g_j < 1- s_j < 1$ . The parameter $\eta _{ij} =\prod _{k=1}^K\alpha _{ik}^{q_{jk}}$ is the ideal response of examinee i to item j, indicating whether the examinee has mastered all the required attributes. Hence, the DINA model is a typical conjunctive model.

2.2 Completeness of $\boldsymbol {Q}_{\text {B}}$ for the DINA model

The concept of a “complete” Q-matrix was first discussed and proven in the asymptotic classification theory for CD (ACTCD; Chiu et al., Reference Chiu, Douglas and Li2009). A Q-matrix is said to be complete if it can identify all possible attribute profiles. Köhn and Chiu (2016) revisited the ACTCD and offered a more general definition of Q-matrix completeness based on the conditional expectation of the observed response given an examinee’s attribute profile. Formally stated, defining $\boldsymbol {S}(\boldsymbol {\alpha })=E(\boldsymbol {Y}|\boldsymbol {\alpha })$ , a test is complete if $\boldsymbol {S}(\boldsymbol {\alpha })=\boldsymbol {S}(\boldsymbol {\alpha }^{\ast })$ implies $\boldsymbol {\alpha }=\boldsymbol {\alpha }^{\ast }$ for all $\boldsymbol {\alpha }$ and $\boldsymbol {\alpha }^{\ast }$ . Based on the definition of completeness of the Q-matrix, Chiu et al. (Reference Chiu, Douglas and Li2009) and Köhn and Chiu (2016) established the sufficient and necessary conditions for a complete Q-matrix for the DINA model. The theorem states that $\boldsymbol {Q}_{\text {B}}$ is complete for the DINA model if and only if it includes all the $\boldsymbol {e}_k$ , where $k = 1, \ldots , K$ . Here, $\boldsymbol {e}_k$ denotes the $1\times K$ unit vector, with the kth entry being 1 and all other entries being 0. Detailed proof can be found in Chiu et al. (Reference Chiu, Douglas and Li2009) and Köhn and Chiu (2016).

2.3 CD for MC items

When traditional CDMs are used, the responses to all the distractors of an MC item are lumped together as the incorrect response. Consequently, the distractors remain unavailable for further analysis, resulting in the loss of valuable diagnostic information. A preferable approach is to employ CD methods specifically tailored for MC items, and the recent developments in CD for MC items are briefly presented below.

In the context of CD for MC items, not only the key but also all or a subset of the distractors are assigned specific q-vectors. In general, an option (which could be a key or a distractor) is said to be coded if it is assigned a q-vector. When the observed responses are analyzed using a CD method developed for MC items, the Q-matrix for MC items, denoted as $\boldsymbol {Q}_{\text {MC}}$ , is augmented from $\boldsymbol {Q}_{\text {B}}$ to account for the specification of q-vectors for the coded distractors. In this case, each MC item corresponds to a sub-Q-matrix, $\boldsymbol {Q}_j$ , with dimensions $H_j^\ast\times K$ , where $H_j^\ast$ is the number of coded options, expressed as follows:

$$ \begin{align*} \boldsymbol{Q}_j = \begin{pmatrix} \boldsymbol{q}_j^{(H^\ast_j)}\\ \vdots\\ \boldsymbol{q}_j^{(1)} \end{pmatrix}_{H_j^\ast \times K}, \end{align*} $$

where $\boldsymbol {q}_j^{(H^\ast _j)}$ is the q-vector of the key, and the others represent the q-vectors of the coded distractors. $\boldsymbol {Q}_{\text {MC}}$ is essentially the collection of all $\boldsymbol {Q}_j$ , having the dimensions ( $\sum _{j=1}^JH_j^\ast) \times K$ . Based on this setup, $\boldsymbol {Q}_{\text {MC}}$ can be expressed as follows:

$$ \begin{align*} \boldsymbol{Q}_{\text{MC}} = \begin{pmatrix} \boldsymbol{Q}_1\\ \vdots\\ \boldsymbol{Q}_J \end{pmatrix}_{(\sum_{j=1}^JH_j^\ast) \times K}. \end{align*} $$

2.3.1 The MC-DINA model and its modifications

The MC-DINA model (de la Torre, Reference de la Torre2009), an extension of the DINA model, is the first CDM designed for MC items. In this study, a constraint that the q-vectors of the coded options were hierarchically nested was imposed, which limited the practicability of the model, and thus, all response options form ordinal options. Let $\boldsymbol {X}_i = (X_{i1}, \ldots , X_{iJ})^\prime $ be the observed response vector, and $X_{ij}$ can take on values $h=1,\ldots ,H_j$ , where $H_j$ is the number of options. Then, the ideal response of examinee i to MC item j is defined as follows:

$$ \begin{align*} g_{ij} = \underset{h^\prime = 0,1,2,\ldots,H_j} {\arg\max} \Big\{ \boldsymbol{\alpha}^{\prime}_i \boldsymbol{q}_{j}^{(h^\prime)} \mid \boldsymbol{\alpha}^{\prime}_i \boldsymbol{q}_{j}^{(h^\prime)} = \boldsymbol{q}_{j}^{(h^\prime)\prime} \boldsymbol{q}_{j}^{(h^\prime)} \Big\}, \end{align*} $$

where $h^\prime $ is recoded from h based on the number of skills required by the coded options. More required skills result in a larger $h^\prime $ , and noncoded options have $h^\prime = 0$ . Then, the IRF of the MC-DINA model is formulated as follows:

$$ \begin{align*} P(X_{ij} = h \mid g_{ij} = g) = P_j(h \mid g), \end{align*} $$

without further specifying how to find $P_j(h \mid g)$ .

The stringency of the nestedness constraint imposed on the MC-DINA model limits its applicability and was relaxed in the work by Wang et al. (Reference Wang, Chiu and Köhn2023). As a result, the coded options are no longer ordinal. To avoid ambiguity between notations, let l represent the level of responses without the nestedness constraint. Accordingly, the relaxed rule was proposed to level non-ordinal coded options: The key is assigned the highest level, $H_j^\ast$ , noncoded options are collapsed to the lowest level of 0, and each remaining coded distractor is assigned a unique value from $\{1,\ldots ,H_j^\ast-1\}$ based on two rules: the coded distractor with a longer length is assigned a higher value; if two coded distractors with the same length, the coded distractor with a higher lexicographic order is assigned a higher value. The ideal response to MC item j from examinee i, $\eta _{ij}$ , is then formulated as follows:

(1) $$ \begin{align} \eta_{ij}=\underset{l=0,1,2,\ldots,H_j^\ast}{\max}\left\{l\prod_{k=1}^KI[\alpha_{ik}\geq q_{jk}^{(l)}]\right\}. \end{align} $$

Equation (1) shows that the inclusion of this ideal response makes the MC-DINA model conjunctive, akin to the DINA model. The IRF of the MC-DINA model can then be explicitly written as follows:

(2) $$ \begin{align} P(X_{ij}=l \mid \boldsymbol{\alpha}_i) = \begin{cases} \frac{1}{H_j} + \frac{H_j - H_j^{\ast} - 1}{H_j} \hspace{.3em} I[l = 0] \hspace{8.7em} &\text{if } \eta_{ij} = 0,\\ \Big( 1 - \sum_{c \neq l} \epsilon_{jcl} \Big)^{I[\eta_{ij} = l, l> 0]} \prod_{l^{\prime} \neq l} \epsilon_{jll^{\prime}}^{I[\eta_{ij} = l^{\prime}]} & \text{if } \eta_{ij} > 0, \end{cases} \end{align} $$

subject to $\sum _{c \neq l} \epsilon _{jcl}<0.5$ for all items. In the above equation, $\epsilon _{jcl}$ is the probability that the observed response level c disagrees with the nonzero ideal response level l.

The MC nonparametric classification (MC-NPC) method (Wang et al., Reference Wang, Chiu and Köhn2023) is a nonparametric counterpart of the MC-DINA model that relaxes the nestedness constraint imposed in Reference de la Torrede la Torre’s (Reference de la Torre2009) MC-DINA model.

Its central idea is to classify examinees by comparing the observed response patterns with the ideal response patterns through a penalized Hamming distance, defined as

(3) $$ \begin{align} d_p(\boldsymbol{X}_i,\boldsymbol{\eta}^{(m)})=\sum_{j=1}^J I[\eta_j^{(m)}>0, X_{ij}\neq \eta_j^{(m)}] + \sum_{j=1}^J w_j I[\eta_j^{(m)}=0, X_{ij}\neq \eta_j^{(m)}], \end{align} $$

where $w_j$ is the penalty weight, formulated as

$$ \begin{align*} w_j = \begin{cases} 1-\frac{H_j^\ast-1}{H_j} \quad H_j^\ast &\!\!\!\!\!\!\!\!\!\!< H_j,\\ 0 \qquad\qquad\;\: H_j^\ast &\!\!\!\!\!\!\!\!\!\!= H_j. \end{cases} \end{align*} $$

The attribute profile of examinee i is estimated by minimizing $d_p(\boldsymbol {X}_i,\boldsymbol {\eta }^{(m)})$ over all latent classes. Because the MC-NPC method does not require full parametric estimation of item parameters, it is computationally simpler and avoids some of the convergence difficulties that may arise in fitting the MC-DINA model. Wang et al. (Reference Wang, Chiu and Köhn2023) also reported that the MC-NPC model can provide more accurate classifications than the MC-DINA model in small-sample settings.

3.1 Level-x design

Recall that when there are no hierarchical relations among the attributes, $\boldsymbol {Q}_{\text {B}}$ is complete if and only if it includes all the $K\, \boldsymbol {e}_k$ under the DINA model. It is therefore reasonable to presume that including all the $\boldsymbol {e}_k$ is essential for achieving the completeness of $\boldsymbol {Q}_{\text {MC}}$ . However, in the MC-DINA framework, a single MC item may include multiple $\boldsymbol {e}_k$ when some distractors are also coded. As a result, the major challenge here is to ensure that the assignment of the x distinct $\boldsymbol {e}_k$ , where $1\leq x \leq K$ , to the coded options within an MC item maximizes the effect of these $\boldsymbol {e}_k$ to distinguish $2^x$ attribute profiles. Specifically and importantly, these assignments cannot be random but governed by a) a particular arrangement such that every $\boldsymbol {e}_k$ can distinguish between examinees who have mastered attribute k and those who have not and b) by the plausible and proper rules. In this regard, new development on how to organize $\boldsymbol {e}_k$ in a $\boldsymbol {Q}_{\text {MC}}$ to guarantee the completeness are required.

Before the location of $\boldsymbol {e}_k$ can be identified, it is necessary to examine how examinees are classified with specific relations between coded distractors and the key. For notational convenience, the following partial-order notation is used: for two binary vectors $\boldsymbol {a}$ and $\boldsymbol {b}$ , $\boldsymbol {a} \succeq \boldsymbol {b}$ denotes that $a_k \geq b_k$ for all k, and $\boldsymbol {a} \succ \boldsymbol {b}$ denotes that $\boldsymbol {a} \succeq \boldsymbol {b}$ and $\boldsymbol {a} \neq \boldsymbol {b}$ . Similarly, $\boldsymbol {a} \preceq \boldsymbol {b}$ and $\boldsymbol {a} \prec \boldsymbol {b}$ are defined analogously. Suppose item j contains a coded distractor, l. Given that the key is always selected when $\boldsymbol {\alpha } \succeq \boldsymbol {q}_j^{(H_j^\ast)}$ , the latent classes have three different types of ideal responses: l, $H_j^\ast$ , and $l^\prime $ , where $l^\prime \neq l$ , $l^\prime \neq H_j^\ast$ , thereby classifying examinees into three corresponding groups, as shown in Table 1. Table 1 also demonstrates that based on whether $\boldsymbol {\alpha } \succeq \boldsymbol {q}_j^{(H_j^\ast)}$ , it is obvious that item j classifies examinees into two latent classes (i.e., $G_1 + G_2$ vs. $G_3$ ). Further inspection reveals that examinees in $G_1$ have not mastered the attributes required by distractor l, and those in $G_2$ have mastered them. It is also true that if $\boldsymbol {q}_j^{(l)} \prec \boldsymbol {q}_j^{(H_j^\ast)}$ , $G_3$ consists of examinees who have mastered attributes required by distractor l. However, if $\boldsymbol {q}_j^{(l)} \not \prec \boldsymbol {q}_j^{(H_j^\ast)}$ , $G_3$ may be composed of examinees who have and have not mastered attributes required by distractor l. Hence, it is likely that item j will fail to identify all examinees who have mastered attributes required by distractor l.

Table 1

Classes formulated by distractor l and the key of item j

Table 1 Long description

The table is structured with two main header rows and one main header column.

Top Header Row:

- The first two columns are grouped under the condition: bold alpha does not succeed bold q sub j super open parenthesis H sub j star close parenthesis.

- The third column is under the condition: bold alpha succeeds bold q sub j super open parenthesis H sub j star close parenthesis.

Second Header Row (Sub-conditions for the first group):

- Column 1: bold alpha does not succeed bold q sub j super open parenthesis l close parenthesis.

- Column 2: bold alpha succeeds bold q sub j super open parenthesis l close parenthesis.

Table Body:

- Row 1 (Ideal response):

- Column 1: l prime, where l prime does not equal l and l prime does not equal H sub j star.

- Column 2: l.

- Column 3: H sub j star.

- Row 2 (Class):

- Column 1: G sub 1.

- Column 2: G sub 2.

- Column 3: G sub 3.

When $\boldsymbol {e}_{k}$ is the focus, there are three possible scenarios for $\boldsymbol {e}_k$ : a) $\boldsymbol {q}_j^{(H_j^\ast)} = \boldsymbol {e}_k$ ; b) $\boldsymbol {q}_j^{(l)} = \boldsymbol {e}_k$ and $\boldsymbol {q}_j^{(H_j^\ast)}\succ \boldsymbol {e}_k$ ; and c) $\boldsymbol {q}_j^{(l)} = \boldsymbol {e}_k$ and $\boldsymbol {e}_k \not \prec \boldsymbol {q}_j^{(H_j^\ast)}$ . Based on the above discussion, item j in cases a) and b) can divide examinees into two groups: examinees who have and have not mastered attribute k. However, in case c), item j fails to do so. To demonstrate, take $\boldsymbol {e}_{2}$ as an example. Suppose there are four items designed to measure the mastery of attribute 2 (A2) when $K=3$ , and their Q-matrices and ideal responses of latent classes are shown in Table 2. Further assume that the four items do not contain other coded distractors than those listed in Table 2. In the table, the four latent classes that have mastered A2 are listed first, followed by the other four that have not mastered A2. $\boldsymbol {Q}_1$ and $\boldsymbol {Q}_2$ are used to demonstrate case a), where $\boldsymbol {q}_j^{(H_j^\ast)} = \boldsymbol {e}_k$ ; $\boldsymbol {Q}_3$ is used to exemplify case b), where $\boldsymbol {q}_j^{(l)} = \boldsymbol {e}_k$ and $\boldsymbol {q}_j^{(H_j^\ast)}\succ \boldsymbol {e}_k$ ; and $\boldsymbol {Q}_4$ shows case c), where $\boldsymbol {q}_j^{(l)} = \boldsymbol {e}_k$ and $\boldsymbol {q}_j^{(H_j^\ast)} \not \succ \boldsymbol {e}_k$ . Item 1 is equivalent to a binary item because it does not include any coded distractor. Therefore, latent classes who have mastered A2 can be differentiated from those who have not mastered. As for item 2 where $\boldsymbol {q}_2^{(2)} = \boldsymbol {e}_2$ and $\boldsymbol {q}_2^{(1)} = \boldsymbol {e}_1$ , latent classes who have mastered A2 were classified into $G_3$ with the ideal response of 2. Those who have not mastered A2 were classified into either $G_2$ or $G_1$ with the ideal response being either 1 or 0. Therefore, a $\boldsymbol {Q}_j$ satisfying $\boldsymbol {q}_j^{(H_j^\ast)} = \boldsymbol {e}_2$ can effectively differentiate between examinees who have mastered A2 and those who have not. In $\boldsymbol {Q}_3$ , $\boldsymbol {q}_3^{(2)}\succ \boldsymbol {e}_2$ and $\boldsymbol {q}_3^{(1)} = \boldsymbol {e}_2$ . Consequently, examinees who have mastered A2 will select either the key or distractor, and those who have not mastered A2 will make a random selection among all options. Hence, $\boldsymbol {Q}_3$ can successfully distinguish between examinees who have mastered A2 and those who have not. With $\boldsymbol {Q}_4$ , examinees who have mastered both A1 and A2 (i.e., $(110)$ and $(111)$ ) and those who have not mastered A2 but mastered A1 (i.e., $(100)$ and $(101)$ ) are classified into $G_3$ . That is, mastery of A1, which is required by the key, masks the distinction between mastery and non-mastery of A2, even though $\boldsymbol {e}_2$ appears as a coded distractor. Therefore, item 4 is unable to differentiate between examinees who have mastered A2 and those who have not. In conclusion, if $\boldsymbol {e}_k$ is linked to a distractor but the key does not require attribute k, this item cannot fully distinguish between the mastery and non-mastery of attribute k.

Table 2

Ideal responses to four items assessing the mastery of A2

Table 2 Long description

The table is organized into columns for four items: Q sub 1, Q sub 2, Q sub 3, and Q sub 4.

Header Information:

- Row 1: Q sub 1, Q sub 2, Q sub 3, Q sub 4.

- Row 2: q super (H sub j super asterisk) values are 010, 010, 110, and 100 respectively.

- Row 3: q super (l) values are dash, 100, 010, and 010 respectively.

Data Rows:

- Mastery of A 2 group:

- Profile (010): Q 1 = 1, Q 2 = 2, Q 3 = 1, Q 4 = 1.

- Profile (110): Q 1 = 1, Q 2 = 2, Q 3 = 2, Q 4 = 2.

- Profile (011): Q 1 = 1, Q 2 = 2, Q 3 = 1, Q 4 = 1.

- Profile (111): Q 1 = 1, Q 2 = 2, Q 3 = 2, Q 4 = 2.

- Non-mastery of A 2 group:

- Profile (000): Q 1 = 0, Q 2 = 0, Q 3 = 0, Q 4 = 0.

- Profile (100): Q 1 = 0, Q 2 = 1, Q 3 = 0, Q 4 = 2.

- Profile (001): Q 1 = 0, Q 2 = 0, Q 3 = 0, Q 4 = 0.

- Profile (101): Q 1 = 0, Q 2 = 1, Q 3 = 0, Q 4 = 2.

The above discussion and examples show that the inclusion of $\boldsymbol {e}_k$ alone is not sufficient to constitute a complete $\boldsymbol {Q}_{\text {MC}}$ ; the assignment of $\boldsymbol {e}_k$ is also crucial to its completeness. In particular, Table 2 illustrates that an item may contain $\boldsymbol {e}_k$ among its coded options but still fail to distinguish mastery from non-mastery of attribute k when the key masks this distinction. Hence, to establish the completeness of $\boldsymbol {Q}_{\text {MC}}$ , the level-x design is first developed and defined to properly organize $\boldsymbol {e}_k$ in $\boldsymbol {Q}_{j}$ .

The intuition underlying the level-x design is combinatorial. Because x binary attributes generate $2^x$ latent mastery patterns, an MC item intended to distinguish all mastery patterns generated by these x attributes must induce $2^x$ distinct ideal-response states. Since the all-zero mastery pattern corresponds to the noncoded option, at least $2^x-1$ coded options are required. However, this counting requirement alone is not sufficient: the corresponding q-vectors must also be arranged so that the item actually distinguishes the $2^x$ latent classes. This requirement motivates the following definition.

Two common designs in practice, the level-1 and level-2 designs, are discussed in detail below. Additionally, the level-3 design is briefly introduced as an illustration for understanding the level-x design. The section is wrapped up with a discussion of the level-0 design, a crucial case where the key masks the distinction established by $\boldsymbol {e}_k$ . Lemma 1 is then presented as a remedy for the level-0 design to differentiate examinees who have mastered attribute k from those who have not. Because the level-0 design is a case requiring special treatment, it is not included when referring to the level-x design in this study.

3.1.2 Level-2 design

A level-2 design allows a $\boldsymbol {Q}_j$ to identify the mastery of two attributes, say k and $k^\prime $ , where $k\neq k^\prime $ , so that $\boldsymbol {Q}_j$ can effectively distinguish examinees into four groups (i.e., $2^2=4$ ): those who have mastered only attribute k, only attribute $k^\prime $ , both attributes k and $k^\prime $ , and neither attributes k nor $k^\prime $ . Notice that although $\boldsymbol {Q}_2$ and $\boldsymbol {Q}_4$ contain two distinct $\boldsymbol {e}_{k}$ , they do not have a level-2 design because they fail to identify examinees who have mastered both A1 and A2. The only way to achieve a level-2 design for these items is to define a coded option as $\boldsymbol {e}_1+\boldsymbol {e}_{2}$ subjective to $\boldsymbol {e}_1+\boldsymbol {e}_{2} \preceq \boldsymbol {q}_j^{(H_j^\ast)}$ . Consequently, $\boldsymbol {Q}_j$ must contain the following structure:

(4) $$ \begin{align} \begin{pmatrix} 1 & 1 & \mathbf{0}_{K-2} \\ 1 & 0 & \mathbf{0}_{K-2} \\ 0 & 1 & \mathbf{0}_{K-2} \\ \end{pmatrix}. \end{align} $$

This means an item conforming to the level-2 design should include not only $\boldsymbol {e}_k$ and $\boldsymbol {e}_{k^\prime }$ but also $\boldsymbol {e}_k+\boldsymbol {e}_{k^\prime }$ . Note that an MC item whose Q-matrix has a level-2 design may include additional coded options as long as the item is proper and useful.

3.1.3 Level-3 design

Likewise, a $\boldsymbol {Q}_j$ with the level-3 design can differentiate among examinees who have mastered only one of the three attributes, namely, k, $k^\prime $ , or $k^{\prime \prime }$ , those who have mastered any two of them, all of them, and none of them, where $k \neq k^{\prime } \neq k^{\prime \prime }$ . After column swapping, a Q-matrix with a level-3 design has the following structure:

$$ \begin{align*} \begin{pmatrix} 1 & 1 & 1 & 0 & \dots & 0 \\ 1 & 1 & 0 & 0 & \dots & 0 \\ 1 & 0 & 1 & 0 & \dots & 0 \\ 0 & 1 & 1 & 0 & \dots & 0 \\ 1 & 0 & 0 & 0 & \dots & 0 \\ 0 & 1 & 0 & 0 & \dots & 0 \\ 0 & 0 & 1 & 0 & \dots & 0 \\ \end{pmatrix}. \end{align*} $$

However, in practice, it is not common for an MC item to have seven or more coded options. Therefore, the level-3 design is not further discussed in the current article.

3.1.4 Level-0 design

As discussed above, a $\boldsymbol {Q}_j$ with the level-x design, for $x\geq 1$ , not only includes $\boldsymbol {e}_k$ but also distinguishes examinees who have mastered attribute k from those who have not. However, including an $\boldsymbol {e}_k$ does not necessarily guarantee the conformation of the level-1 design. A $\boldsymbol {Q}_j$ that contains an $\boldsymbol {e}_k$ but fails to make the distinction is said to have the level-0 design. Based on Definition 1, a $\boldsymbol {Q}_j$ has a level-0 design if distractor l is coded as $\boldsymbol {e}_k$ but $\boldsymbol {q}_{j}^{(H_{j^*})} \not \succ \boldsymbol {e}_k$ , as shown in the above case c). $\boldsymbol {Q}_4$ in Table 2, for example, is an item that has such a design. Therefore, to distinguish examinees who have mastered attribute k from those who have not, it is recommended to begin with a $\boldsymbol {Q}_j$ that contains $\boldsymbol {e}_k$ and has a level-x design. However, if such a $\boldsymbol {Q}_j$ is not available and $\boldsymbol {e}_k$ is included in items that only have a level-0 design, the immediate question is that whether it is possible to separate examinees who have and have not mastered attribute k apart by attaching more items to a level-0-design item. If the answer is yes, what is the procedure one should follow? Lemma 1 is developed to answer the above question. Before it is presented, an example, depicted in Figure 1, is used to illustrate and facilitate the reasoning.

Figure 1

An example set of items to distinguish between mastery and non-mastery of A2.

Figure 1 Long description

The flowchart begins at the top with a rectangular box containing (*, *, *). An arrow points down to a second box labeled Item 4: q sub 4 super (2) = (1, 0, 0), q sub 4 super (1) = (0, 1, 0). Three arrows branch from Item 4:

- Arrow G sub 2 points left to a diamond containing (0, 1, *).

- Arrow G sub 3 points down to a diamond containing (1, *, *).

- Arrow G sub 1 points right to a diamond containing (0, 0, *).

From the (1, *, *) diamond, an arrow points down to a third box labeled Item 5: q sub 5 super (2) = (1, 0, 1), q sub 5 super (1) = (1, 1, 0). Three arrows branch from Item 5:

- Arrow G sub 2 points left to a diamond containing (1, 1, 0).

- Arrow G sub 3 points down to a diamond containing (1, *, 1).

- Arrow G sub 1 points right to a diamond containing (1, 0, 0).

From the (1, *, 1) diamond, an arrow points down to a fourth box labeled Item 6: q sub 6 super (1) = (1, 1, 1). Two arrows branch from Item 6:

- Arrow G sub 3 points down to a diamond containing (1, 1, 1).

- Arrow G sub 1 points right to a diamond containing (1, 0, 1).

Revisit $\boldsymbol {Q}_4$ in Table 2. This item conforms to the level-0 design with the key and a coded distractor defined by $\boldsymbol {q}_4^{(2)}=(1,0,0)$ and $\boldsymbol {q}_4^{(1)}=(0,1,0)$ , respectively. After the item is administered, both $(1,1,\ast)$ and $(1,0,\ast)$ , where $\ast$ denotes an entry either 0 or 1, are assigned to $G_3$ because the item cannot separate them apart based on A2 (see Table 1). Notice that in this case, whether or not the next item requires A1 does not affect the differentiation between $(1,1,\ast)$ and $(1,0,\ast)$ because both have already mastered it. However, the requirement of A3 will mask the mastery of the A2 between $(1, 1, 0)$ and $(1, 0, 0)$ . Hence, to distinguish between $(1, 1, \ast)$ and $(1, 0, \ast)$ , an item with a coded option defined by $(\ast,1,0)$ is needed. If $(\ast,1,0)$ defines the key, then $(1,1,\ast)$ will be assigned to $G_3$ , and $(1,0,\ast)$ will be assigned to $G_1$ or $G_2$ . As a result, all the examinees can be separated apart based on A2. To illustrate a more complex scenario, assume that such an item is not available in the test.

Figure 1 illustrates how additional items are used to evaluate mastery of A2. The labels $G_1$ , $G_2$ , and $G_3$ shown near the arrows denote the groups defined in Table 1. Item 5 with a coded distractor and the key defined by $(1,1,0)$ and $(1,0,1)$ , respectively, is presented to the examinees. After this item is administered, $(1,1,1)$ and $(1,0,1)$ are still mixed in $G_3$ and thus, more items are needed to separate them apart based on A2. Again, whether the next item requires A1 and A3 does not influence the differentiation between $(1,1,1)$ and $(1,0,1)$ . The requirement is that this item should consist of a coded option defined by $(\ast,1,\ast)$ . Item 6 has the key defined by $(1,1,1)$ and thereby can be used to distinguish $\boldsymbol {\alpha }$ from $\boldsymbol {\alpha }^\ast$ . Differing from the level-1 and level-2 designs, where only one item is needed to distinguish the examinees who have and have not mastered A2, in this example, two more items are required when the item containing $\boldsymbol {e}_k$ has a level-0 design.

The example above can be generalized as follows. Suppose item j contains $\boldsymbol {e}_k$ with a level-0 design used to distinguish the mastery of attribute k. After this item is administered, examinees are categorized into three classes, as shown in Table 1: $G_1$ (not mastered attribute k), $G_2$ (mastered attribute k), and $G_3$ (a mix of $\boldsymbol {\alpha } \succeq \boldsymbol {e}_k + \boldsymbol {q}_j^{(H_j^\ast)}$ and $\boldsymbol {\alpha }^\ast \succeq \boldsymbol {q}_j^{(H_j^\ast)}$ but $\boldsymbol {\alpha }^\ast \not \succ \boldsymbol {e}_k$ ). Here, $G_3$ consists of examinees who have and have not mastered attribute k and thus, additional items are required to distinguish between $\boldsymbol {\alpha }$ and $\boldsymbol {\alpha }^\ast$ in $G_3$ .

Let $\boldsymbol {q} =\boldsymbol {q}_j^{(H_j^\ast)}$ denote the attributes that examinees in $G_3$ have already mastered, which do not contribute to distinguishing between $\boldsymbol {\alpha }$ and $\boldsymbol {\alpha }^\ast$ . From the remaining items, select an item that has a coded option defined by $\boldsymbol {e}_k+\boldsymbol {q}_0$ , where $\boldsymbol {q}_0 \preceq \boldsymbol {q}$ . This item is required to differentiate $\boldsymbol {\alpha }$ from $\boldsymbol {\alpha }^\ast$ . Denote this item as $j^{\prime }$ . If $\boldsymbol {q}_{j^{\prime }}^{(H_{j^{\prime }}^*)} \succ \boldsymbol {e}_k$ , based on Table 1, this item can explicitly divide examinees in $G_3$ into two groups based on the mastery of attribute k. However, if $\boldsymbol {q}_{j^{\prime }}^{(H_{j^{\prime }}^*)} \not \succeq \boldsymbol {e}_k$ , $\boldsymbol {\alpha } \succeq \boldsymbol {e}_k + \boldsymbol {q}_0 + \boldsymbol {q}_{j^{\prime }}^{(H_{j^{\prime }}^*)}$ and $\boldsymbol {\alpha }^\ast \succeq \boldsymbol {q}_0 + \boldsymbol {q}_{j^{\prime }}^{(H_{j^{\prime }}^*)}$ but $\boldsymbol {\alpha }^\ast \not \succ \boldsymbol {e}_k$ will be left in $G_3$ and remain un-separated. Update $\boldsymbol {q} =\boldsymbol {q} \lor \boldsymbol {q}_{j^{\prime }}^{(H_{j^{\prime }}^*)}$ . Thus, an item who has a coded option defined by $\boldsymbol {e}_k+\boldsymbol {q}_0$ , where $\boldsymbol {q}_0 \preceq \boldsymbol {q}$ , is required to distinguish $\boldsymbol {\alpha }$ from $\boldsymbol {\alpha }^\ast$ . The process continues with updating $\boldsymbol {q}_0$ and finding additional items until the mastery of attribute k is determined.

The process detailed above outlines the need for additional items to assess the mastery of attribute k when dealing with an item characterized by a level-0 design. Lemma 1, shown below, not only summarizes this process, but makes it more comprehensive, as item $j^\prime $ might not be unique in practice. The proof of Lemma 1 is presented in Appendix A.

Lemma 1. Suppose $\boldsymbol {Q}_{j_0}$ of item $j_0$ involving $\boldsymbol {e}_k$ conforms to the level-0 design; that is, $\boldsymbol {q}_{j_0}^{(H_{j_0}^*)} \not \succ \boldsymbol {e}_k$ . Consider the following algorithm:

1. Step 0. Let $\boldsymbol {q}=\boldsymbol {q}_{j_0}^{(H_{j_0}^*)}$ and initialize set $\mathcal {J}=\{j_0\}$ .

2. Step 1. For the rest of test items, define set $\mathcal {J}_k$ as $\mathcal {J}_k = \{j | \text { item } j$ contains at least one coded option defined by $\boldsymbol {e}_k + \boldsymbol {q}_0, \text { where } \boldsymbol {q}_0 \preceq \boldsymbol {q}\}$ . If $\mathcal {J}_{k} = \emptyset $ , stop; otherwise, update $\mathcal {J} = \mathcal {J} \cup \mathcal {J}_k$ and proceed to Step 2.

3. Step 2. If there exists a $j \in \mathcal {J}_{k}$ such that $\boldsymbol {q}_{j}^{(H_{j}^*)} \succ \boldsymbol {e}_k$ , stop; otherwise, update $\boldsymbol {q} =\boldsymbol {q} \lor \big (\bigvee _{t=1}^{|\mathcal {J}_{k}|}\boldsymbol {q}_{j_t}^{(H_{j_t}^*)}\big )$ and repeat Steps 1 and 2.

Examinees who have and have not mastered attribute k can be separated apart by administering all the items in set $\mathcal {J}$ if and only if the above algorithm stops with $\mathcal {J}_{k} \neq \emptyset $ .

It should be noted that Lemma 1 provides a constructive separation procedure rather than an optimized search algorithm. If Lemma 1 is satisfied, then along any successful path, $\boldsymbol {q}$ can be enlarged at most $K-||\boldsymbol {q}_{j_0}^{(H_{j_0}^*)}||$ times. Therefore, the length of a successful path is bounded above by $K-||\boldsymbol {q}_{j_0}^{(H_{j_0}^*)}|| + 1$ . However, when Step 1 yields multiple admissible candidate items, the lemma does not specify a deterministic choice rule that identifies in advance which candidate leads to termination. As a result, the total search cost may be substantially larger than the depth of a single successful path. Moreover, the number of items satisfying the Step 1 criterion depends on $\boldsymbol {q}_0$ . In particular, when these $\boldsymbol {q}_0$ have overlapping attributes, many items may be in $\mathcal {J}_k = \{j | \boldsymbol {Q}_{j} \text { contains } \boldsymbol {e}_k + \boldsymbol {q}_0, \text { where } \boldsymbol {q}_0 \preceq \boldsymbol {q}\}$ . Without additional assumptions on the item bank, the general worst-case bound on the number of items involved remains J.

Overall, a level-x design is characterized not only by the inclusion of x distinct $\boldsymbol {e}_k$ in $\boldsymbol {Q}_j$ , but more importantly, by the capability of $\boldsymbol {Q}_j$ to differentiate among $2^x$ latent classes constituted by these x attributes. Under the MC-DINA model, when $\boldsymbol {Q}_j$ has a level-x design, where $x\geq 1$ , the essential rule is to include all $\boldsymbol {e}_k$ , as well as the sums of any two, three, and eventually all of $\boldsymbol {e}_k$ . Additional coded options that are not required for achieving this level-x distinction may also be included. Such options do not change the minimum coded-option structure needed for the level-x design, but they must still satisfy the proper and plausible conditions discussed earlier. A full treatment of how to construct additional coded options is beyond the scope of the present study. Table 3 displays the minimum $H_j^\ast$ required to achieve a level-x design, calculated by $2^x-1$ . In addition, the inequality below can be constructed to describe the relationship between $H_j$ and x:

$$ \begin{align*} 2^x-1 &\leq H_j, \end{align*} $$

where $x \geq 1$ . Further details about the relation between $2^x$ and $H_j$ can be found in Wang et al. (Reference Wang, Chiu and Köhn2025).

Table 3

Relationship between the level-x design and minimum $H_j^\ast$

Table 3 Long description

The table consists of two columns. The first column is titled Level-x design and the second column is titled Minimum H sub x. The data is presented in four rows as follows. Row one: Level-x design 1 corresponds to a minimum H sub x of 1. Row two: Level-x design 2 corresponds to a minimum H sub x of 3. Row three: Level-x design 3 corresponds to a minimum H sub x of 7. Row four: Level-x design 4 corresponds to a minimum H sub x of 15.

3.2 Conditions for the completeness of $\boldsymbol {Q}_{\text {MC}}$ under the MC-DINA model

The definition of completeness established for $\boldsymbol {Q}_{\text {B}}$ under the DINA model is applicable to $\boldsymbol {Q}_{\text {MC}}$ , provided specific setups and modifications. Notice that the q-vectors of the coded options in the MC context are only partially ordered, and as a result, $E(X|\boldsymbol {\alpha })$ calculated based on the original scale is no longer meaningful and needs to be recoded such that each individual level of coded option is treated as a random variable. Specifically, let $I_{jl}$ be an indicator variable for response level l of item j. It can then be expressed as follows:

$$ \begin{align*} I_{jl} = \begin{cases} 1, & \text{if }l\text{ is selected},\\ 0, & \text{if }l\text{ is not selected}. \end{cases} \end{align*} $$

Because $I_{jl}$ is binary, it follows that

$$ \begin{align*}E(I_{jl} | \boldsymbol{\alpha})=P(I_{jl} = 1 | \boldsymbol{\alpha})=P(X_{j}=l|\boldsymbol{\alpha}). \end{align*} $$

Let $\boldsymbol {\varsigma }(\boldsymbol {\alpha }) = E(\boldsymbol {I}|\boldsymbol {\alpha })$ , where $\boldsymbol {I} = (I_{11},\ldots ,I_{JH_j^\ast})^\prime $ . Here, $\boldsymbol {\varsigma }(\boldsymbol {\alpha })$ is no longer a $J\times 1$ as in the context of binary responses but rather a $\sum _{j=1}^{J}H_j^\ast \times 1$ vector in the MC context. Based on this setup, Definition 2 is developed to formally describe the completeness of $\boldsymbol {Q}_{\text {MC}}$ under the MC-DINA model.

Definition 2 is then used to establish the sufficient and necessary conditions for the completeness of $\boldsymbol {Q}_{\text {MC}}$ , as formally formulated in Theorem 1. This theorem is inspired by the condition for the completeness of $\boldsymbol {Q}_{\text {B}}$ , reiterated here that $\boldsymbol {Q}_{\text {B}}$ is complete if and only if $\boldsymbol {Q}_{\text {B}}$ includes all the $K\, \boldsymbol {e}_k$ (for more information, please see Section 2.2). However, differing from the binary case, where one item can only associate with at most one $\boldsymbol {e}_k$ , an MC item may involve multiple $\boldsymbol {e}_k$ , either through the key or coded distractors and thus, additional constraints, as depicted in Theorem 1, are required for organizing $\boldsymbol {e}_k$ within an MC item. At this point, it is useful to distinguish three levels of discussion. First, the level-x design is an item-level concept that describes what a particular item submatrix $\boldsymbol {Q}_j$ can distinguish. Thus, when an item is said to have a level-x design, this refers to the level-x structure of its corresponding $\boldsymbol {Q}_j$ . Second, conditions (C1) and (C2) are evaluated separately for each attribute rather than for the test as a whole. Third, completeness is a test-level property of the full $\boldsymbol {Q}_{\text {MC}}$ and is achieved when the required distinction is established for every attribute. Therefore, within the same test, some attributes may satisfy condition (C1), whereas others may satisfy condition (C2), while completeness is still guaranteed. The proof of Theorem 1 is presented in Appendix A.

The flexibility in organizing $\boldsymbol {e}_k$ to achieve Q-matrix completeness, as described in Theorem 1, may present some challenges for practitioners. Hence, to improve its accessibility, a flowchart illustrating the procedure of how the theorem can be applied to evaluate the completeness of $\boldsymbol {Q}_{\text {MC}}$ is provided in Figure 2. From a test-construction perspective, it is generally preferable to include items with level-1 or higher level-x designs when possible. Lemma 1 is therefore not intended as a primary item-design recommendation. Rather, it is mainly a theoretical device that allows the completeness of an existing MC Q-matrix to be evaluated when some attributes appear only in level-0 items. This situation may arise in retrospective applications, where the test has already been constructed and administered.

Figure 2

Flowchart for evaluating the completeness of $\boldsymbol {Q}_{\text {MC}}$ .

Theorem 1 further indicates that if the q-vectors of the item keys contain all possible $\boldsymbol {e}_k$ (i.e., Level-1 design), then $\boldsymbol {Q}_{\text {MC}}$ is complete, regardless of the q-vectors associated with the coded distractors. This result can serve as a first-line criterion for users to quickly assess the completeness of $\boldsymbol {Q}_{\text {MC}}$ before applying the full procedure outlined in Theorem 1. The following Corollary 1 is established to outline this practically useful guideline.

It is worth noting that a pure level-1 $\boldsymbol {Q}_{\text {MC}}$ does not necessarily have to be constructed as a direct augmentation of a complete $\boldsymbol {Q}_{\text {B}}$ , because a level-1 design does not require $\boldsymbol {e}_k$ to define the key. Rather, it only requires the item to distinguish examinees who have mastered a particular attribute from those who have not. For example, if the key is defined by $(1,1,0)$ and the sole coded distractor is defined by $(1,0,0)$ , then the item has a level-1 design with respect to A1, because it separates examinees who have mastered A1 from those who have not. Therefore, an MC item with a level-1 design functions analogously to a binary item for the purpose of distinguishing mastery versus non-mastery of the target attribute, regardless of whether $\boldsymbol {e}_k$ defines the key or a coded distractor.

Two complete $\boldsymbol {Q}_{\text {MC}}$ examples are presented in Table 4. Same as previous examples, within each MC item, the highest-numbered coded option is the key, whereas lower-numbered coded options are coded distractors. Both $\boldsymbol {Q}_{\text {MC1}}$ and $\boldsymbol {Q}_{\text {MC2}}$ include all possible $\boldsymbol {e}_k$ and they satisfy (C1) and (C2), respectively. Specifically, all three attributes in $\boldsymbol {Q}_{\text {MC}1}$ satisfy (C1), whereas in $\boldsymbol {Q}_{\text {MC}2}$ , $\alpha _1$ and $\alpha _2$ satisfy (C1) and $\alpha _3$ satisfies (C2). Their corresponding expected observed responses for each latent class, $\boldsymbol {\varsigma }(\boldsymbol {\alpha })$ , are presented in Table 5. The results show that the $\boldsymbol {\varsigma }(\boldsymbol {\alpha })$ across all $\boldsymbol {\alpha }$ are distinct under the MC-DINA assumption $1 - \sum _{c\neq l} \epsilon _{jcl}> \epsilon _{jcl}$ .

Table 4

Examples of complete $\boldsymbol {Q}_{\text {MC}}$ with $K=3$ and $H_j=5$

Table 4 Long description

The table is split into two main sections: Q sub M C 1 on the left and Q sub M C 2 on the right. Both sections have columns for Item, Option, alpha sub 1, alpha sub 2, and alpha sub 3.

In the Q sub M C 1 section:

* Row 1: Item 1, Option 2, alpha sub 1 is 1, alpha sub 2 is 0, alpha sub 3 is 0.

* Row 2: Item 1, Option 1, alpha sub 1 is 0, alpha sub 2 is 1, alpha sub 3 is 0.

* Row 3: Item 2, Option 3, alpha sub 1 is 0, alpha sub 2 is 1, alpha sub 3 is 1.

* Row 4: Item 2, Option 2, alpha sub 1 is 0, alpha sub 2 is 1, alpha sub 3 is 0.

* Row 5: Item 2, Option 1, alpha sub 1 is 0, alpha sub 2 is 0, alpha sub 3 is 1.

In the Q sub M C 2 section:

* Row 1: Item 1, Option 1, alpha sub 1 is 1, alpha sub 2 is 0, alpha sub 3 is 0.

* Row 2: Item 2, Option 1, alpha sub 1 is 0, alpha sub 2 is 1, alpha sub 3 is 0.

* Row 3: Item 3, Option 2, alpha sub 1 is 1, alpha sub 2 is 1, alpha sub 3 is 0.

* Row 4: Item 3, Option 1, alpha sub 1 is 0, alpha sub 2 is 0, alpha sub 3 is 1.

* Row 5: Item 4, Option 1, alpha sub 1 is 1, alpha sub 2 is 1, alpha sub 3 is 1.

Table 5

$\boldsymbol {\varsigma }(\boldsymbol {\alpha })$ of items shown in Table 4

Table 5 Long description

The table is organized with a header row containing eight alpha conditions: (000), (100), (010), (001), (110), (101), (011), and (111). The rows are grouped by Q sub M C 1 and Q sub M C 2.

Row 1 (E of I sub 11 given alpha): 1/5, epsilon sub 112, 1 minus summation of c not equal to 1 of epsilon sub 1 c 1, 1/5, epsilon sub 112, epsilon sub 112, 1 minus summation of c not equal to 1 of epsilon sub 1 c 1, epsilon sub 112.

Row 2 (E of I sub 12 given alpha): 1/5, 1 minus summation of c not equal to 2 of epsilon sub 1 c 2, epsilon sub 121, 1/5, 1 minus summation of c not equal to 2 of epsilon sub 1 c 2, 1 minus summation of c not equal to 2 of epsilon sub 1 c 2, epsilon sub 121, 1 minus summation of c not equal to 2 of epsilon sub 1 c 2.

Row 3 (E of I sub 21 given alpha): 1/5, 1/5, epsilon sub 212, 1 minus summation of c not equal to 1 of epsilon sub 2 c 1, epsilon sub 212, 1 minus summation of c not equal to 1 of epsilon sub 2 c 1, epsilon sub 213, epsilon sub 213.

Row 4 (E of I sub 22 given alpha): 1/5, 1/5, 1 minus summation of c not equal to 2 of epsilon sub 2 c 2, epsilon sub 221, 1 minus summation of c not equal to 2 of epsilon sub 2 c 2, epsilon sub 221, epsilon sub 223, epsilon sub 223.

Row 5 (E of I sub 23 given alpha): 1/5, 1/5, epsilon sub 232, epsilon sub 231, epsilon sub 232, epsilon sub 231, 1 minus summation of c not equal to 3 of epsilon sub 2 c 3, 1 minus summation of c not equal to 3 of epsilon sub 2 c 3.

Row 6 (E of I sub 11 given alpha under Q sub M C 2): 1/5, 1 minus summation of c not equal to 1 of epsilon sub 1 c 1, 1/5, 1/5, 1 minus summation of c not equal to 1 of epsilon sub 1 c 1, 1 minus summation of c not equal to 1 of epsilon sub 1 c 1, 1/5, 1 minus summation of c not equal to 1 of epsilon sub 1 c 1.

Row 7 (E of I sub 21 given alpha under Q sub M C 2): 1/5, 1/5, 1 minus summation of c not equal to 1 of epsilon sub 2 c 1, 1/5, 1 minus summation of c not equal to 1 of epsilon sub 2 c 1, 1/5, 1 minus summation of c not equal to 1 of epsilon sub 2 c 1, 1 minus summation of c not equal to 1 of epsilon sub 2 c 1.

Row 8 (E of I sub 31 given alpha): 1/5, 1/5, 1/5, 1 minus summation of c not equal to 1 of epsilon sub 3 c 1, epsilon sub 312, 1 minus summation of c not equal to 1 of epsilon sub 3 c 1, 1 minus summation of c not equal to 1 of epsilon sub 3 c 1, epsilon sub 312.

Row 9 (E of I sub 32 given alpha): 1/5, 1/5, 1/5, epsilon sub 321, 1 minus summation of c not equal to 2 of epsilon sub 3 c 2, epsilon sub 321, epsilon sub 321, 1 minus summation of c not equal to 2 of epsilon sub 3 c 2.

Row 10 (E of I sub 41 given alpha): 1/5, 1/5, 1/5, 1/5, 1/5, 1/5, 1/5, 1 minus summation of c not equal to 1 of epsilon sub 4 c 1.

These examples also illustrate that a complete $\boldsymbol {Q}_{\text {MC}}$ does not need to consist of items with the same level-x design. Instead, the level structure may be mixed across items. For example, $\boldsymbol {Q}_{\text {MC}1}$ includes both level-1 and level-2 designs, whereas $\boldsymbol {Q}_{\text {MC}2}$ includes level-1 and level-0 designs. In general, for each attribute k, the test must contain either an item with a level-x design that directly separates mastery from non-mastery of attribute k, or a collection of level-0 items satisfying Lemma 1. Therefore, the completeness of $\boldsymbol {Q}_{\text {MC}}$ is not determined by whether the test uses a pure level-x design, but by whether all attributes are distinguishable under (C1) or (C2).

Notice that $\boldsymbol {Q}_{\text {MC1}}$ requires fewer items than a complete $\boldsymbol {Q}_{\text {B}}$ with $K=3$ items, while $\boldsymbol {Q}_{\text {MC2}}$ requires more items, which means the flexibility in organizing $\boldsymbol {e}_k$ may either reduce or increase the test length to achieve a complete $\boldsymbol {Q}_{\text {MC1}}$ . If item j containing $\boldsymbol {e}_k$ satisfies (C1), only one item is needed to differentiate the examinees who have mastered attribute k from those who have not. However, satisfying (C2) requires more items than (C1) to make the distinction. Therefore, the more $\boldsymbol {e}_k$ are contained in items with a level-x design, the shorter the test needed to achieve the completeness of the $\boldsymbol {Q}_{\text {MC}}$ . In general, placing multiple $\boldsymbol {e}_k$ within higher-level designs can reduce the number of items needed for completeness, because a single item may contribute to distinguishing mastery patterns for more than one attribute. In contrast, if $\boldsymbol {Q}_{\text {MC}}$ consists only of level-1 MC items, then, as in the binary item case, at least K items are needed to achieve completeness. Conversely, the more $\boldsymbol {e}_k$ contained in items with a level-0 design, the longer the test required to achieve the completeness of the $\boldsymbol {Q}_{\text {MC}}$ .

3.3 Minimum test length for complete $\boldsymbol {Q}_{\text {MC}}$ under the MC-DINA model

In situations where minimizing the number of items in a test is desired, such as tests with time constraints, two immediate questions arise: what is the minimal number of required items to achieve the completeness of $\boldsymbol {Q}_{\text {MC}}$ , and is there a particular design that test developers must adopt? Corollary 2 is developed to answer these questions.

The level-x design is specifically devised for MC items, allowing the incorporation of multiple $\boldsymbol {e}_k$ within a single MC item. To reduce the number of test items, it is optimal for MC items to utilize level-2 or higher level-x designs. Given that $3 \leq H_j <7$ appears to be the commonly used range for $H_j$ in practice, the maximum achievable level of x is 2, as indicated by Equation (4). Therefore, to achieve the minimum test length with $3 \leq H_j <7$ , $\lfloor \frac {K}{2} \rfloor $ of the MC items should employ level-2 designs. Specifically, when K is even, the K attributes can be partitioned into $\frac {K}{2}$ disjoint pairs, and each pair can be measured by one level-2 MC item. Therefore, the minimum test length is $\frac {K}{2} = \lceil \frac {K}{2}\rceil $ . When K is odd, $\frac {K-1}{2}$ level-2 MC items can be used to measure $K-1$ attributes in disjoint pairs, and one additional level-1 or level-2 MC item is needed to measure the remaining attribute. Thus, the minimum test length is $\frac {K-1}{2} + 1 = \lceil \frac {K}{2}\rceil $ . For instance, $\boldsymbol {Q}_{\text {MC}1}$ shown in Table 4 represents a complete Q-matrix with the minimum test length $\lceil \frac {3}{2} \rceil = 2$ .

4.1 Design

Two tests designed to measure four attributes ( $K=4$ ), resulting in a total of $4^2=16$ latent classes, were implemented. Both tests contained four MC items ( $J=4$ ) with five options ( $H_j=5$ ). According to Corollary 1, a complete Q-matrix for the DINA model must be complete for the MC-DINA model. Therefore, $\boldsymbol {Q}_{\text {MC1}}$ was designed in such a way that it consisted of all the $\boldsymbol {e}_k$ , where $k=1,\ldots ,4$ , and $H_j^\ast=1$ across all items. $\boldsymbol {Q}_{\text {MC2}}$ was constructed according to Theorem 1, where all distinct $K\, \boldsymbol {e}_k$ were included and each $\boldsymbol {Q}_j$ follows a level-2 design. While Corollary 2 indicates that two items are sufficient for a complete $\boldsymbol {Q}_{\text {MC}}$ with level-2 designs, the two items with this design were repeated to equalize the test length for a fair comparison between $\boldsymbol {Q}_{\text {MC1}}$ and $\boldsymbol {Q}_{\text {MC2}}$ , given that $\boldsymbol {Q}_{\text {MC1}}$ requires four items for completeness. $\boldsymbol {Q}_{\text {MC2}}$ is provided below:

$$ \begin{align*} \boldsymbol{Q}_{\text{MC2}} = \left( \begin{array}{c} \boldsymbol{Q}_1\\ \hdashline \boldsymbol{Q}_2\\ \hdashline \boldsymbol{Q}_{3}\\ \hdashline \boldsymbol{Q}_{4} \end{array}\right) = \left(\begin{array}{cccc} 1 & 1 & 0 & 0 \\ 1 & 0 & 0 & 0 \\ 0 & 1 & 0 & 0 \\ \hdashline 0 & 0 & 1 & 1 \\ 0 & 0 & 1 & 0 \\ 0 & 0 & 0 & 1 \\ \hdashline 1 & 1 & 0 & 0 \\ 1 & 0 & 0 & 0 \\ 0 & 1 & 0 & 0 \\ \hdashline 0 & 0 & 1 & 1 \\ 0 & 0 & 1 & 0 \\ 0 & 0 & 0 & 1 \\ \end{array}\right). \end{align*} $$

Here, $\boldsymbol {Q}_{\text {MC1}}$ served as a benchmark to assess the effectiveness of $\boldsymbol {Q}_{\text {MC2}}$ as a complete Q-matrix.

For each latent class, 10,000 response vectors ( $N=10{,}000$ ) were generated using the MC-DINA model. Specifically, the likelihood that examinee i in group $\eta _{ij}=l'$ made a response $X_{ij} = l$ for item j, denoted as $P(X_{ij} = l|\eta _{ij}=l')$ , is expressed as follows:

$$ \begin{align*} P(X_{ij} = l|\eta_{ij}=l') = \begin{cases} \frac{1}{H_j} + \frac{H_j - H_j^\ast - 1}{H_j}I[l = 0], \quad & \text{if } l' = 0,\\ \lambda_j, \quad & \text{if } l'> 0 \text{ and } l = l',\\ \frac{1 - \lambda_j}{H_j - 1}, \quad & \text{if } l' > 0 \text{ and } l \neq l'. \end{cases} \end{align*} $$

In this equation, the parameter $\lambda _j$ signifies the probability that an examinee’s observed response was consistent with the nonzero ideal response. This parameter was randomly sampled from a uniform distribution, $\mathcal {U}(0.74,0.9)$ . If slipping occurred, the examinee had an equal probability of $\frac {1-\lambda _j}{H_j-1}$ to select any of the remaining options. Furthermore, examinees whose ideal response was zero made a random selection from all $H_j$ options. A previous study (Wang et al., Reference Wang, Chiu and Köhn2023) showed that the MC-DINA model may encounter severe convergence difficulties when the sample size is small. This occurs because some response options may never be selected by any examinees, which leads to failed parameter estimation. In the present simulation, each set of 10,000 response vectors was generated conditional on a single latent class. Under such a design, it is reasonable to expect a similar situation in which certain response options are not selected by any examinees, which may likewise lead to convergence difficulties for the MC-DINA model. To address this issue, the MC-NPC method, rather than the MC-DINA model, was employed to estimate the attribute profiles. This choice is theoretically justified because, as discussed earlier, MC-NPC is a nonparametric counterpart of the MC-DINA model that relaxes the nestedness constraint imposed in Reference de la Torrede la Torre’s (Reference de la Torre2009) MC-DINA model. Moreover, the implementation of the MC-DINA model in the GDINA package in R (i.e., the MCmodel() function) does not impose this nestedness constraint, as it allows coded distractors to require attributes that are not required by the key. Therefore, MC-NPC is appropriate for analyzing data generated under this relaxed MC-DINA framework.

4.2 Evaluation criteria

The proportion of examinees classified into the true latent class was calculated after each item was administered. In the current simulation study, the sample size ( $N=10{,}000$ ) was sufficiently large that the observed proportion provides an approximation of $P(\hat {\boldsymbol {\alpha }} = \boldsymbol {\alpha }|\boldsymbol {\alpha })$ .