Why Create an Exam?

Prior to the development of the examination, all students at our institution completed a thesis as their summative project for the master of arts in bioethics (MAB). In some ways, the thesis worked well: it enabled students to delve deeply into a particular issue that interested them during the course of the program, it allowed them to demonstrate their research and analysis skills, and it required them to demonstrate a sensitivity to context that reflects important attitudinal components of bioethics.^{Footnote 10}

Yet for some students, writing a thesis was problematic. For one thing, the thesis process takes about a year to complete, and this time frame was a challenge for students who were completing the degree as a “gap-year activity” in between other educational programs. In addition, many dual-degree students stepped out of other programs for only a year and therefore had to be able to complete all of the requirements within a 12-month period. Finally, for some students, a comprehensive exam was an attractive option because it enabled them to review the full breadth of knowledge and skills they had acquired during the program, rather than narrowing focus to a single thesis topic.

For the faculty, offering students an examination rather than a thesis presented an intriguing opportunity. Even a high-quality thesis might not provide sufficient evidence that a student had mastered any more than one particular issue in bioethics. A comprehensive exam could give insight into whether the student had met the program’s content and process objectives. More quality improvement evidence—in terms of both quantity and quality of evidence—regarding the adequacy of the entire curriculum was made possible through the use of a common metric, like exam scores, rather than through individual thesis completion. Even more importantly, the creation of a comprehensive exam required deep reflection by the faculty and program leaders on the core concepts, skills, and attitudes that needed to be emphasized in each core course and the interconnections among those courses. Finally, a comparative study of individual student performance in courses with the student’s overall mastery of material through a comprehensive examination led to important conversations about consistency, progress, and the framing of what constitutes “success” in the program.

Therefore, in consideration of the expected benefits to both students and faculty from the creation and administration of a comprehensive exam, we embarked on developing and administering a comprehensive exam as an option for students completing the MAB.

Process of Development

The comprehensive examination is a common and important learning assessment tool in higher education. Exam results not only assist evaluators in measuring students’ substantive knowledge of the field but also allow the student to demonstrate acquired skill in synthesis, integration, and constructive analysis—higher-level thinking skills that are essential to graduate work in most disciplines. Defining the appropriate body of knowledge and manner of analysis also orients both professors and students toward cultivating learning that is by nature comprehensive.

At the doctoral level, students typically tend to undertake a series of comprehensive examinations (both oral and written), which collectively serve as a prerequisite—and as evidence of competence—for the demands of dissertation writing.^{Footnote 11} Leonard Cassuto contends that “the comprehensive exam should therefore bear a concrete relation to the dissertation and the work—that is, the research methods and practices—that will be required to complete it.”^{Footnote 12} In other words, the comprehensive examination must be relevant for the student’s trajectory in the program, rather than being an obstruction to be breached.

In addition to these important considerations, the challenge of designing a comprehensive examination for a master’s degree in bioethics that appeals to a diverse student body that includes postbaccalaureates and mid-career professionals further convolutes the task at hand. Moreover, if an examination is offered as an option—of equal rigor as the thesis—to culminate the program and not as a requirement for the student to proceed to thesis writing, then what “concrete relation” are we trying to forge? A master’s degree program that offers an examination and a thesis as two valid options for the completion of the degree speaks to Cassuto’s call for pedagogical relevance insofar as the student can choose which route best serves his or her expectations from graduate study in bioethics.

Very few universities in the United States that have bioethics degree programs at the master’s level include a comprehensive examination as an integral part of the curriculum. Fewer still present it as an equally rigorous alternative to the thesis. We conducted an inventory of 30 U.S. universities that offer a master’s-level degree (or a concentration) in bioethics, healthcare ethics, or medical humanities. Based on the degree requirements listed on the schools’ program websites, 25 out of the 30 do not require a comprehensive examination of any kind. Only 3 out of the 30 schools require students to pass comprehensive examinations to receive a degree. The 2 remaining programs—at Emory University in Atlanta, Georgia, and Trinity International University in Deerfield, Illinois—have a comprehensive examination as an option. At Emory University, a student can take the exam or write a thesis, whereas at Trinity International, the exam is one option among many—including a thesis, integrative paper, or special project—to meet the capstone requirement. Interestingly, all 5 university programs that either require a comprehensive examination or offer it as an option are master of arts in bioethics programs.

Following the national benchmarking to discover how other bioethics programs used exams, we had several faculty discussions about the form and content of the exam. In order to ensure that we were assessing the appropriate learning outcomes, we attended carefully to the program objectives. Our overall learning goals for the program are as follows:

1. 1) To enhance students’ ability to demonstrate critical analysis of bioethical issues

2. 2) To strengthen students’ ability to develop robust moral arguments

3. 3) To improve students’ ability to communicate clearly in written and verbal formats

Any assessment we used needed to connect to the outcomes of the program and measure success in the achievement of those outcomes. Importantly, note that the exam is simply the final assessment of students’ achievement of our goals. Assessments also occurred throughout the duration of the program to measure students’ mastery of knowledge, skills, and attitudes within the boundaries of a specific course. These in-process assessments were used for both summative and formative purposes.

Exam Features

The comprehensive examination was intended to be a culminating experience of the MAB. Therefore, the exam became the students’ opportunity to demonstrate the way they were able to integrate course, textual, and practical knowledge gained throughout the experience of the program. The exam assessed students’ proficiency in the field of bioethics and their ability to critically and constructively analyze bioethical issues. Accordingly, answers to the exam questions had to be thorough, structured, well deliberated, and engaging.

The exam consisted of six questions based on the core courses that all students had to take. Students were required to answer four of the six questions. The exam was divided into three sections:

• •Section 1: Foundation questions, mostly from our theory course

• •Section 2: Integration questions, mostly from our classic and contemporary issues courses

• •Section 3: An application question, in which students responded to a case study

Students had six hours to complete the exam; there were two three-hour sittings scheduled on the same day. The exam was completed in our office, on a computer provided by the Center for Ethics. No notes, texts, or other resources could be brought into the examination room.

We notified the students of the questions for sections 1 and 2 well in advance of the exam. In fact, a full list of potential questions from which examiners would choose was made available to students on the first day of the fall semester. Thus, exam preparation focused on reviewing and integrating knowledge, rather than guessing what would be asked on the exam. We also informed students that any topic included among the published questions was expected knowledge from the program and might be tested, even if the students did not master those topics in the particular core course that they had taken. Therefore, students needed to rely on self-directed learning in their exam preparation by self-identifying topics about which their understanding was insufficient. Likewise, students were expected to make use of extracurricular materials as needed. We provided a program-level reading list for students that should have been accessed in addition to the material included in the syllabi of the core courses.

Question and Answer Development

Particular questions were created by expert panels of faculty members. The comprehensive exam was intended to measure students’ achievement across the entire program, rather than in a single course or content domain. To accomplish this, the faculty stakeholders needed to come to a consensus on the content that would be examined as well as on the actual formulation of each question. Because students were given the questions in advance, the question bank needed to be truly comprehensive, rather than containing only a sampling of topics. The use of a faculty panel for this task was consistent with a basic tenet of assessment that examiners must have a theory of how examinees will represent knowledge.^{Footnote 13,Footnote 14} Negotiation among the members of the expert panel was a long and engaging process that unified the exam in terms of the expected student cognition and that ameliorated differences among examiners based on professional interests or personal biases. The use of expert panels to define exam content is recommended practice among testing professionals.^{Footnote 15,Footnote 16}

In order to ensure that questions were specific and clear, the expert panel created individual questions and then composed answers to those questions to define what they expected students should be capable of answering. The answer-writing process is a step that we suspect is often skipped pedagogically, as faculty members may feel that they “know” the right answer and therefore do not need to write it out. For the faculty, composing answers revealed fundamental ambiguities in some questions, gave us pause to reflect on what and how we were teaching, and compelled us to think self-critically about what we expected students to demonstrate and whether the question could elicit the expected response. In the vast majority of instances, we then revised individual questions to be more precise in terms of asking for the material we judged to be essential. For example, in the initial drafting phase, one exam question was written as follows: “If we can enhance someone (make them smarter, more agile, more attractive, etc.), then we should do so. Describe this position fully and then offer a compelling objection.” After the participation of the expert panel, the question was revised to read: “In bioethics, what do we mean by ‘human enhancement?’ (2) Some have argued that if we could enhance someone, then we should do so. Describe this position fully by providing at least three pertinent arguments in favor of enhancement (2). Then, offer a compelling objection to this claim by providing at least three pertinent arguments against enhancement (2).”

The answer-writing process was guided by the simultaneous creation of a scoring rubric.^{Footnote 17,Footnote 18} A scoring rubric has three main components: a set of criteria on which students are being assessed, an ordinal scale on which students are scored for each criterion, and descriptors for each level of the scale that describe the features that characterize that score level for that criterion.^{Footnote 19}

Our purpose for creating and using a scoring rubric was threefold:

1. 1) To ensure a consistency of expectations for answers among the faculty graders and to remove construct-irrelevant variance

2. 2) To ensure interrater reliability in scoring particular answers

3. 3) To facilitate the communication of faculty expectations to students, both preexam and postexam

The interpersonal negotiation necessary to create the scoring rubric, especially in writing the descriptors for various levels of the criteria, led to an increased sense of purpose among the faculty for the entire program. Because they were creating a comprehensive exam in which students were required to integrate knowledge from multiple courses in order to achieve a passing score, developing the rubric forced faculty stakeholders to expose their beliefs and assumptions about the knowledge and attitudes that students were achieving in our program. A key component of this refocusing was to remove construct-irrelevant variance from the scoring rubric.^{Footnote 20} For example, nonnative English speakers would not receive lower scores based on grammatical proficiency, as long as the appropriate levels of knowledge and attitudes were demonstrated with clarity and coherency.

Interrater reliability is often borderline when using rubrics^{Footnote 21,Footnote 22} and especially when scoring responses to open-ended questions. High-quality scoring rubrics are very difficult to develop^{Footnote 23} and should be created through an iterative process.^{Footnote 24} Nonetheless, a key part of any rubric’s educational value lies in the process of creating it, which is why we would advise against borrowing rubrics from others. We provide an example of our rubric in the Appendix merely to demonstrate its utility and form as a way of inspiring the creation of rubrics in bioethics. In the attempt to achieve adequate interrater reliability for a rubric, developers, scorers, and examiners must discuss and negotiate the nuances of the criteria, scale, and descriptors used. When such a discussion of program goals occurs among those faculty members who teach courses, advise students, create the exam, and score the papers, then the whole process contributes to the quality of the students’ overall educational experience in the program.^{Footnote 25}

Well-constructed scoring rubrics not only improve scoring reliability but also facilitate communication with students about program goals and expectations.^{Footnote 26} In some instances, a rubric may even be presented to students as notification of the standards to which they will be held; we give a sample rubric to students to demonstrate how their answers will be assessed. As a post facto device, a completed rubric will provide a low-performing student with specific detail about deficiencies or areas for remedial attention.^{Footnote 27}

Scoring of the Exam

Because the questions for sections 1 and 2 were available in advance, students’ responses were expected to reflect careful consideration and integration of the issues described. Students’ answers were expected to be complete, while focusing on the answer to the particular question asked. Answers were expected to be concise, coherent, well organized, and clear.

The overall exam was graded pass/fail. Each question was scored according to a minimal competency standard. That is, students were expected to achieve the level of minimal competence for graduate work. The goal was to assess whether or not the student had acquired a sufficient breadth and depth of knowledge of bioethics. With this philosophy in mind, we established that students must pass all four questions in order to pass the exam.

Each student exam was graded by two faculty members who were assigned that semester by the director of graduate studies. The primary faculty member assigned a score to each question, using the rubric for the answer as a scoring guide. The secondary faculty grader provided a global pass/fail assessment, and then both graders came together to make a final judgment about the acceptability of the answers. In the unlikely event that the two graders could not come to an agreement, the director of graduate studies reviewed the answer and provided assistance. If the director of graduate studies was one of the assigned graders, then disputes were to be resolved by the director or associate director of the Center for Ethics.

Students who failed the exam could retake it up to a maximum of three total times. The first time a student failed the exam, the director of graduate studies contacted the student to report the failure and to provide information related to rewriting the exam. The student was then reexamined only on the area(s) in which the student failed the first exam. This reexamination occurred one month after the failure was reported to the student, and the reexam consisted of (a) new question(s) in said area(s).

In the event of a second failure, the student would take a new version of the full examination the following semester. In this case, there was no retake option; the student had to pass all questions in this final attempt. After three exam failures, the student would be dismissed from the MAB program.

Additional Factors during Development

A number of other factors were essential to the exam development process. Space precludes a detailed discussion of these issues; however, each factor in the creation process is worth noting:

1. 1) Establishing a minimum competency standard for the assessment. Standard setting for any educational measure is complicated and fraught with pitfalls.^{Footnote 28} To define a performance standard for our narrowly tailored master’s degree program, we used the Structure of the Observed Learning Outcome (SOLO) taxonomy as our theoretical framework.^{Footnote 29} The SOLO taxonomy defines level of performance according to quantity of information presented and level of integration of that information. In our case, because this was a comprehensive exam and students were expected to combine what they had learned across multiple courses, the SOLO taxonomy was particularly useful. We employed the direct consensus method,^{Footnote 30} with modifications for a constructed response exam format. Detailed information about the critical step of establishing a minimum performance standard will be presented in a forthcoming manuscript.

2. 2) (Lack of) utility of numbers. As is evident from the sample rubric included in the Appendix, we originally designed the assessment so that a numerical value accompanied each discursive category. However, in the actual administration of the exam, we found the numbers to be more distracting for graders than helpful—examiners spent more time discussing the mathematical relationships of the numbers than they did whether or not the student’s answer represented a satisfactory mastery of the material. This experience could have been predicted based on the literature about rubrics that emphasizes treating ratings as categorical rather than quantitative.^{Footnote 31,Footnote 32} In future iterations, we intend to remove the numbers from the grading rubric.

3. 3) Equivalency of summative assessments. Because our assessments are designed to measure achievement of the core competencies of our program, it is essential that both the exam and the thesis are equally rigorous. In addition to combatting student perception that one path was easier or less time-consuming than the other, we also endeavored to ensure that the time and intensity of the actual preparation for both options (draft writing of the thesis and studying for the exam) required equivalent efforts on the part of the student. Whether or not we were successful in achieving the goal of preparatory similarity will be reevaluated once the comprehensive exam program matures.

4. 4) Curricular revisions. Even after having reaffirmed our commitment to the core competencies prior to creating the exam, it became clear after the first administration of the examination that curricular modifications were likely to be necessary in order to better prepare students for the examination. Rather than immediately making content modifications, faculty members have begun discussing how course-specific assessments do (or do not) engage students in the kind of discourse required for a satisfactory examination response. These conversations are ongoing.