Introduction
The evolutionary success of mammals has been attributed to key dental innovations, such as a differentiated dentition and the evolution of the tribosphenic molar, which facilitated mastication (Gregory Reference Gregory1921; Ungar Reference Ungar2010), enabling major changes in feeding strategy, life history, and ecology over the past ~160 Myr. Dental morphology is ideal for studying mammalian evolution because of its high preservation potential and relationship to ecological function (Owen Reference Owen1845; Osborn Reference Osborn1907; Jernvall et al. Reference Jernvall, Hunter and Forteluis1996; Lucas Reference Lucas2004), though comparisons of disparate morphologies across clades have often proven difficult to quantify. Recently, new methodological approaches (Evans et al. Reference Evans, Wilson, Fortelius and Jernvall2007; Boyer Reference Boyer2008; Bunn et al. Reference Bunn, Boyer, Lipman, St Clair, Jernvall and Daubechies2011) have provided robust, quantitative methods for comparing dental morphology across disparate mammalian clades, illuminating shared ecological and developmental modes (Evans et al. Reference Evans, Wilson, Fortelius and Jernvall2007).
Marine mammals exhibit dentition that departs in significant ways from that of their near terrestrial relatives. In cetaceans, for example, both extant clades (Mysticeti and Odontoceti) differ notably from the general terrestrial mammalian condition (Fig. 1). Stem cetaceans (also known as “archaeocetes”) retained a heterodont dentition adapted for mastication throughout the Eocene during their transition from land to sea (Uhen Reference Uhen2004; Thewissen et al. Reference Thewissen, Sensor, Clementz and Bajpai2011). By contrast, extant mysticetes filter feed using baleen plates made of keratin and are entirely edentulous as adults, though rudimentary tooth buds in utero reflect their toothed ancestry (Saint-Hilaire Reference Saint-Hilaire1807; Eschricht Reference Eschricht1849; Karlsen Reference Karlsen1962; Peredo et al. Reference Peredo, Pyenson and Boersma2017). Extant odontocetes are predominantly suction feeders, bearing only single-rooted conical teeth lacking accessory cusps; while the anterior teeth may play a role in prey acquisition, the cheek teeth do not process prey whatsoever, and prey is swallowed whole (Werth Reference Werth2000; Armfield et al. Reference Armfield, Zheng, Bajpai, Vinyard and Thewissen2013; Marshall and Goldbogen Reference Marshall and Goldbogen2015). By the Oligocene, early members of both extant clades diversified during a time with high origination rates, which resulted in the origin of nearly all extant cetacean lineages, along with many extinct ones (Lipps and Mitchell Reference Lipps and Mitchell1976; Uhen and Pyenson Reference Uhen and Pyenson2007; McGowen et al. Reference McGowen, Spaulding and Gatesy2009; Geisler et al. Reference Geisler, McGowen, Yang and Gatesy2011; Marx Reference Marx2011).
Three-dimensional models of select cetacean feeding systems demonstrating overall patterns of cetacean dental evolution through time. (A) Cheek teeth of stem cetaceans USNM 508060 (Protocetidae) and USNM 11962 (Zygorhiza kochii); (B) cheek teeth of odontocetes USNM 205491 (Xenorophidae), USNM 10484 (Squalodon calvertensis), USNM 214911 (Dilophodelphis fordycei), and USNM 241401 (Orcinus orca); and (C) cheek teeth of mysticetes USNM 25210 (Aetiocetus cotylalveus) and UWBM 50004 (Salishicetus meadi) and baleen rack of USNM 594182 (Balaenoptera physalus). Crown symbols denote crown members of each lineage (crown Cetacea, crown Odontoceti, and crown Mysticeti). See Table 1 for complete list of specimens and source collections.

Fossil and molecular evidence indicates that the crown groups of both mysticetes and odontocetes originated just after the Eocene/Oligocene boundary (McGowen et al. Reference McGowen, Spaulding and Gatesy2009). This boundary marks major reorganization of global climate systems that drastically altered the marine environments (Prothero et al. Reference Prothero, Ivany and Nesbitt2003; Steeman et al. Reference Steeman, Hebsgaard, Fordyce, Ho, Rabosky, Nielsen, Rahbek, Glenner, Sørensen and Willerslev2009) and likely spurred the diversification of crown odontocetes and mysticetes (Uhen and Pyenson Reference Uhen and Pyenson2007). Quantifying dental morphologies through the Oligocene is therefore critical to understanding the evolution of modern cetacean diversification. Previous studies have qualitatively noted a pattern of simplification (Werth Reference Werth2000; Uhen Reference Uhen2017). Here, we quantify this pattern by measuring dental complexity and confirm that mysticetes and odontocetes independently converge on dental simplicity.
Methods
Taxonomic Scope
We selected cheek teeth from cetacean specimens to encompass the broadest available range of taxonomy and dental morphologies, with an emphasis on stem odontocetes and mysticetes from the Oligocene. This data set includes 3 stem cetaceans, 5 mysticetes, and 14 odontocetes (Table 1). In crown cetaceans, establishing homology of teeth within the dental row is challenging; it is especially difficult to distinguish premolars and molars of isolated teeth by count alone, as cetaceans evolved polydonty by the late Eocene (Armfield et al. Reference Armfield, Zheng, Bajpai, Vinyard and Thewissen2013) and monophyodonty with the origin of crown cetaceans (Uhen Reference Uhen2017). All crown cetacean teeth used in this study represent cheek teeth from the center of the tooth row. For stem cetaceans, which have distinct premolar and molar morphologies, we used only premolars for maximum comparability. This study does not measure the complexity of odontocete tusks (e.g., Ziphiidae, Monodon, Odobenocetops), as the tusks are not homologous with cheek teeth. One taxon in our analysis, Ninoziphius, bears both cheek teeth and tusks; this analysis measures only the cheek teeth for comparability. By contrast, Monodon and Odobenocetops bear only tusks and lack cheek teeth entirely.
Orientation patch count rotated (OPCR) and relief index (RFI) results for teeth included in this analysis. USNM, National Museum of Natural History, Washington, D.C.; UWBM, Burke Museum of Natural History and Culture, Seattle, Wash.; pc, postcanine tooth.

The geologic ages of the teeth are binned at the stage level following the Gradstein et al. (Reference Gradstein, Ogg, Schmitz and Ogg2012) timescale. Later-diverging, fully aquatic stem cetaceans from the late Eocene (e.g., basilosaurids) are undersampled relative to other time bins because of a lack of well-preserved and accessible cheek teeth, but that is not a concern, as this study focuses on patterns that postdate this epoch, during the second major phase of cetacean diversification, when the modern families originate (Pyenson Reference Pyenson2017). A full analysis of dental morphologies among stem cetaceans throughout the Eocene and the land-to-sea transition is beyond the scope of this paper; instead, we point to discussions by other authors (Thewissen et al. Reference Thewissen, Sensor, Clementz and Bajpai2011; Armfield et al. Reference Armfield, Zheng, Bajpai, Vinyard and Thewissen2013). Fetal mysticete tooth buds were not analyzed here, mostly because of the challenges in collecting these data (e.g., specimen availability) and interpreting their results. Historically, authors have disagreed over tooth bud homology and morphology; it remains unclear whether they are homologous to the deciduous or permanent mammalian dentition; see Peredo et al. (Reference Peredo, Pyenson and Boersma2017) for a full review. Moreover, given the challenges of collecting voucher specimens and the miniscule size of the tooth buds, it is unlikely that such an effort would result in models of high-enough resolution (10,000 faces) for complexity analyses. Though recent studies suggest that the morphology of these tooth buds is complex (Thewissen et al. Reference Thewissen, Hieronymus, George, Suydam, Stimmelmayr and McBurney2017), previous authors figure fetal tooth buds as conical in shape (Eschricht Reference Eschricht1849; Ridewood Reference Ridewood1923; Ishikawa and Amasaki Reference Ishikawa and Amasaki1995). It is therefore unclear whether tooth bud complexity would be similar to that of extinct or extant odontocete cheek teeth. Instead, we assign a value of zero to all mysticetes that lack a dentition at birth and consider this to represent maximum simplification.
Computed Tomography Scanning and 3D Modeling
We used X-ray and computed tomography (CT) to scan our study samples, and then segmented the resultant images to create 3D models, which were then oriented for analysis. Nine specimens were scanned at the Smithsonian Institution Bio-Imaging Research Center in the Department of Anthropology at the National Museum of Natural History (Washington, D.C.) with a Siemens Somatom Emotion 6 at slice thickness of 0.63 mm. The other 13 specimens were scanned using Nikon Metrology’s 225 kV microfocus X-ray CT cabinet system at National Technical Systems (Belcamp, MD), with a slice thickness of 0.30 mm. In all cases, we processed resultant DICOM files in Mimics Innovation Suite 19 (Materialise NV, Leuven, Belgium) and created a mask based on the threshold of the specimen, relative to the nominal density of air. This mask produced a 3D object exported as a binary STL and loaded in MeshLab (Cignoni et al. Reference Cignoni, Callieri, Corsini, Dellepiane, Ganovelli and Ranzuglia2008), where we cropped individual tooth models to include only the enamel crown following Boyer (Reference Boyer2008), as suggested by Pampush et al. (Reference Pampush, Winchester, Morse, Vining, Boyer and Kay2016). We then simplified the models to 10,000 faces using Quadratic Edge Collapse Decimation to maintain consistency with previous studies (Boyer Reference Boyer2008; Bunn et al. Reference Bunn, Boyer, Lipman, St Clair, Jernvall and Daubechies2011) and standardized for size and scan resolution as recommended by Pampush et al. (Reference Pampush, Winchester, Morse, Vining, Boyer and Kay2016). We oriented models with the occlusal plane perpendicular to the z-axis and exported them as ASCII PLY files for compatibility with molaR (Pampush et al. Reference Pampush, Winchester, Morse, Vining, Boyer and Kay2016).
Complexity Analyses
We measured dental complexity using orientation patch count (OPC), a quantitative 3D geographic information system analysis developed by Evans et al. (Reference Evans, Wilson, Fortelius and Jernvall2007). OPC maps the surface of a tooth to a grid and divides it into colored segments based on orientation (Evans et al. Reference Evans, Wilson, Fortelius and Jernvall2007). Continuously adjacent faces of identical orientation are binned into patches; a change in orientation corresponds to a new patch. The overall OPC value is the number of distinct patches on a surface and thus represents diversity of orientation. Orientation patch count rotated (OPCR) repeats the OPC process eight times per tooth, with the model rotated 5.625° between trials, to collect an average patch-count value (Smits and Evans Reference Smits and Evans2012; Evans and Janis Reference Evans and Janis2014). To verify that size was not a driving factor of complexity, we analyzed each tooth with the relief index (RFI) function in molaR. The RFI function calculates an object’s 3D surface area and 2D footprint and returns these values along with the relief index, a weighted ratio of the two (Boyer Reference Boyer2008). These areas were used to examine the correlation between OPCR and tooth size.
Evans et al. (Reference Evans, Wilson, Fortelius and Jernvall2007) first devised OPC as a method to quantify disparity between carnivoran and rodent dentitions using the entire dental arcade, though subsequent studies have executed OPCR and RFI across single cheek teeth instead (Boyer Reference Boyer2008; Evans and Janis Reference Evans and Janis2014). In particular, Evans et al. (Reference Evans, Wilson, Fortelius and Jernvall2007) used the entire tooth row, because they sought to link the complexity values to dietary function, rather than merely quantify patterns of complexity. Very few Eocene and Oligocene cetaceans preserve complete dental arcades, and fewer still have preservation suitable for OPCR analysis. Accordingly, we follow Boyer (Reference Boyer2008) and Evans and Janis (Reference Evans and Janis2014) in executing OPCR and RFI on single teeth. Given the wide variation in tooth count and the difficulty in establishing tooth identity (or homology) in cetaceans, we selected only representative cheek teeth from the center of the dental arcade to maximize comparability across taxa.
Results
OPCR Results
Our results demonstrate an initial expansion followed by a drastic decrease in cetacean dental complexity through time (Fig. 2, Table 1, Supplementary Fig. S1). Peak dental complexity is observed in the late Eocene, corresponding to a drastic expansion of complexity range. High complexity values persist into the Oligocene (greater than 200 patches), though the overall range of values downshifts in complexity. Notably, the absolute minimum complexity (complete edentulism) is first observed in the Oligocene. By the Miocene, complexity values are low and continue to decrease into the Quaternary; extant cetaceans demonstrate highly simplified (or absent) dental morphology relative to their ancestors. This fundamental trend toward decreasing complexity values in the Neogene occurs in both odontocetes and mysticetes, representing a convergence in each clade on dental simplification.
Dental complexity of cetacean teeth through time. Data points are binned by geologic stage. Complexity is measured in patches by orientation patch count rotated (OPCR; see “Methods”). Twenty-two teeth were scanned, but 28 data points are displayed, as mysticete data points were added at zero in time bins containing odontocete data to represent crown Mysticeti, which are edentulous as adults. Of note is the broad diversity of dental complexity established during the late Eocene and persistent through the Oligocene that disappears toward the modern, as both odontocetes and mysticetes converge on dental simplicity.

Eocene Expansion of Dental Complexity
Our results show an initial increase in the breadth of complexity between the middle and late Eocene (Fig. 2). The middle Eocene teeth have OPCR values separated by 22 patches, and the late Eocene teeth have values separated by 62 patches. However, the late Eocene values are notably higher; the late Eocene includes Zygorhiza and Llanocetus, two of the most complex teeth in the data set. It is worth noting, however, that the Eocene data set is limited; we suspect that our results are constrained by morphologies exclusive to family-level groups with specific stratigraphic ranges. Improved sampling in the middle Eocene will test the antiquity of the expansion of dental complexity.
Oligocene Diversification of Dental Morphologies
Range of complexity broadens in the Oligocene coincident with the extinction of stem cetaceans and the evolution of odontocetes and mysticetes. The four stem mysticetes from the Oligocene span a range of 269 patches. This range is even more exaggerated, considering that the earliest edentulous mysticetes also evolved during the Oligocene, producing the lowest possible value (i.e., 0; Fig. 2). The four stem odontocetes show even more disparity in complexity, spanning 429 patches.
Convergence on Dental Simplification
The Miocene shows a tremendous narrowing of complexity range relative to other time bins; because it encompasses more than 20 Myr of time, it represents overall a major component of the trend toward simplification in cetacean dentitions during the Neogene. Toothed, stem mysticetes became extinct, and only edentulous mysticetes persisted, represented by values of zero. Similarly, the earliest homodont odontocetes evolved during the Miocene (Lambert et al. Reference Lambert, Muizon and Bianucci2015; Tanaka and Fordyce Reference Tanaka and Fordyce2015, Reference Tanaka and Fordyce2016) and quickly diversified. It is important to note that this analysis does not include all Miocene heterodont odontocetes (i.e., Neosqualodon); however, this analysis does capture the overall shift from multicusped, denticulate cheek teeth to incisiform cheek teeth associated with homodonty. Neogene odontocetes in this study represent both a constricted range of complexity (separated by only 48 patches) and an overall simplified dentition (maximum complexity of 70 patches). Although sampling is limited from the middle Miocene to the Pleistocene, any odontocetes from this time range are homodont and possess conical cheek teeth; there is no reason to expect that their complexity values would not resemble those of the other conical cheek teeth included in this study.
The constricted range and overall simplicity of the dentition is especially pronounced in the Quaternary. The six extant odontocetes span a range of only 26 patches. All six of these teeth have OPCR values below 40 patches, and three of these have values below 20. This pattern reflects the fact that all extant odontocete teeth are simple cones. Even river dolphins such as Platanista and Inia, which have more complex cheek teeth than most odontocetes and are sometimes considered heterodont, show extremely low values compared with any heterodont fossil odontocete sampled. The teeth sampled for both Platanista and Inia represent cheek teeth that differ superficially from the incisors; nonetheless, their complexity values more closely approximate the conical teeth of other homodont odontocetes and not the heterodont cheek teeth from the fossil record. The Quaternary thus represents a time of unprecedented simplicity; the mean OPCR value from this time is 7.64% of the middle Eocene and 7.90% of the Oligocene. Simplicity is not merely the result of the extinction of stem cetaceans; both clades exhibit a broad range of dental morphologies in the Oligocene before independently converging on simplicity.
Discussion
Tooth Size as a Relevant Factor
We calculated the 3D surface area and the 2D footprint of each tooth using the RFI function in molaR (Pampush et al. Reference Pampush, Winchester, Morse, Vining, Boyer and Kay2016) (Fig. 3, Table 1) and show that our results bear no size bias. Neither the 3D surface area nor the 2D footprint show any significant correlation with OPCR values (r 2=0.03976 and 0.01274, respectively). Notably, nearly the complete size range of extant odontocetes (Platanista to Physeter) is captured by the Quaternary sample, and each tooth exhibits very low complexity. This finding demonstrates that dental complexity is independent of tooth size, as expected using OPC. Nevertheless, qualitative observations indicate that both absolute and relative tooth size decrease during the transition from stem to crown Cetacea. Applying quantitative approaches to measuring morphological diversity across an increasingly well-resolved phylogeny, along with the discovery of fossil cetaceans preserving more complete dental arcades, will add to our understanding of this important evolutionary shift in the lineage of mysticetes.
Dental complexity of cetacean teeth as a function of (A) 3D surface area and (B) 2D footprint. Complexity is measured in patches by orientation patch count rotated (OPCR) and area is calculated by relief index (see “Methods”) and is reported in square millimeters. The r 2 values for both A and B (0.03976 and 0.01274, respectively) demonstrate that tooth complexity is independent of tooth size.

Relationship between Cusps and Complexity
The evolution and subsequent loss of accessory cusps in cetacean cheek teeth have been linked to key functional components such as mastication (Armfield et al. Reference Armfield, Zheng, Bajpai, Vinyard and Thewissen2013). Our data set, which spans the evolution of accessory cusps between the middle and late Eocene, as well as their loss in the Miocene, suggests that cusps are not reliable indicators of complexity. In the Oligocene, the most complex mysticete tooth in the data set (Aetiocetus cotylalveus) exhibits only putative, diminished accessory cusps, while the least complex mysticete tooth (USNM 256597, cf. Aetiocetidae) bears several well-developed accessory cusps on either side of the central cusp. The Miocene Squalodon, one of the last surviving multicusped odontocetes, has a low complexity value relative to other odontocetes with well-developed cusps. This finding suggests that overall complexity values are driven not just by the number of accessory cusps, but also by the topography of both the whole tooth (e.g., conical, curved, compressed) and the cusps themselves; additional cusps alone do not substantially increase complexity if they are merely additional, simple cones. Thus, overall dental complexity is likely driven by a confluence of factors, including the number of accessory cusps and the number and shape of wear facets on the occlusal surface from abrasion and attrition.
Functional and Ecological Explanations
Trends in cetacean dental morphology have been attributed to both functional and ecological factors. For example, the evolution of accessory cusps in stem cetaceans has been attributed to increased efficiency of mastication (Uhen Reference Uhen2004; Armfield et al. Reference Armfield, Zheng, Bajpai, Vinyard and Thewissen2013). Notably, although all stem cetaceans are hypothesized to undergo some degree of prey processing (Uhen Reference Uhen2017), neither modern lineage of cetacean orally processes prey. While the evolution of marine mammal feeding has been studied theoretically (Hocking et al. Reference Hocking, Marx, Park, Fitzgerald and Evans2017), the exact function of fossil cetacean teeth remains uncertain, as functional hypotheses for the feeding modes of stem odontocetes and mysticetes remain untested by biomechanical or performance tests.
Both odontocetes and mysticetes evolved key innovations that facilitated prey acquisition (i.e., filter feeding and echolocation) in the Oligocene (Lipps and Mitchell Reference Lipps and Mitchell1976; Lindberg and Pyenson Reference Lindberg and Pyenson2007; Uhen and Pyenson Reference Uhen and Pyenson2007; McGowen et al. Reference McGowen, Spaulding and Gatesy2009; Geisler et al. Reference Geisler, Colbert and Carew2014). Subsequently, each branch independently simplified its dentition; mysticetes became edentulous and odontocetes developed conical teeth. We attribute this convergence on simplification to a broader loss of prey processing in each lineage.
Future studies can further explore patterns in cetacean dental evolution by focusing on additional factors not measured by OPC. For example, OPC does not capture the decrease in relative tooth size, nor does it capture the change from double- to single-rooted teeth (Fig. 4). Additionally, quantifying changes in dental complexity across the tooth row of a single individual may provide insight into the origin of homodonty. Unfortunately, few fossil cetaceans preserve complete tooth rows in which all teeth have reliably established homologies and relatively undamaged occlusal surfaces.
Orientation patch count rotated models of select cetacean cheek teeth demonstrating overall patterns of dental simplification through time. (A) Stem cetaceans, (B) odontocetes, and (C) mysticetes. See Fig. 1 for taxonomic sampling shown. Crown symbols denote crown members of each lineage (crown Cetacea, crown Odontoceti, and crown Mysticeti).

The evolution of filter feeding mysticetes and echolocating odontocetes, which have probable origins near the Eocene/Oligocene boundary (Pyenson Reference Pyenson2017), may explain the overall trend away from prey processing in crown cetaceans. Both filter feeding and echolocation represent novel modes of prey acquisition and are likely enabling factors for the lack of prey processing and subsequent origin of simple dental morphologies. The lack of prey processing in crown lineages may explain the origin of simple dental morphologies, but it does not explain the extinction of complexity in each lineage, which is an entirely post-Oligocene pattern. The biotic response to the Oligocene/Miocene transition in the oceans is poorly understood, especially relative to the better studied Eocene/Oligocene transition (Zachos et al. Reference Zachos, Pagani, Sloan, Thomas and Billups2001a, Reference Zachos, Shackleton, Revenaugh, Pälike and Flower2001b; Prothero et al. Reference Prothero, Ivany and Nesbitt2003), and it remains unclear what factors initiated the onset of dental simplification in the early Miocene. Ultimately, the resultant simplification persists through the remainder of the Cenozoic evolutionary history of cetaceans.
Conclusion
We used quantitative methods to analyze the dental complexity of cetacean dentitions through time and to examine the evolution of modern dental morphologies. In doing so, we quantified changes in dental morphology and identified a narrowing of complexity coincident with a drastic simplification through the Neogene. This shift encapsulates the departure of both odontocetes and mysticetes from the ancestral mammalian condition, with both independently converging upon a reduced and simplified dentition; surviving mysticetes lose the dentition entirely and become filter feeders, while odontocetes simplify to conical pegs that do not process prey. The modern morphologies were established in the Oligocene (mysticetes) and early Miocene (odontocetes), showing that the cetacean dental morphologies remained virtually unchanged over the last 20 Myr of their evolutionary history.
Acknowledgments
We thank David Bohaska, John Ososky, and Darrin Lunde for access to USNM specimens. We thank Christian Sidor and Meredith Riven for access to the Burke Museum specimen. We thank the authors of Pampush et al. (Reference Pampush, Winchester, Morse, Vining, Boyer and Kay2016) for making the molaR package freely available. Finally, we thank Chris and Ryan Peitsch at National Technical Systems and Sabrina Sholts at the USNM Department of Anthropology for facilitating CT scanning. Finally, we thank the reviewers and editors who contributed to the improvement of this article.
Supplementary material
Data available from the Dryad Digital Repository: https://doi.org/10.5061/dryad.6qh0qf1




