Using the Burrows Method

It was once thought that authors’ use of the very common words of English showed no patterned variation: any competent user of the English language used them at the same rate, and individuality of style only emerged from less common lexical words. In the early days of computer analysis of texts, however, Professor John Burrows showed that the incidence of these very common words varies significantly between texts by different authors while remaining comparatively constant within a single author's work.Footnote ² By subjecting the most common words of a large set of texts to statistical procedures, Burrows was able to show that rates of usage vary systematically, and that most authors have a characteristic use of these common words that distinguishes their style from that of their fellows. Burrows followed this up by founding the Centre for Literary and Linguistic Computing at the University of Newcastle, Australia, where, for the past thirty years he and his colleagues have been devising and refining the statistical procedures that can most effectively isolate these distinctive usages.

Attribution of authorship is one of the major areas in which the Burrows Method has been used. It has proved to be a particularly useful tool for testing hypotheses devised by the researcher on the basis of fragmentary or ambiguous “external” evidence, or on “internal” evidence of style, tone, or subject matter (Love 142–45). The first step is to create a “base-set” of texts known to be by the putative author, and a “counter-set” of texts addressed to a similar audience by other known authors, and subject them to statistical analysis of the frequency of a set of commonly used words. It is expected that authorial groupings based on the authors’ idiosyncratic usage will emerge, and that if a text of doubtful or contested authorship is placed among these texts, the validity of the attribution can be assessed. The results of such tests are, of course, matters of probability, not of certainty.

Since the 2006 pilot study tests were completed, a greatly enlarged corpus of Victorian periodical articles has been developed. The counter-set of articles available for comparison now consists of 191 articles by twenty-one authors, while the Mozley base-set, instead of consisting of six periodical articles, as in the pilot study, now contains twenty.Footnote ³ This enlarged corpus has been used in the analyses reported on here. A list of the 200 most commonly used function words in the entire corpus has also been constructed. Footnote ⁴ We are able to make use of these function words in various ways according to the requirements of the test. For initial testingFootnote ⁵ we generally follow Burrows's preference for using an unselected set of the most common wordsFootnote ⁶ – thus allowing the “texts themselves . . . to dictate which word-types should be studied” (Burrows 311).

An example of how this method operates can be seen in Figure 8 where a Principal Component Analysis (PCA) is used to test the assertion (Nicoll 255) that Anne Mozley wrote the review of Villette published in the Christian Remembrancer in 1853. PCA is a data reduction technique, which finds “in order of importance the sets of weightings for the variables that account for the most significant variations in the data” (Craig 200). It highlights the most important likenesses and differences among the specimens and allows us to see something of the complexity of the relationships between the variables in the data. For this analysis the ninety-nine most common function words in the CLLC periodicals corpus (see Appendix B:1) are used to assess the likelihood that Mozley was the author of the Villette review by comparing the use of function words in that article with that in articles by Mozley and two of her contemporaries.

Figure 8.

Principal Component Analysis plot using 99 most common function words as variables.

As can be seen, the Villette review is positioned among the articles by Mozley in the north-east quadrant, while the articles by Macaulay and Martineau are located in the north-west and south east sectors. The figure also demonstrates a problem that a quarter of a century of work at the CLLC on projects covering diverse centuries and genres has highlighted: even in the choice of the most common words an author is influenced by the conventions of writing in the particular genre chosen. Articles by two separate authors with similarities in tone and subject matter can, on occasion, show a greater correspondence in the use of the most common words than two articles by the same author with divergent emphases. This can be seen in Figure 8 where the Macaulay articles fall into two groups, with one of the groups closer to the Martineau articles than to the larger Macaulay group. The explanation for this can be found in the fact that, though all the Macaulay articles are biographical, the four in the left quadrant discuss recently deceased or historical figures, while the other three deal, like the Martineau articles, with largely contemporary issues. Whereas the words used for the test seen in Figure 8 were unselected, in a further refinement of the Burrows Method we can seek to identify “marker” words for a particular author. These are the words that this author uses significantly more or significantly less often than another author or a group of other authors.

Mozley Markers

A set of mozley markers was obtained by comparing the word usage in the twenty Mozley texts with that of two texts (the first and last in our alphabetical lists) by each of the other twenty-one journalists in our corpus.Footnote ⁷ A distribution testFootnote ⁸ using the 200 most common function words in the total corpus isolated 86 words used significantly more often or significantly less often by Mozley than by the other authors. This list of 86 words can be found in Appendix B:2.

It was decided to assess the effectiveness of these words for separating Mozley texts from those of her contemporaries by running Principal Component Analyses using two distinct groups of authors for comparison. Two sets of texts by authors likely to have been subjected in forming their style to cultural influences similar to those impacting on Mozley were chosen. The differences between them can then be seen as resulting from what is idiosyncratic in Mozley's individual signature. The first of these tests compares the usage of the 86 marker words in the twenty articles known to be by Mozley with the usage in articles by four of her exact contemporaries, and the other compares it with that of three women journalists writing at the same time as Mozley.

Five Authors Born in 1809

Five authors in the corpus (including Mozley) were born in the year 1809. This circumstance suggested the notion of testing Mozley's relative use of the marker words with that of her precise contemporaries – John Stuart Blackie, John Hill Burton, William Rathbone Greg, and Elizabeth Rigby Eastlake. Nine texts were selected from each of these authorsFootnote ⁹ for testing with Mozley's twenty texts.

A principal component analysis test was then run on the selected texts using the set of 86 Mozley marker words. The results of this test can be seen in Figure 9 below. In this Figure Mozley's texts form a tighter authorial cluster than the texts of her four contemporaries which fan out across the left hand side of the plot. A line has been drawn in order to highlight the separation between the Mozley and non-Mozley articles.

Figure 9.

Principal Component Analysis plot using 86 marker words as variables.

Anne Mozley and Three Women Journalists

As jordan noted, mozley began her literary career as a contributor to periodicals issued by her family's publishing business and “did not begin publishing in mainstream, as distinct from sectarian, periodicals until she was in her fifties” (316). Her major identified articles are in consequence contemporaneous not with those of her female birth contemporaries (Elizabeth Rigby Eastlake and Harriet Martineau), but with the younger writers, Frances Power Cobbe, Eliza Lynn Linton, and Margaret Oliphant, who made their mark in the 1850s and 1860s. It was felt that comparison with this groupFootnote ¹⁰ could further test the validity of the set of 86 marker words. Figure 10 below shows once again that these 86 Mozley markers can separate Mozley's texts from those of her contemporaries. In this plot, although only prior knowledge of the authorship reveals where the discriminating line should be drawn, there is no intermingling of the two groups. Furthermore Mozley's texts again form a tighter authorial cluster than the texts of her four contemporaries which fan out across the right hand side of the plot.

Figure 10.

Principal Component Analysis plot using 86 marker words as variables.

The Mozley Word Usage Signature

A close examination of mozley's use of these 86 marker words compared to that of her contemporaries (see the “Marker Words Analysis” tables in Appendix B:2) gives some indication of the idiosyncracies of her style. Mozley, it seems, is distinguished by her preference for the conjunctions as and because, and her relative avoidance of if, since and nevertheless, while the prepositional preferences she shows include in, into, with, without, through, and amongst, while of, by, against, beside, during, throughout, and among show relative avoidance. Other word frequency preferences are the adverbs what, whatever, where, and wherever, and the negative forms not, never, and nobody.

Mozley also shows a decided preference for using the indefinite articles a and an and a very decided disinclination to use the definite article.Footnote ¹¹ Coupled with this liking of the indefinite articles is the highly significant frequency of her use of the indefinite personal pronouns it and its, which, along with what and whatever suggests something of a preference for the impersonal. While her contemporaries make more frequent use of past tense verb forms (was, had, been, were, should, could, might, and did), Mozley has more frequent resort to the present forms (is, can, cannot, must, do, and does).

The sort of style implied by some of these word selection preferences can be seen in the openings of a few of her articles. Mozley marker words are italicized.

This tendency towards generalization and impersonality can also be seen in the opening paragraphs of most of her reviews; for example her review “Novels by Sir Edward Bulwer Lytton.”

These tests and analyses suggest that it can be reasonably assumed that these 86 marker words reflect the more habitual and enduring characteristics of Mozley's style. It therefore seemed feasible to use them to discover whether any of the as yet unattributed essays in the Christian Remembrancer seem likely to have been written by Anne Mozley.

Anne Mozley and the Christian Remembrancer

As a first step in trying to identify articles by Mozley that appeared in the Christian Remembrancer, twenty-seven articles dealing with the kind of topics Mozley favoured in her articles in Bentley's, Blackwood's and the Saturday Review were selected, specifically reviews of recent literature and discussions of social issues relating to women and to the Church of England.Footnote ¹² There is some external evidence for her authorship of two of these articles: in 1919 W. Robertson Nicoll proposed Anne Mozley as the author of “New Novels by Lady G. Fullerton and Currer Bell” (255–57), and more recently Joanne Wilkes (110) has noted identical passages in “Youth as Depicted in Modern Fiction” and in a letter from Anne Mozley to John Blackwood dated April 1865. It was, however, felt that further confirmation would be useful so these articles were included among those tested. Table 1 below lists the articles selected, giving each text the number that identifies it in the plots that follow.

Table 1.

Twenty-Seven Christian Remembrancer Articles

To get an initial indication of which of the Christian Remembrancer articles seemed most likely to have been written by Mozley a PCA test, using the 86 marker words in our list, was run comparing the Christian Remembrancer articles with the twenty known Mozleys. The resulting plot is shown as Figure 11 below.

Figure 11.

Principal Component Analysis plot using 86 marker words as variables.

As can be seen, the known Mozley articles form a relatively tight group on the left-hand side of the plot, with twelve of the Christian Remembrancer articles distributed amongst them,Footnote ¹³ while the rest of those from the Christian Remembrancer spread across the right-hand half of the plot. The two Christian Remembrancer articles already attributed to Mozley, “New Novels” (26) and “Youth” (27), sit comfortably among her articles from other journals, thus affirming the accuracy of the external attributions. Yet, as can also be seen, six of the Christian Remembrancer articles, though not distributed among the known Mozleys, are only a little to the right of the dividing line, quite close to each other and fairly distant from the remaining, more widely scattered articles from the Christian Remembrancer. The articles occupying this ambiguous position are: “The Prelude, by Wordsworth” (19), “Sir Edward Bulwer Lytton's King Arthur” (11), “The Moral Character of Story Books” (15), “Parish Schools and Catechetical Teaching” (18), “Children's Books” (3), and “Tupper's Works” (25). Three further articles, “Final Memorials of Charles Lamb” (2), “Recent Novels” (20), and “Church Poetry” (5), lie just beyond the main Mozley cluster. These nine articles were all written before 1851, which was nine years before Mozley began publishing in the wider periodical press. Her final style, on which the marker words are based, may still have been maturing.There is, unfortunately, no further information attainable from this plot on whether they are or are not by Mozley.

There is, however, another way of expressing graphically the relationship between sets of texts that appears to give less equivocal answers to questions of authorship. This is a cluster analysis. According to Chatfield and Collins “The basic aim of cluster analysis is to find the ‘natural groupings’, if any, of a set of individuals” (212), and this can be a useful tool in testing for attribution. Figure 12 below gives a rather more decisive picture of which of the Christian Remembrancer articles seem likely to be by Mozley. In this type of plot, those articles bearing the greatest similarity unite earliest, while the most dissimilar unite last. Though it is somewhat difficult to read the graph because of the large number of individual texts involved, it can clearly be seen that the very last division, that of greatest dissimilarity, separates nine Christian Remembrancer articles from all the Mozley and remaining Remembrancer articles.

Figure 12.

Cluster Analysis plot using 86 marker words as variables.

The Mozley branch contains three sub-branches, with the left-most of these dividing into a further two branches. The twenty known Mozley articles are scattered throughout these four groupings; the line representing each of these known articles has been marked with an asterisk to facilitate identification. All of the Remembrancer articles which grouped with the Mozley articles in Figure 11 are to be found within the four Mozley groups, as well as six of the doubtful articles. The remaining three doubtful articles, “Church Poetry” (5), “Recent Novels” (20), and “Tupper's Works” (25) are located in the non-aligned Christian Remembrancer group.

Unfortunately cluster analyses, even when using marker words, do not invariably give entirely accurate authorial groupings. Whereas PCA plots can present findings that are too nuanced to be definitive, cluster analyses may present an over-positive case. Although in this Figure eighteen of the Christian Remembrancer articles appear in the Mozley side of the tree, there is always the possibility that one or two of them may be there, not because they were written by Mozley, but because they differ radically from all those in the non-Mozley side. “Final Memorials of Charles Lamb” (2), “Miss Sellon” (14), and to a lesser extent, “Moral Character” (15) for example, though positioned under the general Mozley umbrella, are the last articles to join their particular sub-branches. This makes their attribution to Mozley somewhat less certain. On the other hand, other Christian Remembrancer articles (including the two articles for which there was some external evidence) appear to affirm their affinity by uniting early to one or other of the known Mozley articles.

It can also happen, as the plots in Appendices C:2, C:3, and C:4 demonstrate, articles with a similar focus and subject matter, though written by different authors, can be drawn away from the main cluster of articles by each author into a greater proximity to one another. It is thus possible that an article actually written by Mozley may appear on a tree containing articles by one of her contemporaries. It is for these reasons that practitioners believe they must exercise caution when interpreting such plots. This caution usually takes the form of running additional comparative analyses with the works of other authors.

A series of tests was therefore run using the same Mozley and Christian Remembrancer articles with the articles of each in turn of the other twenty-one journalists in the CLLC corpus. Five of these tests can be seen in Appendices C:1 to C:5. In most cases the results supported the attributions shown in Figure 12, but there were occasional variations.Footnote ¹⁴ When this occurred with only one author it was presumed that this was probably due to underlying similarities in focus or subject matter between the Remembrancer article and a particular article by the author used in the test and that there was still a high likelihood it was by Mozley. On the other hand, when a Remembrancer article was attached to the Mozley tree in tests with only one or two of the other authors, it was assumed that all this showed was that it had greater similarities with the known Mozley articles than with any of those by the other author, and that this was not grounds for attributing it to Mozley.

Findings

A summary of the results of the tests is given in Table 2. The table displays the articles in four categories: those texts which invariably demonstrated affinity with the known Mozley articles; those which usually showed such affinity, but occasionally failed to; those which usually showed no affinity with the Mozley articles, but occasionally did so; those that never showed affinity. It should be noted that the two articles (26 and 27) for which there is external evidence of authorship invariably showed affinity with Mozley. These tests, it may be further noted, confirm the attributions made by the cluster analysis displayed in Figure 12.

Table 2.

Proposed Attribution Status of Twenty-Seven Christian Remembrancer Articles

These findings suggest that thirteen of the Christian Remembrancer articles tested can be attributed to Mozley with a high level of confidence, while a further five articles may be called probable. The other nine articles, though dealing with the sort of subjects Mozley chose for her other journalism, are very unlikely to have been written by her.