2.1 Set of languages for which poems have been generated

Efforts to generate poetry automatically have been carried out in a number of languages. The majority of the systems developed addressed English and ranged from the generation of simple haikus (e.g., Wong and Chun (Reference Wong and Chun2008); Netzer et al. (Reference Netzer, Gabay, Goldberg and Elhadad2009)), to more complex user-given forms (e.g., Manurung (Reference Manurung2003); Colton et al. (Reference Colton, Goodwin and Veale2012)), including also song lyrics (e.g., Barbieri et al. (Reference Barbieri, Pachet, Roy and Esposti2012)). There is also a minority of known approaches for generating poetry in other languages, some of which very different, not only in terms of covered poetry forms, but also in terms of phonetics, syntax and other linguistic rules.

Spanish was one of the first languages where the automatic generation of poetry was explored in the context of Artificial Intelligence, and related issues were discussed (Gervás Reference Gervás2000a, Reference Gervás2001). For Portuguese, another romance language, song lyrics have been automatically generated for a given melody (Gonçalo Oliveira et al. Reference Gonçalo Oliveira, Cardoso and Pereira2007), and poetry has been produced according to user-given structures that would set the number of lines, stanzas, syllables per stanza or the rhyme pattern (Gonçalo Oliveira Reference Gonçalo Oliveira2012).

Traditional eight-line Basque poems, that aim to be sung, have also been produced automatically (Agirrezabal et al. Reference Agirrezabal, Arrieta, Astigarraga and Hulden2013). Although as Portuguese and Spanish, Basque is spoken in the Iberian Peninsula, it has different origins and is significantly different from romance languages. Toivanen et al. (Reference Toivanen, Toivonen, Valitutti and Gross2012) have produced poetry in Finnish with the particularity of exploiting two corpora: one for providing semantic content and another for collecting poetic forms.

Asian languages have also been targeted, some of which with specific tonal and rhythm requirements in poetry generation. This includes the generation of song lyrics in Tamil (Ramakrishnan A et al. Reference Ramakrishnan, Kuppan and Devi2009), a phonetic language; ancient Chinese classic poetry (Yan et al. Reference Yan, Jiang, Lapata, Lin, Lv and Li2013; Zhang and Lapata Reference Zhang and Lapata2014), with strict tonal and rhythm requirements; follow-up lines in Bengali (Das and Gambäck Reference Das and Gambäck2014), matching the rhythm of a user-given line; or poetry inspired by news articles, in Indonesian (Rashel and Manurung Reference Rashel and Manurung2014).

The previous systems seem to have been specifically tailored for a single language. As far as we know, there have been no language-independent attempts to poetry generation nor systems adapted to different languages. The closest is probably the architecture of Colton et al. (Reference Colton, Goodwin and Veale2012) and the constraint satisfaction approach of Toivanen et al. (Reference Toivanen, Järvisalo and Toivonen2013), both originally applied to English, but roughly combined in the system of Rashel and Manurung (Reference Rashel and Manurung2014), which targets Indonesian.

2.2 Computational approaches in poetry generation systems

There have been numerous efforts to model the generation of poetry using Artificial Intelligence techniques. Most of these efforts can be seen as solutions based on the reuse of existing text, subject to certain modifications or reorganisations. The main differences between them lie on the degree of modification of the source text and the choice of unit of recombination during the process of adaptation.

Solutions based on direct reuse of text fragments from existing sources, recombined into different orders, range from the combinatorial approach of Queneau (Reference Queneau1961) – which managed to obtain a large number of new poems in French by interchanging the lines of a set of poems, always respecting the same relative position of each line within a set template stanza – to the reuse of complete tweets, or sentences extracted from them, as featured in the FloWr system (Charnley et al. Reference Charnley, Colton and Llano2014), which generated English verses. Yan et al. (Reference Yan, Jiang, Lapata, Lin, Lv and Li2013) also reuse text from a large poem corpus, but they formulate poetry generation as an optimisation problem based on generative summarisation. Their approach involves the retrieval of candidate lines for a user query, their segmentation into constituent terms, term clustering and selection of sentences that conform the structural constraints, using terms from different clusters. A similar approach is followed by Lamb et al. (Reference Lamb, Brown and Clarke2015) and Malmi et al. (Reference Malmi, Takala, Toivonen, Raiko and Gionis2016). Because these solutions reuse text directly, they are applicable to any language. However, they are also restricted in that they take almost no advantage of basic principles of language such as compositionality: they simply search for combinations of existing fragments of text, with no ability to create new interesting instances of similar fragments. We consider this to be a poor solution.

Slightly more elaborate solutions reuse material from known sources but replace specific words with material from a different source. The first known example is the French rimbaudelaires, by Oulipo (Reference Oulipo1981), which used words mined from Baudelaire’s poems to replace particular positions in verses from Rimbaud, subject to metrical match between the elements in each substitution. Toivanen et al. (Reference Toivanen, Toivonen, Valitutti and Gross2012) combined two different corpora, one – fragments selected from a corpus of old Finnish poetry – to mine for verse drafts that were converted into templates by leaving out certain keywords, the other – a word association network built from the Finnish Wikipedia – to obtain candidate words to plug the resulting gaps. In later work (Toivanen et al. Reference Toivanen, Gross and Toivonen2014), the replacement process is further constrained with restrictions on words already associated with prior knowledge on the poem’s topic. This approach of filling templates combined with various restrictions is elaborated further by Colton et al. (Reference Colton, Goodwin and Veale2012), who use material extracted from news to fill in handcrafted templates. To drive this process, they additionally introduce emotional constraints, in the form of a mood – from the analysis of newspaper articles for a particular day – that influences decisions taken in the construction of poems in English. Solutions based on substitution of particular words in existing fragments of text start to introduce linguistic elements into the construction process, and take advantage of the resulting flexibility. Because they rely on basic linguistic information – lexical category, morphological information and metric scansion of both the words being replaced and the words being used in their place – they require language specific functionality in the form of part-of-speech (POS) tagging, solutions for morphological inflection and solutions for metric scansion. In this sense, they are slightly more difficult to generalise to other languages than the solutions based on direct reuse of text fragments. However, POS tagging and morphological flexion are relatively easy to obtain for most languages. Metric scansion presents more difficulty, as different languages rely on different metric traditions, and some of these are more difficult to automate than others.

The process of stripping down a reference text to leave gaps that can be refilled with material from a different source can be taken to the extreme when all the words from a reference fragment are eliminated, retaining only the corresponding sequence of POS tags, to be filled in with words from a different source, selected based on POS tag match and possibly further aptness criteria. This is the approach followed by systems such as the early version of the WASP system (Gervás Reference Gervás2000a) – which combined the structure of known poems with vocabulary provided by the user to generate Spanish poems – the work of Agirrezabal et al. (Reference Agirrezabal, Arrieta, Astigarraga and Hulden2013) – which used both syntactic and semantic information to ensure that structures from Basque poems of different line lengths were filled in with particular attention to the correct transfer of morphological information, extremely relevant in the case of the Basque language. A similar approach is applied to Finnish poetry (Toivanen et al. Reference Toivanen, Järvisalo and Toivonen2013), where a complex set of constraints is also imposed. As a more elaborate version of the approach discussed previously, the replacement of every word in a given fragment by words deemed appropriate requires much the same set of language-specific functionalities. When metric scansion has to be carried out over fragments of text larger than a single word, the differences across languages may become more marked, due to different traditions with respect to metric phenomena such as synaloepha. Depending on the tag set chosen, simple POS tag preservation during substitution over larger fragments of text may not guarantee grammaticality. It is also likely to result in texts of poor idiomatic correctness.

The technique of Case-Based Reasoning (Aamodt and Plaza Reference Aamodt and Plaza1994) was used in an evolution of the WASP system (Gervás Reference Gervás2001), which matched a user input with a prose rendition of a poem, and reused the structure of the poem – in terms of its POS tags – together with the words from the user input. Such an approach basically relies on a case base that encodes transformation between two different styles of writing – prose and poetry – for a given language. The construction of such case bases for a new language requires a substantial effort in annotation and alignment. As a result, this approach is inadequate for a solution to be applied across languages.

Besides word-based specification of content combined with emotional aspects, Misztal and Indurkhya (Reference Misztal and Indurkhya2014) introduce a different way of constraining form and combining the modules that deal with different aspects of poetry generation. It relies on a multi-agent approach where each module is a set of artificial experts, focused on a particular aspect, that interact by sharing results on a blackboard. Types of experts include word-generating – that contribute with words matching a given topic or emotion –, poem-making – that arrange words in the common pool into phrases or sentences, guided by Context-Free Grammars – and evaluating experts. Content is constrained in terms of a particular topic and an emotion extracted from the input text, while form is constrained through the implementation of the various experts. In this approach, specific literary forms are introduced explicitly by the set of system modules, rather than arising from the reuse of an existing corpus of poetic texts. This type of system allows for consideration of a very rich set of features – such as emotion or grammar. However, the addition of each of these features requires the implementation of the particular expert module. In most cases, these modules will have to be language-specific, and possibly require a number of resources to inform/encode the behaviour of the feature for that particular language. For these reasons, we do not consider this approach to be particularly appropriate for application across languages.

At the other extreme of the spectrum lie systems, which start from an actual message to be conveyed by the poem, specified in terms of a semantic representation. Manurung (Reference Manurung1999) pioneered this kind of system by applying a generate & test approach based on chart generation, which received as input a specification of the target semantics in first-order predicate logic (FOL) and a specification of the desired poetic form in terms of metre. Systems of this type require resources to encode or inform the transitions between a FOL representation and a set of grammatical sentences. A chart was built incrementally using words from a lexicon that subsumed the input semantics, and partial solutions were semantically checked, to avoid incompatibility with the original input and ensure compatibility with the desired poetic form. A similar approach, but based on mirroring the meaning of a given textual document, was employed more recently (Tobing and Manurung Reference Tobing and Manurung2015). In addition to the poetic features, the system tries to keep the predicate-argument structure of the document. But the authors of these efforts conclude that dealing with so many constraints is computationally impractical. The automated transcription from FOL representation to grammatical sentences is an unsolved open problem in the field of natural language generation, even for those languages for which it has been addressed empirically. Solutions of this type capable of operating across a number of languages would require a very substantial effort of FOL to sentences encoding for each language involved.

A final approach to reuse of existing texts foregoes syntactic structure as featured by POS tag sequences and relies instead on the use of n-grams to model the probability of certain words following others. This can be understood as a process of reusing corpus fragments of size n, to be combined into larger fragments based on the probability of the resulting sequence. Such a procedure is used by the Poetic Machine (Das and Gambäck Reference Das and Gambäck2014) to build single poetic lines matching a given rhyme pattern to be matched, by a redesigned version of the WASP poetry generator (Gervás Reference Gervás2013a, Reference Gervás2013b) that used an evolutionary programming approach to model the poet’s ability to iterate over a draft, and by the work of Barbieri et al. (Reference Barbieri, Pachet, Roy and Esposti2012), which relied on Constrained Markov Processes to generate texts as lyrics in the style of an existing author, integrating constraints on grammaticality, rhyme, meter, and, to a certain extent, semantics, into the search procedure itself. N-gram-based language models have shown great applicability in many contexts – such as machine translation, speech processing, text prediction or dialog systems – and they present the important advantage of being language independent to a great extent. At least in the sense that they can be trained on large volumes of text in whatever language is required. These solutions do apply the principles of articulation of language, which gives them significant expressive power. However, they take no advantage of other basic principles such as lexical categories or grammar.

All the different approaches to the computational generation of poetic text illustrate a variety of possible ways of partitioning and recombining existing sources of poetic and non-poetic texts so they can be recombined into new poems.

2.3 Evaluation of poetry generation systems

The field of computational creativity is progressively evolving beyond the early stages where acquisition of valid samples of a given style of artefact was acceptable without further consideration, to a point where evaluation of the outcome has become an important requirement (Jordanous Reference Jordanous2012). This is seen as a sign of scientific maturity of the field.

Due to the underlying difficulty of evaluating creative artefacts, and claiming that the intended audience of poetry consists of people, the evaluation of computer-generated poetry has often resorted to human judges, who assess produced poems according to a set of predefined dimensions.

In his thesis, Manurung (Reference Manurung2003) defined three fundamental properties that should be satisfied by poetic text, namely: meaningfulness – a poem must convey a conceptual message, meaningful under some interpretation – grammaticality – a poem must obey linguistic conventions prescribed by a given grammar and lexicon – and poeticness – a poem must exhibit poetic features, such as a regular metre or the presence of rhymes. While those properties can sometimes be validated by the methods applied (Manurung Reference Manurung2003; Misztal and Indurkhya Reference Misztal and Indurkhya2014), they can also be assessed by the observation of the obtained results. In fact, some researchers (Yan et al. Reference Yan, Jiang, Lapata, Lin, Lv and Li2013; Das and Gambäck Reference Das and Gambäck2014; Zhang and Lapata Reference Zhang and Lapata2014) evaluated the output of their system based on the opinion of human judges on set of poems, who answered questionnaires designed to capture Manurung’s properties. Still relying on human opinions, other authors got conclusions on the quality of their results with questions that rated slightly different dimensions, though with some overlap. Those include the typicality as a poem, understandability, quality of language, mental images, emotions and liking (Toivanen et al. Reference Toivanen, Toivonen, Valitutti and Gross2012); or structure, diction, grammar, unity, message and expressiveness (Rashel and Manurung Reference Rashel and Manurung2014).

Among the systems with outputs evaluated by humans, some ended up conducting a Turing test-like evaluation, where the scores of the systems produced by their poems were compared to those for human-created poems (Netzer et al. Reference Netzer, Gabay, Goldberg and Elhadad2009; Toivanen et al. Reference Toivanen, Toivonen, Valitutti and Gross2012; Agirrezabal et al. Reference Agirrezabal, Arrieta, Astigarraga and Hulden2013; Rashel and Manurung Reference Rashel and Manurung2014). Despite also relying on human evaluation, other researchers compared poems produced only by their systems but using different parameters or strategies (Gervás Reference Gervás2000a; Gonçalo Oliveira et al. Reference Gonçalo Oliveira, Cardoso and Pereira2007; Barbieri et al. Reference Barbieri, Pachet, Roy and Esposti2012; Yan et al. Reference Yan, Jiang, Lapata, Lin, Lv and Li2013); or poems produced by other systems with a very similar purpose (Zhang and Lapata Reference Zhang and Lapata2014).

Established methods to evaluate human creativity, from the psychology domain, have also been proposed to assess computational creativity, including automatically generated poetry. In this scope, van der Velde et al. (Reference van der Velde, Wolf, Schmettow and Nazareth2015) present a map of words related to creativity, obtained from an association study. Among others, these words may be used to define the dimensions to evaluate the creativity of a poem and its creation process. Another example is Lamb et al. (Reference Lamb, Brown and Clarke2016), who relied on a group of human experts to rate the creativity of 30 poems. Some poems were written by human authors and others generated automatically, by different creative systems, though judges were not informed of this. Despite some consensus on the best and worst poems, judges disagreed on the remaining, which made the authors unsure on the suitability of their approach for computer-generated poetry.

There has been a huge discussion on the suitability of the Turing test for evaluating computational creativity approaches (Pease and Colton Reference Pease and Colton2011). The main criticism is that it is focused on the resulting products and not on the involved creative process, which encourages the application of simpler processes, some of which might be merely concerned with tricking the human judge into thinking their outputs were produced by a human.

In addition to the previous issues, human evaluation of poetry in different languages would pose an additional difficulty. It would either require a group of judges that were fluent enough in each target language, or different but comparable groups for each languages, both highly unlikely to achieve. Though somehow limited, a possible solution is to design an automatic evaluation focused on a subset of equivalent aspects that can be measured in poems written in any of the target languages.

Besides the evaluation of the form constraints (metre and rhymes), few approaches have tried to evaluate poetry generation systems or their results automatically. The previous evaluation is often part of the generation process, as it happens with Misztal and Indurkhya (Reference Misztal and Indurkhya2014)’s or Gervás (Reference Gervás2013a)’s automatic experts, or with many other systems that assess the metre, rhymes and other properties of produced texts during generation time.

Notable exceptions on the automatic evaluation of poetry generation systems include the applications of metrics typically used in the scope of automatic summarisation and machine translation, such as ROUGE, to access the performance of Yan et al. (Reference Yan, Jiang, Lapata, Lin, Lv and Li2013)’s system, which is based on a generative summarisation method; or BLEU, to assess the ability of Zhang and Lapata (Reference Zhang and Lapata2014)’s system to generate valid sequences of lines.

3.1 Line generator module

The Line Generator module produces semantically coherent natural language fragments, with the help of the following:

• • A semantic network, managed by the Relations Manager, that connects words according to relation predicates (see Figure 2 for a very simple network, centred on the words adapt and language).

• • A line grammar, handled by the Grammar Processor, with possible ways of expressing semantic relations in text (hereafter, renderings), to be used in the generation of lines (see Figure 3 for a very simple example of a valid rule set, with eight renderings for different semantic relations).

To produce a single line, the Line Generator follows four steps:

1. (1) Select a random relation instance from the semantic network (e.g., adapt verb-synonym-of conform).

2. (2) Retrieve a rendering for this relation from the grammar, which is possible because there are renderings for each covered relation type (e.g., <arg1>, my children, and you shall <arg2>).

3. (3) Insert the relation arguments in specifically marked placeholders of the rendering (e.g., adapt, my children, and you shall conform).

4. (4) Return the resulting fragment.

Generation can be constrained by a set of words (hereafter, seeds), which will result in using only a part of the semantic network in this process, defined by the seeds and their surroundings. The modules involved in this process are described next.

Fig. 2.

Semantic network example.

Fig. 3.

Grammar example rule set.

3.2 Generation strategy module

A Generation Strategy follows an algorithm for organising produced lines such that they suit, as much as possible, the structure of a poetic form and exhibit certain poetic features. An input structure file contains the number of stanzas, lines per stanza and syllables in each line of the poem. This file may also use a symbol for denoting the target rhyme for the lines. Figure 4 shows the files for a haiku-like structure, a block of four, and an English sonnet. Both the first and the second are single-stanza structures, but the haiku-like has three lines with five, seven and five syllables, while the block has four lines with ten syllables each. The sonnet is configured as a poem with four stanzas: the first three have four lines with ten syllables, and the last one has two lines with ten syllables. The system considers consonance rhymes and a target pattern of this kind of rhyme can be specified in the structure file. In Figure 4, there is no rhyme pattern specified for the haiku, but each line of the sonnet structure has a symbol that results in the following rhyme pattern: ABBA CDCD EFEF GG. When a rhyme pattern is not specified, the Generation Strategy may, for instance, consider that a rhyme occurs when any two lines in the same stanza have the same termination. During generation, the Generation Strategy may also consider alliteration, but this is not explored in this work.

Fig. 4.

Structure files for a haiku, a block of four ten-syllable lines, and an English sonnet with a rhyme pattern.

An instantiation of the Generation Strategy module does not generate the lines. It exploits the Line Generator module to retrieve natural language fragments, which might be used as poem lines. Each instantiation implements a strategy that may differ in the number of fragments requested from the Line Generator at any time, and how they are organised into the poem structure, considering for this purpose features like metre, rhyme – based on the Syllable Utils - coherence between lines or others, depending on the desired goal.

Besides the poem structure file, the Generation Strategy has additional inputs, used to constrain the Line Generator, including a set of seed words. Before generation, this set may be expanded, in order to obtain additional relevant words that are not directly related to the seeds. This is achieved by the Seed Expander module. Expansion can be simple or biased towards an input sentiment (positive, neutral, negative), for which a polarity lexicon is used. In addition to describing the sub-modules used by the Generation Strategy, this section ends with an illustrative example of the strategy used in this work.

4.1 Portuguese instantiation

A total of 4,518 rules (renderings) were discovered for Portuguese, using the lexical-semantic network CARTÃO 3.5 (Gonçalo Oliveira et al. Reference Gonçalo Oliveira, Antón Pérez, Costa and Gomes2011), enriched with association relations, from The Leipzig Corpora Collection,Footnote ¹ and similes, extracted from the AC/DC corpora collection (Santos and Bick Reference Santos and Bick2000). The following textual collections were exploited for this purpose:

• • Poems in Versos de Segunda, a web portal dedicated to Portuguese poetry.Footnote ² These included mostly classical forms of poetry, especially sonnets and other poems that followed a strict metre, rhythm and rhyme pattern.

• • Portuguese song lyrics, transcribed in the scope of project Natura.Footnote ³ As lyrics tend to follow the rhythm of the song, these poems tend to have a higher degree of freedom, concerning their form, as compared to strict forms of poetry.

CARTÃO is a large lexical-semantic network for Portuguese, extracted automatically from three dictionaries. It covers several relation types, including not just synonymy, hypernymy and meronymy, but also others such as causation, purpose-of, or property-of. Combining CARTÃO with the associations and similes resulted in a Portuguese semantic network with 332,302 triplets.

The Morphology Lexicon LABEL-Lex (Ranchhod et al. Reference Ranchhod, Mota and Baptista1999) was used to inflect Portuguese nouns and adjectives. It contains 938,445 inflected words and their morphological properties.

In order to get the polarity of Portuguese words, we relied on SentiLex-PT02 (Silva et al. Reference Silva, Carvalho and Sarmento2012), a sentiment lexicon compiled from several publicly available Portuguese resources. It contains 82,347 entries (7,014 distinct lemmas), corresponding to words, associated with their morphological properties, including lemma and POS-tag, and predicted sentiment towards human subjects. Of the 82,347 entries, 76,738 were added manually and 5,609 automatically. About 21,000 of them are positive, 54,000 negative and 7,000 neutral.

In order to split words into syllables, to detect the most stressed syllable and to identify the words termination, SilabasPTFootnote ⁴ was used. This tool had been developed originally for the same purpose, in the Tra-la-Lyrics system (Gonçalo Oliveira et al. Reference Gonçalo Oliveira, Cardoso and Pereira2007), and already covered the operations required by the Syllable Utils module.

4.2 Spanish instantiation

For the Spanish instantiation, mostly described in Gonçalo Oliveira et al. (Reference Gonçalo Oliveira, Hervás, az and Gervás2014), a total of 1,281 relation textual renderings were discovered after exploiting 395 poems taken from an anthology of Spanish poetry on the web.Footnote ⁵ The previous extraction relied on a semantic network acquired from the Spanish Wordnet, part of the Multilingual Central Repository 3.0 (Gonzalez-Agirre et al. Reference Gonzalez-Agirre, Laparra and Rigau2012), where Wordnets from the Spanish languages and Princeton WordNet (Fellbaum Reference Fellbaum1998) are integrated. We obtained 366,126 relational triplets from the relation tables of the Spanish Wordnet, plus 58,052 synonymy pairs from words included in the same synset. After filtering some less relevant relation types (e.g., rgloss, see_also), we ended up with 102,457 triplets between lemmas.

To perform the inflection of Spanish nouns and adjectives, we used the Spanish dictionary from the FreeLing open suite (Padró and Stanilovsky Reference Padró and Stanilovsky2012), which contains 650,000 inflected word forms and their morphological properties.

In order to get the polarity of Spanish words, we used the ElhPolar Dictionary 1.0 (Urizar and Roncal Reference Urizar and Roncal2013), which contains the typical polarity for 5,210 distinct words, 1,899 positive and 3,304 negative. ElhPolar was created from the automatic translation of the positive and negative words in an English polarity lexicon, where ambiguities were manually solved, and was enriched with additional Spanish words highly associated with positive and negative tweets.

Finally, to compute the metric scansion of the poems in Spanish, in terms of syllables, the corresponding module of the WASP poetry generator (Gervás Reference Gervás2000a) was employed. This module is a re-implementation of an original set of rules designed as a logic program (Gervás Reference Gervás2000b). For its integration in PoeTryMe, an interface with the operations required by the Syllable Utils module, and shared by the Portuguese tool, was implemented.

4.3 English instantiation

For English, a total of 8,303 textual renderings were discovered from about 3,400 poems extracted from the Representative Poetry Online, a web anthology of poetry by the University of Toronto Libraries,Footnote ⁶ using a semantic network acquired from Princeton WordNet 3.0 (Fellbaum Reference Fellbaum1998). From the previous source, we used only the poems belonging to the Early Modern English and Present-Day English categories, as much of those in the older categories were written in archaic English.

To extract the semantic network from WordNet 3.0, we followed a similar procedure as for the Spanish network. We first obtained 1.2M relation triplets, which became 696,377 after filtering less relevant relation types (same as those for Spanish) and inverse relations.

In English, there are no gender specific inflections, so, for nouns, only plurals were added. For this purpose, we used the Freeling English noun dictionary, which contains 40,539 entries.

We got the polarity of English words from Bing Liu’s Opinion Words (Liu et al. Reference Liu, Hu and Cheng2005), which contains 6,800 pairs of words and their typical polarity, manually compiled over many years. More precisely, it covers 2,012 positive words and 4,788 negative.

On the other hand, metric scansion is more complex for English than for Portuguese and Spanish. While, for the latter, we could cover most cases with a rule-based approach, relying only on the orthography, for English, there are many different combinations of letters that are pronounced the same way (e.g., eye rhymes with lie, apply and levi; air rhymes with aware and bear). Therefore, in order to perform syllable division, stress and rhyme identification, we relied on the CMU Pronouncing Dictionary,Footnote ⁷ which contains over 134,000 words and their pronunciations in North American English. This involved the development of a specific parser for this dictionary and an implementation of the Syllable Utils interface, to perform the syllable related operations on English words.

This option means that, in contrast to Portuguese and Spanish, we would only retrieve the syllable-related properties of English words that are in the CMU dictionary, which has still a great coverage (more than 133,000 words). For non-covered words, our fallback mechanism uses the Portuguese rules. Still, to minimise this issue, in this work, we removed all non-covered words from the semantic network, and were left with 175,821 triplets. We empirically noticed that this has also a positive effect on the occurrence of rhymes (see Section 5.2).

4.4 Summary

Table 1 summarises the resources and tools used for each language, namely the source of the original poems, the semantic network, the polarity lexicon, the morphology lexicon and the syllables operations tool. For easier reference, we added the number of entries behind the lexicons, and Table 2 has qualitative and quantitative information on the semantic networks and generation grammars used for each language. More precisely, it shows the relation types covered, the number of triplets of each type, and the number of renderings in the generation grammar, discovered for each relation type, from the exploited poems, with the help of the semantic network. These numbers show that the resources used for each language are significantly different in terms of size and quality. The Portuguese network is substantially larger than the other two and also covers more relation types. But this does not necessarily result in a larger grammar, as the English grammar contains almost twice the number of renderings of the Portuguese and eight times more than the Spanish. For the three languages, the large majority of the renderings is for synonymy and hypernymy, which are also the relations with more triplets. All of this should make an impact in the poetry generated for each language.

Table 1.

Summary of tools and resources used for each language

Table 2.

Relation types, number of triplets in the semantic networks, and renderings in the generation grammar used for each language

5.1 Evaluation samples

Different languages are governed by different poetic traditions and thereby subject to different constraints on form. And even though some poetic forms are given the same name in different languages, there can be some variations. For instance, the sonnet, one of the most common poetic forms, has fourteen ten-syllable lines but, depending on the language, the traditional rhythm, grouping of lines or rhyme schema is not always exactly the same.

In order to evaluate the different dimensions of the resulting poems, ninety poems were generated for each language. All of them followed a relaxed interpretation of an English sonnet that would suit any language: three blocks of four lines and a final block of two lines, with a free rhyme scheme (see its structure file in Figure 4). Each sonnet was produced by a generate & test strategy, used in previous instantiations of PoeTryMe and briefly illustrated in Section 3.2.3.

The generate & test strategy was used with some fixed parameters, namely the following:

• • Neighbourhood depth (δ) =1.

• • Surprise factor (ν) =0.0005.

• • Maximum generated fragments per target line (n) = 2,000.

• • Progressive multiplier = 0.75.

• • Rhyme bonus =+2, unless when the word is the same, when it is −1.

• • Different length than the target results in a −1 penalty per syllable.

• • Alliteration bonus =0.

The combination of δ and ν means that only words directly related with the seeds were used, or, with a probability of 0.0005, words that were two edges far from them. In the generate & test process, at most 2,000 fragments were produced for the first line of each block, but this number n could increase up to n + n × 0.75 × (i − 1), for the ith line in the block. For instance, for the fourth line, n = 6, 500. Each line is scored according to the absolute difference between its number of syllables and the target number in the poem template. Alliteration was not considered here. The overall score of a poem sums the score of each of its lines plus two bonus points for each rhyme.

Two parameters were changed across the experiments: the seeds and the polarity. From an initial set of concepts suggested by each author of this paper, ten were manually selected and translated to words in the three languages. Selection had in mind the inclusion of words of different POS and from different domains. Table 3 shows the selected seeds in the three languages.

Table 3.

Seed words used for generating the poems of the evaluation sample

Finally, the seed words were expanded in the following way: for each seed, the four most relevant words according to PageRank were selected, three times considering no polarity, three with positive polarity and three with negative polarity. Of course, since we were using different semantic networks, with different structures, coverages and created by different means, the additional words for each language were not comparable. For illustrative purposes, Table 4 shows the expansions for the word artificial in each language and polarity.

Table 4.

Additional relevant words for the word artificial, in each language and polarity

In the end, we had a total of nine sonnets for each seed word (three for each selected polarity), and a total of ninety sonnets per language when using the ten seed words. The poems in Figures 6–11 illustrate the evaluation sample. All of them used the seed, the additional relevant words given the polarity (provided in each caption), or words related to the previous. Figures 6 and 7 are examples in Portuguese. The first was generated with the seed ‘artificial’ and negative polarity and uses negative synonyms of these words, such as fingida or postiça (in English, fake), and also antonyms, such as natural. Figures 8 and 9 are examples in Spanish. The first was generated with the seed ‘leer’ and a positive polarity and uses hyponyms of this verb, including dictar (in English, dictate), and other related words, such as entender (in English, to understand), which is usually positive, or adivinar (in English, to guess). Figures 10 and 11 are examples in English. Instead of our own explanation of the word choice, we put each English poem side-by-side its contextualisation, as provided by the Contextualiser module – a list of triplets that explain the semantic connection between each seed word and the other words used. When one of the words used is one level further from a seed, a → is used between the two relevant triplets.

Fig. 6.

Portuguese poem, generated with the seed ‘artificial’ and negative polarity. Additional relevant words were: fingida (fake), afectada (affected), artificiais (artificials) and artificialidade (artificiality).

Fig. 7.

Portuguese poem, generated with the seed ‘amor’ and positive polarity. Additional relevant words were: amorosos (lovely), afecto (affection), amorosidade (loveliness) and cultas (cult).

Fig. 8.

Spanish poem, generated with the seed ‘leer’ and positive polarity. Additional relevant words were: examinar (examine), adivinar (guess), reconocer (recognise) and entender (understand).

Fig. 9.

Spanish poem, generated with the seed ‘amor’ and no polarity set. Additional relevant words were: afección (affection), afectas (affected), cariño (endearment) and amores (loves).

Fig. 10.

English poem, generated with the seed ‘poetry’ and no polarity set. Additional relevant words were: rhyme, still, hush and verse.

Fig. 11.

English poem, generated with the seed ‘love’ and negative polarity. Additional relevant words were: fear, crush, fuck and object.

Looking at these samples, and given our knowledge about PoeTryMe, the system outputs can be described according to Manurung (Reference Manurung2003)’s definition of poetic text:

• • Grammaticality: the lines of the generated poems follow (roughly) grammatical and syntactic rules for each language, which is taken care by the lines grammar and the morphology lexicon.

• • Poeticness: the lines of the generated poems have a regular metre, frequent rhymes, and are organised in a poetic form, given as input. These features are assessed automatically in Section 5.2.

• • Meaningfulness: the generated poems use words semantically related with the seeds, in semantically coherent lines. But the latter is not enough to hold the property of meaningfulness. The cohesion provided by using words of the same semantic domain is not always enough for the poem to transmit a clear message as a whole, despite the high level of abstraction. This happens mainly because the generation strategy does not guarantee a logical sequence of lines, and also because different senses of the same words can be mixed. We rather argue that, though poems do not always transmit a clear message, they are related to a certain topic. This aspect is assessed automatically in Section 5.4.

5.2 Evaluation of poetic features

To evaluate the poetic features of PoeTryMe outputs, we focused on two important features of a poem: the number of syllables per line and the presence of rhymes. For each of the languages – Portuguese (PT), Spanish (ES), English (EN) – Table 5 shows the average ratio of syllables per line and Table 6 shows the average ratio of rhymes per line, both according to the used seed. The first ratio is computed as the sum of the syllables in each line of the produced poems, divided by the total number of lines. Since the number of syllables is not upper-bounded, we present the mean absolute deviation (MAD) around the ideal number, which is always ten in a sonnet and also happens to be the mode. The rhymes per line is the ratio between the number of lines that rhyme with another in the same poem and the total number of lines in a poem. In this case, the ratio is always between 0 (no rhymes) and 1 (all lines rhyme with another), so averages are presented together with the standard deviation.

Table 5.

Average ratio of syllables per line and mean absolute deviation from the ideal (ten syllables) according to language and seed or polarity

Table 6.

Average ratio of rhymes per line and standard deviation, according to language and seed or polarity

Given the line scoring function of the generate & test, lines without the target number of syllables can be used due to two main reasons: either it was not possible to find a suitable line, given the generation parameters, or there was a line with one syllable more or less but that rhymed with a previous line. In our experimentation scenario, the former is unlikely. First, because at least 2,000 lines are generated when a best match is not found. Second, because the majority of the human-created poems used to extract the grammars were sonnets, in all the languages. So there should be more than enough renderings with length close to 10.

On the presence of rhymes, the numbers show that, on average, for any seed, at least half of the lines end up in rhyme. This number is similar for Portuguese and Spanish (0.68 and 0.69), and higher for English (0.76). The higher ratio for English is likely to have benefited from the removal of triplets with words not covered by the CMU dictionary, mostly unfrequent words, some of which with terminations that are harder to rhyme with.

For Spanish, the three verbs (seeds 4, 6 and 8) were the seeds that lead to a higher rhyme ratio. This happens especially due to the following situations: (1) the termination of verbs in the infinitive is very regular in Spanish, as they always end in -ar, -er or -ir; (2) PoeTryMe does not inflect the verbs (3) using our expansion approach, a verb seed tends to be expanded with other verbs. For instance, with negative polarity, cantar (sing) is expanded with vociferar (shout), chirriar (creak), traicionar (betray), criticar (criticise), all ending in -ar.

For Portuguese, although the verbs have similar constructions, this phenomena is only clear for the seed cantar and not for the other verbs. This happens not only due to the higher number of renderings for Portuguese, but especially because there are more cross-categorical relations in the semantic network for this language, which leads to seed expansions with words of different POS. For instance, the expansion of the seed ler (read) – notícia (news), leitoras (readers), espaço (space), deixar (let) – contains just one additional verb.

Polarity does not have a significant impact on the number of syllables per line. On the other hand, it has some impact on the rhyme ratio but, curiously, the polarity with higher rhyme ratio is different for every language. For Portuguese, it is positive, for Spanish, negative, and for English, no polarity. Although we could go further to analyse these numbers for the seed/polarity combinations, for the sake of clarity, we decided to leave this analysis out of the present work.

To analyze the variation of words in each poem, Table 7 shows the average ratio between distinct words out of all the content words used in a poem.Footnote ⁸ English is clearly the language with more distinct words, followed by Portuguese and Spanish. More than looking at the size of the respective semantic networks, these numbers are explained by the highest number of renderings available for English, about 8,000 against the 4,000 for Portuguese and just about 1,000 for Spanish. For English, the seeds that lead to more distinct words are the four verbs (seeds 4, 6, and 8) and the adjective blue. The seeds that lead to less distinct words in this language, artificial and new, still have a higher average ratio than the highest ratios for Portuguese (0.70 against 0.63), curiously for the seed novo (new). In fact, Portuguese seems to have a different behaviour than English, as the lower ratios for the former language are obtained with two verbs (cantar and construir). For Spanish, only the seed fútbol leads to an average ratio higher than 0.5. Similarly to Portuguese, the seeds that lead to lower variation are the equivalent verbs (cantar and construir).

Table 7.

Average ratio of distinct words out of all the content words, according to language and seed

5.3 Evaluation of structure variation

To assess the variation of the produced text in terms of structure, we relied on evaluating the structural similarity of the poems in the evaluation samples. For this purpose, we decided to compute metrics that are typically used for evaluating automatic summarisation tasks, though in a different way and with a different purpose than Yan et al. (Reference Yan, Jiang, Lapata, Lin, Lv and Li2013). ROUGE (Recall-Oriented Understudy for Gisting Evaluation) (Lin and Och Reference Lin and Och2004) is a set of measures to determine the quality of an automatically extracted summary by comparing it to other ideal gold reference texts, created by humans. The covered measures compute the number of overlapping units, such as n-gram, word sequences and word pairs, between the computer-generated text and the gold references. This metric is appropriate in this case, as we want to compare the structural similarity of poems in terms of n-grams and the appearance of consecutive words in their lines. Therefore, if a pair of poems has a high ROUGE value, it means that the poems have very similar text. If the average ROUGE values for different poems are very high, then the poems generated by the system are too repetitive.

For this evaluation, we have used the following ROUGE measures:

• • ROUGE-N counts the number of matching n-grams in the compared texts. The n-gram lengths usually range from 1 to 4.

• • ROUGE-L is based on longest common subsequence statistics between a candidate and a reference text.

• • ROUGE-SU-N considers n-grams that do not have to be consecutive in the text, but could present a maximum of N terms between them, plus unigram-based co-occurrence statistics.

All ROUGE values range from 0 to 1, with one meaning that the compared texts are the same and, therefore, present the same n-grams in the same order. As an example, Table 8 presents the evaluation results obtained for two pairs of compared sentences:

• 

Table 8.

Examples for the evaluation metrics

When both sentences are equal (a) all ROUGE measures are 1 as all n-grams are the same. However, when one word is different (b) the change is reflected in the n-gram metrics of ROUGE. For example, with ROUGE-1 all 1-grams are the same except one, with ROUGE-2 only one bi-gram of three is the same, and there are no equal 3- or 4-grams, so ROUGE-{3,4} results are 0.

We first computed ROUGE results for each set of poems from the evaluation sample that had been generated with the same seed and target polarity. Then, inside each of those sets, each poem was compared to each other, and the average value of each metric was computed. When comparing a pair of poems, we used three different configurations:

1. (1) One-One Comparison, where each line of the first poem is compared with the line in the same position in the second poem.

2. (2) One-All Comparison, where each line of the first poem is compared with all the lines in the second poem.

3. (3) Whole Comparison, where two poems are compared as whole texts, not considering the organisation in lines.

In addition, a fourth configuration (One-Alone Comparison) was considered involving only one poem. More precisely, each poem was compared with itself by comparing each of its lines with all the other lines in the same poem. Figure 12 shows a graphical representation of the four configurations considered.

Fig. 12.

Configurations used in the evaluation of structure: (1) One-One Comparison; (2) One-All Comparison; (3) Whole Comparison and (4) One-Alone Comparison.

The results are shown in Table 9. Extra experiments were performed for calculating structural similarity of poems with different seeds or polarities, but there were no significant differences in the values obtained, so we have not included these values for the sake of clarity.

Table 9.

ROUGE results obtained for the evaluation of the structural similarity of the generated poems

The higher values are obtained with the Whole Comparison configuration. This means that the poems present a reasonable structural similarity when they are considered as a whole text, not taking into account their lines. In any case, the values obtained for this configuration are not exceedingly high if we consider that a result of 0.50 (only obtained in Spanish) means that half of the n-grams in a poem appear in another one but not necessarily in the same order. The other three configurations, which consider the lines as the units of comparison, present similar results with very low values (less than 0.10 in most cases). We can therefore conclude that the generated poems are not only different among them but varied enough in their words and structure.

Differences in the results obtained by the five metrics are expected, with slightly higher values in metrics that use smaller n-grams. For example, the number of individual words appearing in two poems or lines (R-1) will always be higher than the number of 4-grams (R-4 or R-SU4). R-L scores, that are very close to R-1 ones, suggest that, in most cases, the longest common subsequence has a length of one.

From the point of view of the differences across languages, the structural similarities are higher in Spanish, followed by Portuguese and very closely by English. This makes sense due to the lower number of renderings available for Spanish (1,000), which makes poems more ‘repetitive’, whereas there are about 4,000 renderings for Portuguese and 8,000 for English. Also, the smaller size of the Spanish semantic network, 102k triplets, in contrast to the 332k for Portuguese and the 175k for English), makes the vocabulary for Spanish less rich than for the other two languages. Nevertheless, the obtained values are quite low in all cases, meaning that the generated poems are very different from the point of view of their textual structure, a desirable property in a poetry generator.

5.4 Evaluation of topicality

One of the goals of PoeTryMe is to produce meaningful text with a semantic connection to the given seed words. In other words, the seed should define the topic the poem is about or clearly set its semantic domain. Since the lines of the poems are based on original fragments, where two words are replaced by two other words semantically related in the same way, each fragment should make sense alone. Moreover, the replacing words should be also semantically associated to the seeds, so that a coherent meaning emerges from the poem. To evaluate the semantic similarity between the generated poems and their seeds, we have applied Pointwise Mutual Information (PMI) (Church and Hanks Reference Church and Hanks1990), a statistical measure typically used to compute distributional similarity/word association. This measure considers not only the connection between the expanded words, obtained from the semantic network, and their co-occurrence with the seed in an external corpus, but also the co-occurrence between the seed and the rest of the (fixed) content words that appear in the poem.

The PMI of a pair of outcomes (e.g., words a and b) quantifies the discrepancy between the probability of their coincidence (e.g., co-occurrence, P(a, b)) given their joint distribution and their individual distributions (P(a) and P(b)), assuming their independence. Mathematically, PMI can be computed according to equation 1. In order to bound PMI values and thus ease their interpretation, we normalised this measure according to equation 2, as others (Bouma Reference Bouma2009) did previously.

On the semantic similarity domain, PMI has successfully been used to identify synonyms (Turney Reference Turney2001), or to evaluate the coherence of topic models (Newman et al. Reference Newman, Lau, Grieser and Baldwin2010), among others. In both previous cases, it has been noticed that PMI scores had a high correlation with those by human judges. When measured on a large corpus, pairs of words typically used together, and rarely alone (semantically similar), have higher PMI values than those that do not co-occur or only co-occur by accident. In particular, the NPMI between identical words is one.

For our evaluation, NPMI values were computed based on the Wikipedia editions for each of the three languages. Wikipedia is a large-coverage corpus, available in many languages, and created independently from our grammars and semantic networks. Moreover, it provides a REST APIFootnote ⁹ that eases the process of counting the number of documents where a word or a pair of words occur.

The NPMI value between each word and each content wordFootnote ¹⁰ of each poem in the evaluation samples has been computed. These values range from 1, for identical words, to 0, for words that do not co-occur in Wikipedia articles. Some typical values are listed below:

• • PT: (poema, poesía) = 0.660, (futebol, artificial) = 0.013

• • SP: (procesadores, ordenador) = 0.509, (construir, jugar) = 0.092

• • EN: (hardware, computer) = 0.484, (computer, flower) = 0.044

To assess the similarity between each poem and its original seed, we performed an information retrieval task, based on the precision at the top-nine similar poems for each seed. This means that, if all the nine poems generated by a seed are between the nine most similar in the ranking (those with high NPMI(seed, poem)), then the precision will be 1. If none of the top-nine poems was generated with the seed, precision will be 0. Table 10 shows the results obtained in this task.

Table 10.

Precision at top-nine results, obtained for the evaluation of the semantic similarity between the seeds and the generated poems

These results show that the words in poems generated by a seed tend to be more similar to this seed than the words in other poems. Therefore, the seeds play a relevant role in the process of generating poems, as it biases the meaning transmitted by them. We can observe that, on average, the highest precision is obtained for English (0.68), and the worst for Spanish (0.53), with Portuguese in the middle (0.66), though with a lower standard deviation than for English (0.19 vs 0.24). These differences are explained with the size and nature of the resources used. English is the language with the larger set of renderings, its semantic network is larger than the Spanish one and more systematic. It is also focused on fewer but more common semantic relations (synonymy, hypernymy and part-of) than the Portuguese network, largest and more heterogeneous, with most relations extracted automatically from dictionaries. Last but not least, the English Wikipedia is much larger than the Spanish and the Portuguese (4.9M articles against 1.2M and 890K, respectively), which may contribute to more accurate NPMI values.

For each language, the highest result was hundred per cent precision, but obtained with different seeds. For English, it was blue, for Spanish, leer and for Portuguese, cantar. The seed that lead to the lower precision was also different for each language. It was futebol for Portuguese (P = 0.33), artificial and azul for Spanish (P = 0.0), and artificial for English (P = 0.33). The two seeds with precision 0 for Spanish, artificial and azul, are adjectives that may modify significantly different nouns and appear in very different contexts, which might have contributed to these low results. Moreover, these seeds are neutral/objective and thus not clearly connected with words with a typical polarity. This results in poems that do not use the seed nor any word with a clear semantic connection with the seed, thus shifting the desired meaning. For instance, for the seed azul (blue) and a negative polarity, the seed set is expanded with pegar (hit), mortal (mortal), cortar (cut) and quitar (remove). This is also a limitation of the small size of the Spanish semantic network and polarity lexicon.

In general, the higher positions in the rankings are obtained for the poems generated with no target polarity. This was expected because the words obtained with the expansion are the most semantically relevant for the original seed. Still, while this behaviour was clear for Portuguese and Spanish, for English there was not a clear order in the poems generated with different target polarities. In particular, for Portuguese there were four seeds (amor, azul, futebol, poesía) where the top-three similar poems were those generated without polarity. Though different from the Portuguese seeds, for Spanish there were also four seeds (amor, cantar, ordenador, poesía) where this happen, but for English there was only one (computer).