2.1 Information extraction

IE task is a subpart of natural language engineering and it focusses on extracting of structured information from unstructured documents. In case of human language texts processing, we can observe many directions of IE, with subtasks aimed at recognition of

• Named entities – Recognition of known entity name, e.g. person names, organisation names, location names. Several solutions were proposed for Polish as open source frameworks, e.g. Liner2 (Marcińczuk and Kocoń Reference Marcińczuk, Kocoń and Janicki2013; Marcińczuk, Kocoń and Janicki Reference Marcińczuk, Kocoń and Janicki2013), which consists of several universal methods for sequence chunking like dictionary look-up, statistical processing or pattern matching.

• Events – Event recognition (Andersen et al. 1992; Llorens, Saquete and Navarro-Colorado Reference Llorens, Saquete and Navarro-Colorado2010a) is an important subtask in more complex natural language processing tasks like question answeringFootnote ¹ or automatic summarisation (Daniel, Radev and Allison Reference Daniel, Radev and Allison2003).

• Temporal expressions – Many texts, mainly newspapers, describe the world by talking about people, events and states they participate in. The exact temporal locations of events are rarely explicit, many timexes are vague. Automatic extraction of timexes identifies when something occurred by the recognition and normalisation of expressions which refer to time. Often used in automatic question answering systems (Pustejovsky et al. 2005b).

The above list is not exhaustive and still the number of IE subtasks increases, because it is becoming an interesting topic for many researchers. IE allows the information from the text to be expressed in a structured way and allows joining the information from multiple sources of documents.

2.2 Temporal information

Timexes in text refer to the time. According to Mazur (Reference Mazur2012), these expressions are divided into two main categories: instants and intervals. These are atoms of time, which can be used to represent and reason about time. In the literature, we can find many terms to describe instants, e.g. a time point, a point, a point in time, a moment. Also, an interval is sometimes called period like in Benthem (Reference Benthem1983), where Benthem uses an interval as something which is between boundaries. On the other side, in Allen (Reference Allen1995), we find interval timexes in Benthem’s meaning denoted by the term duration. The main difference between instants and intervals is that instants have no duration (treated as a feature of a period).

One of the most widely used specifications for English to describe temporal information in natural language corpora is TimeML (Saurí et al. 2006). It was developed in the context of a workshop TERQAS,Footnote ² as a part of the ARDA-funded programme AQUAINTFootnote ³ in a multi-project effort to improve the performance of question answering systems over documents written in natural language (Pustejovsky et al. 2005a). The aim of this research was to improve accessing of information from text through content rather than keywords. The main problem was the recognition of events and their temporal anchorings.

Events are naturally anchored in time and these temporally grounded events are the source of information from which we can reason about the world changes. Also, other entities and their properties change over time; therefore, a database of assertions about entities will be incomplete if it does not capture how these entities are temporally updated Pustejovsky et al. (Reference Pustejovsky, Ingria, Sauri, Castano, Littman, Gaizauskas, Setzer, Katz and Mani2005a).

Previous works, focussed on temporal and event-based reasoning task, were ACL Workshop on Spatial and Temporal Reasoning (2001) and LREC Workshop “Annotation Standards for Temporal Information in Natural Language (2002)”. Several papers from these workshops were about the time representation and identification Filatova and Hovy (Reference Filatova and Hovy2001); Schilder and Habel (Reference Schilder and Habel2001). One of the results created on the basis of workshops conclusions is TimeML – Markup Language for Temporal and Event Expressions. It is designed to address these issues which remain unresolved during workshops. In the article by Pustejovsky et al. (2005a), four major problems were introduced:

• • Time stamping of events.

• • Ordering of events.

• • Reasoning with contextually underspecified timexes (such as last week, two weeks before).

• • Reasoning about the persistence of events (how long an event lasts).

We see that capturing the temporal information is important in the process of reasoning about the changes described in text documents. This document describes the first stage of the temporal information capturing and focusses on annotating timexes in text.

2.3 PLIMEX

PLIMEX is the temporal annotation language, suitable to describe timexes in Polish text documents. It is based on TIDES Instruction Manual for the Annotation of timexes Ferro (Reference Ferro2001), which describes TIMEX2 annotation format. TIDES manual is also the core of TIMEX3 annotation format, used in TimeML specification Saurí et al. (Reference Saurí, Littman, Gaizauskas, Setzer and Pustejovsky2006). Both documents present how to use the special Standard Generalised Markup Language tags to annotate timexes, by inserting them directly into the text. In section (Annotation Process), we describe our solution of annotating temporal information. We adapted types of timexes from TIMEX3: DATE, TIME, DURATION and SET. These specific types are described in Section 3.

TimeML was successfully adapted to many languages and one of the most widely used rule-based system – HeidelTime (Strötgen and Gertz Reference Strötgen, Zell and Gertz2013; Strötgen, Zell and Gertz Reference Strötgen, Zell and Gertz2013), which uses TIMEX3 annotation standard, currently supports thirteen languages (rules have been made manually): English, German, Dutch, Vietnamese, Arabic, Spanish, Italian, French, Chinese, Russian, Croatian, Estonian and Portuguese. The most recent version of that tool supports more than 200 languages Strötgen and Gertz (Reference Strötgen and Gertz2015), but these resources are of lower quality than the manually created ones. Authors claim that HeidelTime can be used as a baseline for temporal tagging of all these languages or as a starting point for developing temporal tagging capabilities for them. Our research gives the opportunity to create the cross-domain temporal tagger which supports Polish, with different approach than suggested by HeidelTime creators. Instead of creating boundaries identification rules, our system is using machine learning approaches to create a model to recognise timexes’ boundaries automatically. We have chosen that approach, because in case of languages with rich inflection (like Polish or Croatian), the number of rules which have to be created is much higher than for languages with low inflection. For example, in the article by Skukan, Glavas and Snajder (Reference Skukan, Glavas and Snajder2014), authors compare several languages by the number of rules and Croatian has got 199 rules, which is much higher than for English, but of comparable size to resources for other inflected languages, such as French, which has 157 rule resources.

3.1 DATE

DATE is a type of timex which denotes a point on a timeline, i.e. a unit of time greater than or equal to the day. The key question is when.

In general, different categories of DATE class can be mentioned. The first one considers dates, like czerwiec 2004 (June 2004) or 15.08.2014, as contextually independent. At this point, researchers of this issue agree, although they use different terminology for describing dates. The most commonly used term is fully specified (Schilder and Habel Reference Schilder and Habel2001; Han, Gates and Levin Reference Han, Gates and Levin2006), then explicit (Saquete, Muñoz and Martínez-Barco 2003; Schidler 2004) and absolute (Negri and Marseglia Reference Negri and Marseglia2005; Vicente-Diez, Samy and Martinez Reference Vicente-Diez, Samy and Martinez2008).

Other timexes are named and categorised variously. It is possible to mention types of timexes such as

1. (1) wczoraj (yesterday), jutro (tomorrow)

2. (2) [5 dni] temu (5 days ago), [5 dni] później (5 days later)

3. (3) 28 lipca (July 28)

Some researchers use only one term for all of these types, like relative Negri and Marseglia (Reference Negri and Marseglia2005), implicit Saquete et al. (Reference Saquete, Muñoz and Martínez-Barco2003), indexical Schilder (Reference Schilder2004) and others. In the article by Han et al. (2006), the term deictic is used for the type (1) and relative for types (2,3). Smith (Reference Smith2010) mentions deictic for the type (1), dependent for (2) and flexible anchoring for (3), whilst Busemann et al. (Reference Busemann, Declerck, Diagne, Dini, Klein and Schmeier1997) use deictic, anaphoric and underspecified for (1, 2, 3). Mazur (Reference Mazur2012) provides a full list of the terminology used in the literature.

We need also to mention points – timexes with vague boundaries: zima (winter), weekend (weekend). These units can be used like any type of point expression:

• fully specified – zima 2007 (winter of 2007)

• deictic – ostatnia zima (last winter)

• anaphoric – następująca zima (the following winter)

• underspecified – zima (winter)

Other examples of DATE:

1. (4) [poniedziałek, 16 marca 1985 roku] (on [Monday, 16th March 1985]);

2. (5) to wydarzyło się [drugiego listopada] (it happened on [the second of November]);

3. (6) w [październiku 1963 roku] (in [October 1963]);

4. (7) to będzie we [wtorek osiemnastego] (it will be on [Tuesday, the eighteenth]);

5. (8) byłem nad jeziorem [latem tamtego roku] (I was at the lake in [the summer of that year]).

3.2 TIME

It is type of point expression that describes timexes which refer to the time of day, even if it is not clearly defined. The key question is also when. For example, Smith wrócił (Smith returned):

1. (9) [za dziesięć trzecia] (at [ten to three])

2. (10) [dwadzieścia po dwunastej] (at [twenty past twelve])

3. (11) o [ósmej rano] (at [eight in the morning])

4. (12) o [9.00 w piątek 1 października 1999 roku] (at [9 am on Friday, October 1st, 1999])

5. (13) [wczoraj późno w nocy] ([yesterday late at night])

6. (14) [wczoraj w nocy] ([last night])

3.3 DURATION

DURATION, in contrast to DATE, has two points on a timeline associated with it – a start and an end point. A different name used in literature is period Saquete et al. (Reference Saquete, Muñoz and Martínez-Barco2003). The key question is how long.

Sometimes range expressions are also included to this group (Mizobuchi et al. 1998), but these expressions can be treated as separate points in time Mani and Wilson (Reference Mani and Wilson2000). For example, Smith był tutaj (Smith stayed there):

1. (15) [dwa miesiące] (for [two months])

2. (16) [48 godzin] (for [48 hours])

3. (17) [trzy tygodnie] (for [three weeks])

4. (18) [całą ostatnią noc] ([all last night])

5. (19) [20 dni] w lipcu ([20 days] in July)

6. (20) przez [trzy godziny] w zeszły poniedziałek (for [three hours] last Monday)

If a specific piece of information, which relates to the calendar, occurs in timex, then DATE is the right type of annotation. This is true even if the context suggests that this type of timexes indicates the duration of an event, e.g. [Cały 1985] przebywał na emigracji ([The entire 1985] he lived in exile).

3.4 SET

The SET expression is a type of timex, relating to more than one instance of a time unit – either a point or a period. The key question is how often.

This type of an expression is named variously, but the most popular name is SET Mani and Wilson (Reference Mani and Wilson2000). Han (Reference Han, Gates and Levin2006) also distinguished (inside of this type):

• recurrence – e.g. co poniedziałek (every Monday)

• rate – e.g. trzy razy na miesiąc (three times per month)

Other subtypes are given by Niemi and Koskenniemi (Reference Niemi and Koskenniemi2007):

• parity expressions – e.g. nieparzyste poniedziałki miesiąca (odd Mondays of the month)

• ordinal expressions – e.g. co czwarty rok stulecia (every fourth year of the century)

• containment expressions – e.g. lata z 29 lutego (the years with a 29th of February)

Examples – Jan wraca pijany (John comes back drunk):

1. (21) dwa razy w tygodniu (twice a week)

2. (22) co dwa dni (every two days)

3. (23) każdej niedzieli (every Sunday)

4.1 What to annotate

Our purpose is to annotate expressions which refer to time. We define three types of timexes:

• obvious and precise – it is possible to determine the exact meaning of the expression, for example,

  1. (24) W dniu [15.11.1999 r.] doszło do wypadku (On [15.11.1999] there was an accident)

  2. (25) Przez [dwa dni] nie mógł mówić (He could not speak for [two days])

• obvious and not precise – it is not possible to determine the exact meaning of the expression (fuzzy reference to time) with the obvious type of time quantification (e.g. year, month, day, etc.), as it was underlined in the following example:

  1. (26) Przez [kilka lat] pozostawał w ukryciu (He remained in hiding for [several years])

  2. (27) Miał czekać za drzwiami przez [następnych parę minut] (He had to wait behind the door for the [next few minutes])

• not obvious – we do not know the type of time quantification, for example,

  1. (28) W przeszłym okresie byli do tego zdolni (In the past period they were capable of this)

We decided not to annotate timexes which are not obvious and we focussed only on the following types: DATE, TIME, DURATION and SET.

4.2 The interpretation of timexes

The purpose of the interpretation of timexes is to achieve the contextual meaning of those expressions. From the information processing point of view, it is a case of normalisation of these timexes. They need to be given in an unambiguous format that can be understood by a computer. A solution that takes both the local and the global semantics of an expression into account needs to be adopted.

The local semantics is the result of an analysis of a timex without taking the context of the document into account. Only tokens building the timex and tokens forming a sentence in which the expression occurs are taken into account. One of the proposed specifications which takes the semantics of this type into account is the LTIMEX Mazur (Reference Mazur2012).

In the case of a determination of the global semantics, the context of the document and, in some cases, other sources of knowledge (e.g. dates of holidays and well known events) are taken into account. This level of semantic meaning is defined in the TimeML specification for both TIMEX2 and TIMEX3.

We need to adopt the normalisation process of timexes which takes into account the interpretation at both the local and the global levels. First, the local semantics is determined by an analysis of the structure of a timex and of a local context. Then, the expression is analysed in the context of the entire document, taking into account other sources of knowledge (geolocation, calendar, etc.).

4.3 The local and the global semantics

The semantic meaning of a timex is determined independently from the information concerning the context of the expression. The local semantics reflects the meaning of the individual tokens forming the expression. Sometimes the local and the global semantics are the same, especially for unambiguous points in time:

1. (29) Pierwsza wojna światowa wybuchła [28 lipca 1914 roku] (The First World War broke out on [July 28, 1914])

2. (30) Przysłał nam swoje zdjęcia z [lat dziewięćdziesiątych dwudziestego wieku] (He sent us his pictures from [the 1990’s])

Expressions which are context-dependent do not contain enough information in order to obtain the global semantics. Only the contextual analysis of these expressions makes the definition of an exact point in time:

1. (31) Otwarcie gospodarcze stało się faktem na początku [lat dziewięćdziesiątych] (The opening up of the economy became a reality at the beginning of [the 90’s])

2. (32) Byłem [wczoraj] w kinie na ciekawym filmie ([Yesterday] I was at the cinema for an interesting film)

The local semantics are always the same, regardless of the broader context. For example, lipiec (July) always refers to the seventh month of the year and wczoraj (yesterday) always refers to the day before the current date. The global semantics depends on the context and, for that reason, the interpretation of these examples may differ in different contexts. The value of such an expression will be different, depending on the context. In the case of vague expressions, the interpretation process assumes the determination of a maximally precise meaning of the timex. The global semantics must take into account the local information and all of the information that can be deduced from an analysis of the document. For example, at the local semantics level, expression 7 rano w listopadowy poranek (7 am in the morning in November) specifies a month and an hour. To determine the global semantics, other information (different than the hour and the month, e.g. the day and the year) must occur in a broader context and must refer to the analysed expression. Sometimes that information does not exist, and in such cases the local and the global semantics are the same. The components of the reference date (date of a creation of the document or the date in the text) do not always specify a date directly. Consider the following sentence:

1. (33) Wyjeżdżamy do Nowego Jorku w [lutym] (We go to New York in [February])

If this sentence is derived from a document created in April 2014, the timex in this sentence refers to February 2015.

Finding the right reference date is important in the normalisation process of timexes, for example,

• • Reference to the date of the utterance, depending on the document’s source. It can be the document creation time, the date of the publication or the date of the last modification (especially in the case of newspaper articles), but also the date of sending an e-mail or a date of the communicator message, etc. In general, all these dates are mentioned in TimeML specification (TIMEX3) as document creation time.

• • Reference to another timex in the text (timexes that directly relate to other timex used in the text), for example,

  1. (34) Zbigniew urodził się 13 grudnia 1942 roku. Trzy dni wcześniej zmarł jego ojciec. (Zbigniew was born on December 13, 1942. His father had died three days earlier.)

  It is necessary to determine the correct meaning of the timex A in the first sentence and to refer the local semantics of the timexes B in the second sentence to the date set by A, in order to determine the global semantics B.

• • Reference to the event described in the text (timexes that refer to elements described in the TimeML specification as descriptions of events), for example,

  1. (35) [Trzy dni] po wypadku Zosia odzyskała przytomność (Zosia regained consciousness [three days] after the accident)

  In this example, the term wypadek (accident) is an event description and refers to the change of state in reality. As long as it is not possible to directly determine when the accident occured, it is not possible to determine the global semantics in the timex trzy dni (three days). However, these types of expressions are not studied in this article.

4.4 Representation of the global semantics

In this article, the global semantics are adopted, as e.g. in the TimeML specification, in order to describe timexes (TIMEX2 and TIMEX3, both of which include the same normalisation format). The attribute VAL (Value) is important in the normalisation context. It specifies the normalised value of a timex. VAL is a textual representation of a timex, which is assigned using guidelines described in the ISO-8601 standard. In order to describe it, the first letters of the English (but not only English) names that specify time (the case of letters is important) are used (e.g. Year, Month, Day, hour, minute, BC – before Christ, AD – Anno Domini, Time, MOrning, Half, Quarter, SUmmer, Period).

Each timex can be determined with respect to the proper type, using the coding proposed in the ISO-8601 standard, for example,

• • calendar date: YYYY-MM-DD,

• • week of the year: YYYY-Wxx,

• • hour: hh:mm:ss,

• • date and hour: YYYY-MM-DDThh:mm:ss,

• • duration (the number of weeks): PxW.

Table 1 shows example values of the VAL attribute and the semantic meaning Mazur (Reference Mazur2012).

Table 1.

Examples of VAL attributes and the semantic meaning of timexes. VAL values can be assigned manually during annotation or by automatic system (see Section 8.3)

Both standards (TIMEX2 and TIMEX3) expand the ISO-8601 standard, such that they have the ability to save a normalised form of vague expressions, using an X sign to identify those elements of notation which are unknown. For example, a sunny day in June can be written as XXXX-06-XX. This standard was also expanded for new time tokens, such as the dekada (DEcade), wiek (CEntury) and tysiąclecie (ML – millennium). Normalisation in the TimeML standard involves the determination of the global semantics for a timex. There is no indirect form of notation of the local semantics. The introduction of the intermediate stage of determining the global semantics is reasonable from the normalisation point of view. It can be seen in systems that recognise timexes in the English language (e.g. HeidelTime,Footnote ⁴ Strötgen and Gertz (Reference Strötgen and Gertz2013)), which often use their own intermediate standard of normalisation. In this article, LTIMEX standard proposed by Mazur (Reference Mazur2012) is adapted. It can be used to determine the local semantics of timexes.

4.5 Representation of the Llocal semantics

LTIMEX standard was designed to be compatible with existing annotation schemes, especially TIMEX2 (and TIMEX3). It is beneficial for both design and evaluation to recognise the semantics of the expression with no context involved, what is called local semantics, representing the partial and underspecified context-free meaning of timexes Mazur (Reference Mazur2012). The compatibility with the existing schemes has two purposes:

• • It is human-readable and requires minimum effort to use for annotators familiar with TIMEX3.

• • It provides a relatively easy means of conversion from local semantics to global semantics.

LTIMEX specification extends the set of TIMEX3 attributes in order to describe the local meaning of timexes. One of the most important extensions is LVAL (local value) attribute. With this attribute, it is possible to obtain the local representation of timexes in the following way:

• • Underspecified values are completed with x sign (i.e. November: xxxx-11).

• • Underspecified time is separated from date by the lowercase t (i.e. five past nine: xxxx-xx-xxt09:05).

• • Time offsets are represented using the following signs: + − >< (i.e. three days earlier: -0000-00-03; next Wednesday: >D3).

• • Ordered expressions are represented using the number, sometimes followed by $ sign (i.e. third day: 3D; last day: $1D).

If the timexes is explicit, there is no difference between LVAL and VAL representation. The specification is fully described in work Mazur (Reference Mazur2012). Table 2 shows the example values of the LVAL attribute and the semantic meaning.

Table 2.

Examples of LVAL attributes and the semantic meaning of timexes for the types described in Section 4.5. LVAL values can be assigned manually during annotation or by automatic system (see Section 8.3)

5.1 Inforex and KPWr

Inforex (Marcińczuk, Kocoń and Broda Reference Marcińczuk, Kocoń and Broda2012) is a web-based system designed for managing and annotating text corpora at the semantic level including annotation of Named Entities, anaphora, Word Sense Disambiguation (WSD) and relations between named entities. The system also supports manual text clean-up and automatic text pre-processing including text segmentation, morphosyntactic analysis and word selection for WSD annotation. Inforex allows adding custom metadata to annotations (e.g. local and global meaning of timexes). Inforex uses CCLFootnote ⁵ annotation format, which differs from format suggested by TimeML specification for English, but it is compatible with our Liner2 tool, which was adapted to a problem of the recognition of timexes (see Section 6).

KPWr – Korpus Języka Polskiego Politechniki Wrocławskiej (Polish Corpus of Wroclaw University of Technology) (Broda et al. 2012; Radziszewski, Maziarz and Wieczorek Reference Radziszewski, Maziarz and Wieczorek2012) is a corpus of written and spoken documents available on the Creative Commons license. Documents are divided into fourteen categories and annotated at the level of shallow syntactic chunks, selected predicate-argument relations, named entities, relations between named entities, anaphora relations, word senses, timexes and events.

5.2 Annotation statistics

Currently, all documents from KPWr (1637) were annotated with timexes and each annotation was described with the local and global meaning of the related timex (LVAL and VAL attributes described in Section 4.4). We compared statistics after the annotation process with other publicly available corpora annotated using TIMEX3 annotation format. The evaluation data was prepared for the Semantic Evaluation 2013 WorkshopFootnote ⁶ evaluation tasks. One of the proposed tasks was TempEval-3.Footnote ⁷ The aim of that task was to advance research on temporal information processing which could increase the quality of applications like question answering, textual entailment, text summarisation, etc. It was a three-part task covering event, timex and temporal relation extraction, using the complete TimeML specification for English and Spanish. The result of the evaluation is presented in UzZaman et al. (2012).

Table 3 shows the result of analysis of datasets provided by SemEval 2013 organisers, compared to KPWr data. We provide a short description of these from the main TimeML websiteFootnote ⁸ and the article UzZaman et al. (Reference UzZaman, Llorens, Allen, Derczynski, Verhagen and Pustejovsky2012).

Table 3.

The result of analysis of the dataset provided by SemEval 2013 organisers, compared to KPWr data (see Section 5.2). ∑ all is the sum of dates, times, durations and sets in each corpus. ∑ corp is the number of dates, times, durations and sets in all corpora except KPWr. % means what part of all timexes covers particular type. Languages: ENglish, SPanish, PL – Polish. Corpora: Aquaint, TB – TimeBank, T3Platinum, T3Silver, T3Spanish, KPWr – Corpus of Wroclaw University of Technology

Several TimeML corpora have been created over the years. TimeBank started as an illustration and proof of concept of the TimeML specifications. TimeBank 1.1 was created in the early days of TimeML and follows the 1.1 version of the specifications. The more recent TimeBank 1.2 and the AQUAINT corpus were compiled following the 1.2.1 specifications. The TempEval corpus was created for the SemEval-2007 workshop at the ACL in Prague. It contains the same documents as TimeBank 1.2 but uses a simplified set of temporal relations, grouped in three separate tasks. The TempEval-2 corpus is a multilingual corpus created for the SemEval-2010 workshop in Uppsala. We compared these corpora:

• AQUAINT TimeML Corpus – The AQUAINT TimeBank contains seventy-three news report documents. It is very similar in content to, and uses the same specifications as, TimeBank 1.2. This corpus is sometimes referred to as the Opinion Corpus.

• TimeBank 1.2 – The TimeBank 1.2 Corpus contains 183 news articles that have been annotated following the TimeML 1.2.1 specification Saurí et al. (Reference Saurí, Littman, Gaizauskas, Setzer and Pustejovsky2006). TimeBank 1.2 is free and it is distributed by the Linguistic Data Consortium.

• T3Silver – The TempEval-3 silver evaluation corpus is a 600K word corpus collected from Gigaword.Footnote ⁹ This corpus was automatically annotated by TIPSem, TIPSem-B Llorens et al. (Reference Llorens, Saquete and Navarro-Colorado2013) and TRIOS UzZaman and Allen (Reference UzZaman and Allen2010). These systems were retrained on the corrected TimeBank and AQUAINT corpus to generate the original TimeML temporal relation set. Then, these three state-of-the-art system outputs were merged UzZaman et al. (Reference UzZaman, Llorens, Allen, Derczynski, Verhagen and Pustejovsky2012). In a selected merged configuration, all entities and relations suggested by the best system (TIPSem) were added in the merged output. Suggestions from other systems (TRIOS and TIPSem-B) were added in the merged output, only if they were also supported by another system.

• T3Spanish – For Spanish, Spanish TimeBank 1.0 corpusFootnote ¹⁰ was used. It is the same corpus that was used in TempEval-2, with a major review of entity annotation and an important improvement regarding temporal relation annotation. For TempEval-3, ESTimeBank link types were converted to the TimeML standard types.

• T3Platinum – The TempEval-3 platinum evaluation corpus was annotated/reviewed by the SemEval 2013 organisers, who are experts in the area. This process used the TimeML Annotation Guidelines v1.2.1 Saurí et al. (Reference Saurí, Littman, Gaizauskas, Setzer and Pustejovsky2006). Every file was annotated independently by at least two expert annotators, and a third was dedicated to adjudicating between annotations and merging the final result.

• KPWr – Korpus Języka Polskiego Politechniki Wrocławskiej (Polish Corpus of Wroclaw University of Technology) Broda et al. (Reference Broda, Marcińczuk, Maziarz, Radziszewski and Wardyński2012); Radziszewski et al. (Reference Radziszewski, Maziarz and Wieczorek2012) is a corpus of written and spoken documents available on the Creative Commons license.

The results of corpora analysis shows that the distribution of timexes in corpora for both English and Spanish is similar to KPWr. It is worth mentioning, that T3Silver corpus is a machine-annotated and automatically merged dataset based on outputs of multiple systems UzZaman et al. (Reference UzZaman, Llorens, Allen, Derczynski, Verhagen and Pustejovsky2012), whereas each document in KPWr was annotated manually with timexes.

5.3 Inter-annotator agreement

The inter-annotator agreement was measured on randomly selected hundred documents from the Corpus of Wrocław University of Technology called KPWr. We used the positive specific agreement Hripcsak and Rothschild (Reference Hripcsak and Rothschild2005) as it was measured for T3Platinum corpus UzZaman et al. (Reference UzZaman, Llorens, Allen, Derczynski, Verhagen and Pustejovsky2012) and two domain experts to annotate the subset of hundred documents from KPWr. We calculated the value of positive specific agreement for each category. The results are presented in Table 4.

Table 4.

The value of positive specific agreement (PSA) calculated on the subset of hundred documents from KPWr, annotated independently by two domain experts using PLIMEX 1.0 guidelines. 1 and 2 means all annotations in which annotators 1 and 2 agreed at the level of type and boundaries. Only 1 is the number of annotations made only by annotator 1 and only 2 – the number of annotations made only by annotator 2

According to UzZaman et al. (2012), the best quality of data was achieved for TempEval-3 platinum corpus (T3Platinum) and it was annotated and reviewed by the organisers. Every file was annotated independently by at least two expert annotators. The result of overall T3Platinum inter-annotator positive specific agreement at the level of annotating timexes with types was eighty-eight per cent. In our case, for hundred randomly selected documents, the positive specific agreement value achieved was 86.25 per cent (annotating using PLIMEX 1.0 specification).

6.1 Features

In recognition, the value of features is obtained at token level. We used the following features as a baseline. These features were used in Named Entities Recognition task during our previous research Marcińczuk and Kocoń (Reference Marcińczuk and Kocoń2013); Marcińczuk et al. (Reference Marcińczuk, Kocoń and Janicki2013).

• Morphosyntactic – lemma, grammatical class, case, number, gender.

• Orthographic – word, word shape (pattern), prefix, suffix, starts with upper case, starts with lower case, starts with symbol, starts with digit, has upper case, has symbol, has digit.

• Semantic – word synonym, hypernym.

• Dictionary – person first name, person last name, country name, city name, road name, person prefix, country prefix, person noun, person suffix, road prefix, specific triggers (country, district, geographic name, organisation name, person name, region, settlement).

We decided to implement special features, which better characterise timexes’ constituents:

• Orthographic

  is_number – is word a number,

  structure – more sophisticated pattern, where each character composing a word is converted to: x (if a character is a letter), d (if a character is a digit), - (in other cases),

  structure_packed – same as above, except that each sequence of the same characters in structure is converted to a single character, e.g. ddd → d,

  all_upper – a word contains only upper case letters,

  all_letters – a word contains only letters,

  all_digits – a word contains only digits,

  all_alphanum – a word contains only alphanumeric characters,

  no_letters – a word does not contain letters,

  no_alphanum – a word does not contain alphanumeric characters

  length – a word length.

• Semantic

  tophyper – This feature uses plWordNet (Maziarz et al. 2013; Piasecki et al. 2014) to find the possible root of the given word in a graph built from the hyponymy relations joining lexical units in plWordNet. This process is currently not preceded by WSD (Kędzia et al. 2014).

• Dictionary

  timex – A lexicon prepared by domain expert, which contains words referring to time, e.g. godzina (Eng. hour), minuta (Eng. minute), etc.

8.1 Baseline versus new features

The baseline models use a set of features used for named entity recognition for Polish Marcińczuk and Kocoń (Reference Marcińczuk and Kocoń2013); Marcińczuk et al. (Reference Marcińczuk, Kocoń and Janicki2013). We added new features (described in Section 6.1) to a baseline set. Table 6 shows the result of strict and MUC match evaluation of timexes recognition, performed as ten-fold cross-validation on train data (see Table 5). It presents results of four models. Two types of models are: 4-class (boundaries recognition and classification of timexes; available classes: date, time, duration, set) and 1-class (boundaries recognition only, all classes casted to a single timex class). For each type, we compare results for the following set of features: baseline and baseline+new features. We used the following quality measures: Precision, Recall, F ₁score.

Table 6.

Strict and MUC match evaluation of TIMEX recognition (ten-fold cross-validation on train data set) – comparison of 1-class and 4-class models utilising the following feature sets: B – baseline; B+N – baseline+new features. Quality measures: Precision, R ecall, F ₁ score

8.2 Features after selection

Table 7 shows the result of the feature selection for TIMEX recognition. The procedure was performed for both 1-class and 4-class model. The initial set of features for the first phase of selection was baseline+new features. We used average strict F ₁score of ten-fold cross-validation on train data to evaluate the result after each step of the selection. Window size is the number of tokens representing context surrounding the target word Li, Kipper-Schuler and Savova (Reference Li, Kipper-Schuler and Savova2008) – in our case window size is 5, and surrounding token 0 relative indexes used in Table 7 are [-2, -1, 0, 1, 2].

Table 7.

Result of the feature selection (Phase 1 and Phase 2) based on TIMEX recognition evaluation in each Iteration for two Model types: 1 -class (boundaries recognition) and 4 -class. Used measure: average strict F ₁score of ten-fold cross-validation on train data. Initial set of features for the first phase of selection: baseline+new features. Indexes in the second phase are relative to a current token in a context window. The general description of features can be found in Section 6.1 and the detailed description of selected features is in Section 8.2. Gain is an increase of F ₁-measure after Ith iteration

We see, in Table 7, that only few of features from the full list presented in Section 6.1 were selected:

• hypernymN – A hypernym of input word W in Polish WordNet, Piasecki et al. (Reference Piasecki, Maziarz, Szpakowicz and Rudnicka2014), where N is the distance from W. Because there is no WSD, in case of more than one possible result, the procedure selects the first word from all possibilities ordered alphabetically. If a hypernym is not present, a result is a lemma of an input word W.

• pattern – For input word W, there are some patterns defined:

  • ALL_UPPER, AU – a word contains only upper-case letters,

  • ALL_LOWER, AL – a word contains only lower-case letters,

  • DIGITS, D – a word contains only digits,

  • SYMBOLS, S – a word contains only symbols,

  • UPPER_INIT, UI – the first letter of a word is upper-case, the rest is lower-case, e.g. Anna,

  • UPPER_CAMEL_CASE, UC – the first letter is upper-case, the rest is lower-case and upper-case, e.g. CamelCase,

  • MIXED, M – in other cases.

• dict_timex_base, DTB – a dictionary feature, which value is given in IOB format if the stream of base forms from the input text exists in a dictionary prepared by domain experts,

• tophyperN – this feature uses Polish WordNet to find the possible root (the most general hypernym) of the given word in a graph built from the hyponymy relations joining lexical units in plWordNet. This process is currently not preceded by WSD, so if there is more than one possible root, the procedure selects N-th word from all possibilities ordered alphabetically,

• case – a case of the input word,

• struct – more sophisticated pattern, where each character composing a word is converted to: x (if character is a letter), d (if character is a digit), - (in other cases),

• structP – packed structure, same as above, except that each sequence of the same characters in structure is converted to a single character, e.g. ddd → d,

• dict-trigger-int-district, DTID – a dictionary feature, which value is given in IOB format if the stream of base forms from the input text exists in a dictionary of potential words which are triggers of districts, e.g. okręg, rejon (Eng. area,region),

• starts-with-upper-case, SWUC – a boolean feature which value is true if an input word starts with upper-case letter,

• prefixN – first N letters of an input word W,

The interesting thing is that despite the richer morphology in Polish (e.g. comparing to English), the most important features in the recognition task are orthographic (prefixN, struct, pattern, structP) and semantic (hypernymN, tophyperN) features. Observations are similar to these presented in the article UzZaman et al. (Reference UzZaman, Llorens, Derczynski, Verhagen, Allen and Pustejovsky2013), that semantic features play important role and the best data-driven systems include them. These features are highly dependent on the quality of the resource used to determine the value of the feature – in case of hypernym and tophyper, it is Polish Wordnet,Footnote ¹³ which is currently greater than Princeton Wordnet (comparing both the number of synsets and links between them). Another observation in UzZaman et al. (Reference UzZaman, Llorens, Derczynski, Verhagen, Allen and Pustejovsky2013) is that syntactic and gazetteers do not provide any statistically significant increment of performance with respect to the morphological features alone. In our case, the dictionary with some Polish constituents of timexes prepared by domain experts has been selected as the important feature.

If the index is given, only some words surrounding the described token (with index 0) in a window compose a value of the feature. If the index contains more than one value, it is a complex feature, which is a composition of feature values for words with indexes represented by more than one value. Consider a sentence: Catalina urodziła się w Bogocie 19 kwietnia 1981 r. (Eng. Catalina was born in Bogota on 19 April 1981.). Table 8 shows that sentence divided into tokens with values of the selected features. The second row in this table shows the index relative to token 6. In case of token 6, the value of feature structP for index 0 is d, and the value of feature structP for index -1/0 is Xx/d (a complex feature described in Section 7).

Table 8.

Example sentence Catalina urodziła się w Bogocie 19 kwietnia 1981 r. (Eng. Catalina was born in Bogota on April 19, 1981.) divided into tokens, with values of features selected in each phase of the feature selection (see Section 8.2)

Table 9 shows strict and MUC match evaluation of TIMEX recognition (ten-fold cross-validation on train data set) – comparison of 1-class and 4-class models utilising the following feature sets: baseline+new features, remaining features after the first phase of the feature selection and remaining features after the second phase of the feature selection.

Table 9.

Strict and MUC match evaluation of TIMEX recognition (ten-fold cross-validation on train data set) – comparison of 1-class and 4-class models utilising the following feature sets: B+N – baseline+new features, 1st – remaining features after the first phase of the feature selection, 2nd – remaining features after the second phase of the feature selection. Quality measures: Precision, Recall, F ₁score. Results in bold are the highest in comparison to other evaluation variants

Results in Table 9 show that we do not observe better quality after selection in fifteen of thirty-six evaluation variants. We do not count the statistical significance here, because feature selection was performed on train data. In Section 9, we provide a detailed comparison of different feature sets performed on a test set, together with the analysis of training and processing time.

8.3 Normalisation

We prepared module for Liner2 tool to perform the normalisation process in order to get the local and global meaning of timexes (see Section 4). We created the normalisation process as a two-step rule-based approach. The first step is to get the local meaning of the timexes, and the second (created on the basis of the local interpretation) is to get the global meaning. Rules were created by domain experts. For each class of timexes, we defined several files with rules, patterns and normalisation dictionaries. The number of these items are presented in Table 10.

Table 10.

Normalisation resources for determining local value (LVAL) of temporal expressions, prepared using train data set

Example of date class normalisation rule for the description of the part of the age, e.g. pierwsza połowa XV wieku (Eng. the first half of the fifteenth century):

If the recognised timexes matches the extract part of the rule (which contains patterns starting with $ symbol), then extracted groups are normalised if they match any of the records in the normalisation dictionary and the final local value is composed using normalise part of the rule. In this example, lemmas of the input expression are pierwszy połowa XV wiek and 0 group is matched by $rePart and 1 group is matched by $reAgeNumber. Then, $rePart is normalised using dictionary record ”pierwszy połowa”:”H1” and $reAgeNumber is normalised using dictionary record ”pierwszy połowa”:”H1” and $reAgeNumber is normalised using dictionary record ”^xv$”:”14”. Finally, normalised groups are combined using normalise part of the rule @group(1)-@group(0) and the final local LVAL value is 14-H1.

Currently, there are 224 rules for local normalisation, for three classes of timexes: date, time, duration. Global normalisation is made as a Java code and contains sixteen rules for two classes: date, time (for duration local and global interpretation is the same). Rules were both created and tuned using train data set (see Table 5). We evaluated rules on test set. The evaluation was fully described in the article UzZaman et al. (Reference UzZaman, Llorens, Derczynski, Verhagen, Allen and Pustejovsky2013). We performed the evaluation on a full pipeline for the best model to recognise boundaries of entities and their types (trained on train+tune data set). Detailed evaluation of each class of timexes is described in Section 9. Here, we focus on the end-to-end comparison with other systems presented in the article UzZaman et al. (Reference UzZaman, Llorens, Derczynski, Verhagen, Allen and Pustejovsky2013).

To evaluate timexes, we need to evaluate how many entites are correctly identified, if the extents for the entities are correctly identified and how many attributes are correctly identified. We use classical precision and recall for recognition UzZaman et al. (Reference UzZaman, Llorens, Derczynski, Verhagen, Allen and Pustejovsky2013):

• How many entities are correctly identified: We evaluate our entites using the entity-based evaluation with equations below:

  Precision = $\frac{|Sys_{entity} \cap Ref_{entity}|}{|Sys_{entity}|}$ ,

  Recall = $\frac{|Sys_{entity} \cap Ref_{entity}|}{|Ref_{entity}|}$ ,

  where Sys_entity contains entities extracted by the system that we want to evaluate, and Ref_entity contains entities from the reference annotation that are being compared.

• If the extents for the entities are correctly identified: We compare our entities with both strict match and relaxed match. When there is an exact match between the system entity and gold entity, then we call it strict match, e.g. Sunday morning versus Sunday morning. When there is an overlap between the system entity and gold entity, then we call it relaxed match, e.g. Sunday morning vesus Sunday. When there is a relaxed match, we compare the attribute values.

• How many entity attributes are correctly identified: We evaluate our entity attributes using the F-measure which captures how well the system identified both the entity and attribute (attr) together.

  Attribute Precision P = $\frac{|\lbrace \forall x | x \in (Sys_{entity} \cap Ref_{entity}) \wedge Sys_{attr}(x) == Ref_{attr}(x) \rbrace |}{|Sys_{entity}|}$ ,

  Attribute Recall R = $\frac{|\lbrace \forall x | x \in (Sys_{entity} \cap Ref_{entity}) \wedge Sys_{attr}(x) == Ref_{attr}(x) \rbrace |}{|Ref_{entity}|}$ ,

  Attribute F-measure F1 = $\frac{2 * P * R}{P + R}$ .

Results in Table 11 show, that our system achieves similar quality as other systems in case of extraction of timexes. It is not easy to compare the quality for different languages, but the result shows, that the complexity of that task for Polish is more close to English in case of normalisation. A strict F-measure of Liner2 tool for the extraction task is very high (85.06 per cent) and it is closest to the best system HeidelTime for Spanish (85.30 per cent) and outperforms the best data-driven system TIPSem for Spanish (82.60 per cent). In case of global normalisation, there is probably a lot of room for improvement, mainly from the perspective of validating the training data. Still guidelines are not precise enough in case of determining the global value and it requires further work to prepare the exact procedure of determining VAL value for timexes.

Table 11.

Evaluation of the end-to-end systems for recognition and normalisation of temporal expressions – comparison of Liner2 tool with other systems presented during SemEval 2013 (only the best results regarding VAL F1 measure for each tool). Extraction of temporal entities in these systems are: RB – rule-based, DD – data-driven, H – hybrid. Languages: SPanish, ENglish, PL – Polish. We used strict and relaxed variants of evaluation measures presented in Section 8.3. VAL and LVAL values are evaluated using attribute measures presented in Section 8.3