Hostname: page-component-6898cfcb55-cbk24 Total loading time: 0 Render date: 2026-11-11T11:54:36.211Z Has data issue: false hasContentIssue false

Mostly-unsupervised statistical segmentation of Japanese kanji sequences

Published online by Cambridge University Press:  04 August 2003

RIE KUBOTA ANDO
Affiliation:
IBM Thomas J. Watson Research Center, P.O. Box 704, Yorktown Heights, NY 10598, USA e-mail: rie1@us.ibm.com
LILLIAN LEE
Affiliation:
Department of Computer Science, Cornell University, Ithaca, NY 14853-7501 USA e-mail: llee@cs.cornell.edu
Rights & Permissions [Opens in a new window]

Abstract

Core share and HTML view are not available for this content. However, as you have access to this content, a full PDF is available via the 'Save PDF' action button.

Given the lack of word delimiters in written Japanese, word segmentation is generally considered a crucial first step in processing Japanese texts. Typical Japanese segmentation algorithms rely either on a lexicon and syntactic analysis or on pre-segmented data; but these are labor-intensive, and the lexico-syntactic techniques are vulnerable to the unknown word problem. In contrast, we introduce a novel, more robust statistical method utilizing unsegmented training data. Despite its simplicity, the algorithm yields performance on long kanji sequences comparable to and sometimes surpassing that of state-of-the-art morphological analyzers over a variety of error metrics. The algorithm also outperforms another mostly-unsupervised statistical algorithm previously proposed for Chinese. Additionally, we present a two-level annotation scheme for Japanese to incorporate multiple segmentation granularities, and introduce two novel evaluation metrics, both based on the notion of a compatible bracket, that can account for multiple granularities simultaneously.

Information

Type
Research Article
Copyright
2003 Cambridge University Press