Rationale for DocuskyFootnote ¹

The past two decades have seen many digitization projects, resulting in a large number of digital archives available over the Web. Many of these digital collections not only have high-quality digital content, but are also equipped with sophisticated tools for analyzing the content and exploring the textual contexts of (relationships between) documents. Some examples in Sinology are ctext.orgFootnote ² for the Chinese classics, CBETAFootnote ³ for the Buddhist canon, and THDLFootnote ⁴ (Taiwan History Digital Library) for Taiwanese historical documents.

Although these digital archive systems significantly expanded a scholar's ability to access and analyze research material, they are not without drawbacks. There are two main problems. The first of these is the difficulty of dealing with a researcher's personal collection of data. A digital archive system is a closed system. Its content is collected and organized according to certain criteria, and tools are designed to work with its content. The content and tools interact within the confines of the system itself, while adding new content and new tools can only be done by the developers (or administrators) of the system. However, scholars often have their own personal collection of research material. Even if a tool in a specific archive system fits a scholar's need precisely, it is difficult to apply that tool to contents that are not already in the system. The second problem is the inability to cater to specific research problems. Scholars usually have their own research objectives. Different objectives lead to different emphases when collecting research material. On the other hand, the digital content of a digital archive system is often developed around a specific domain of interest, and its collection strategy is usually to make it as comprehensive as possible so that it can be used by scholars for different research purposes. In doing so, however, a digital archive system becomes inherently data-centered, and cannot cater to individual research needs.

To meet the individual needs of humanities scholars while maintaining the benefit of digital archive systems, we have developed a personal digital humanities platform called DocuSky.Footnote ⁵ DocuSky allows users to upload their own data and supplement them with additional data from certain web resource sites; it provides processing tools for the user to prepare/enrich the data (such as adding metadata or tagging) and a one-click function to build a full-text searchable database with textual context discovery. In addition to searching, retrieving, and exploring the built-in context discovery mechanisms in the database, the user can also use additional tools provided by DocuSky for analysis and visualization to explore further.

A central design philosophy behind DocuSky is the separation of content and tools. This allows the scholar (user) to have full control of the data she has in hand. To realize this methodology, we needed to design a way to represent the data and to let content and tools communicate with each other. We call this format DocuXML.Footnote ⁶ DocuXML provides an XML format of pre-defined metadata attributes as well as tags for annotation. Users can also define their own tags and metadata. Not only can content and tools be connected through DocuXML, it can also connect humanities scholars and IT researchers. A scholar can describe the tool that she needs, and as long as a tool uses a data format that is accepted by DocuSky, it can be used within the DocuSky platform freely. Another core technology in DocuSky is its ability to convert any file in DocuXML format into a searchable text database.

Building a Full-Text Searchable Database in Docusky

Scholars may amass files to the point of unmanageability on a local hard disk. A key feature of DocuSky is its ability to transform that set of documents into a full-text searchable database. Because the database building function of DocuSky requires files in DocuXML format, DocuSky provides facilities for converting several formats into DocuXML. Documents in text format, are input into the text2DocuXML tool and with just one click it generates a file with the documents in DocuXML format. When that is uploaded to the db-builder tool in DocuSky it produces a full-text searchable database with the original file names as natural delimiters separating the texts. The user can also declare separate groupings of files (such as using the folder names where the files are stored) so that each grouping becomes a different corpus and can serve as an attribute for post-classification of search results. This simple, powerful construct allows the user to create a full-text searchable personal database with a few clicks.

It is also straightforward to convert a spreadsheet (in Microsoft Excel or csv format using the the xls/csv2DocuXML tool to transform the spreadsheet into DocuXML format. In this case the user also must declare the correspondence of her own attributes with the metadata attributes in DocuXML. Then, using the same db-builder tool, she can immediately create a personal database of the spreadsheet. In addition to full-text search, such a database has an additional feature of using the metadata attributes to post-classify the query result. For instance, if “year” is a metadata attribute, the query result can be ordered chronologically. We call such relationships contexts from metadata.

Another important scholarly activity is to annotate or tag person names, place names, or any chosen set of terms. This is particularly important for a language like Chinese which lacks natural delimiters such as the word-separating whitespace in Western languages. Among the tagging tools provided in the platform, DocuSky encourages the use of Markus,Footnote ⁷ a popular tool for annotation developed at Leiden University, as the main tagging/annotation tool. Markus, in turn, draws on some of the utilities discussed in this issue, such as the code tables in the China Biographical Database (CBDB) and the China Historical GIS (CHGIS). Once documents are annotated using Markus, the resulting files can be converted into DocuXML format via the M2D conversion tool. The db-builder tool can then build a personal database from the files. The database built from texts with Markus tagging has the additional feature of contexts from tagging: relations that present the textual contexts of tags. (For instance, the frequencies of person names appearing in a query result, or the locations on a map if geographic coordinates are given in tags.)

Textual Context Discovery

These textual contexts reflect the design methodology of THDL. Most retrieval systems are based on the precision/recall model, which treats documents as independent entities. Scholars, however, usually consider documents as related and try to find the relationships (context) among documents.Footnote ⁸ Thus, instead of ranking the documents in the query result, THDL assumes the documents in the query result are related and tries to find the textual contexts (relationships). The aforementioned contexts from metadata and contexts from tagging are two important types of textual contexts incorporated in THDL. Other contexts include statistical contexts (n-gram analysis, for example) and semantic contexts. Semantic contexts are domain-dependent. In THDL we provided two such contexts: the Citation Graph of Imperial Decrees and MemorialsFootnote ⁹ and Land Transaction Relation of land deeds.Footnote ¹⁰

Except for domain-dependent semantic contexts, all the textual contexts are incorporated into any database generated from DocuSky. Another very useful statistical context is Simple Concordance Analysis, which allows the user to query a keyword with a prefix or postfix number of words, or any string that begins with a keyword and ends with another. For example, if one designates a keyword with a prefix of 2, the system will retrieve all terms with the keyword and the preceding two characters (with their frequencies).

A Working Example

In the following we show an example of a text database, generated by DocuSky, that was jointly created with Dr. Michael Stanley-BakerFootnote ¹¹ when he was working at the Max Planck Institute in Berlin. The example is Bencaojing jizhu (本草經集注), a sixth-century work by Tao Hongjing (陶弘景). The book contains 730 herbal medicines and is among the most studied Chinese medicine books. The full text of Bencaojing jizhu was downloaded from Wikisource.Footnote ¹² Stanley-Baker defined a set of tags, such as #DiseaseName, #DrugName, #Drug_Action, each with a list of associated terms. Using Markus he tagged these terms in the text; person names (using CBDBFootnote ¹³ and BSPADFootnote ¹⁴) and place names (using CHGISFootnote ¹⁵) are also tagged. The metadata are simply the original chapters and sections of the source book. The tagged file was then converted to DocuXML (with one click) and fed to db_builder (another click) to create the following database.

Figure 1 shows the resulting database, in which the left column is the classification of results from a query for the tag #Drug_Action.

Clicking on “tag cloud” shows the same post-classification in tag cloud form (Figure 2).

Figure 1.

The Bencaojingjizhu database generated by DocuSky

Figure 2.

The tag cloud feature

Clicking on the “show places on the map” button produces Figure 3, with all locations names shown on the map.

Figure 3.

The GIS feature built into the database

By clicking on a specific place name, its location will be highlighted on the map. Figure 4 shows the same interface by clicking on Lantian (藍田).

Figure 4.

Showing the location of Lantian

The following example shows the efficacy of the Simple Concordance context. It is a common practice in modern Chinese to put book titles in double brackets. For example, The Analects (論語) of Confucius will be presented as 《論語》. This convention was also used in the full-text of Wikisource. We take advantage of this property, and issue a query that asks for any string of up to 20 characters that is between《 and 》thus generating a set of all the books (with titles up to 20 characters) cited in the Bencaojing jizhu with the frequencies of appearance. Figure 5 is the image of the query, and Figure 6 shows the resulting set.

Figure 5.

The Simple Concordance query box

Figure 6.

Outcome of the Simple Concordance query

The complete list of books cited in Bencaojing jizhu with frequencies of occurrence is given in Figure 7.

Figure 7.

Complete list of books cited in Bencaojingjizhu

Utilizing External Digital Sources in Docusky

There are many high quality, free Sinological resources on the Web. In addition to creating their own content, scholars can also enrich their data by extracting material from Web resources through DocuSky. The interaction between an external source and DocuSky is performed through a Web API (Application Programming Interface). For each resource we need to build a separate API. The following are available at the moment:

1. 1. Ctext.org (https://www.ctext.org): Download plain text through API.

2. 2. Kanripo (https://www.kanripo.org): Download plain text with simple metadata through API

3. 3. CBETA (https://www.cbeta.org): Download files of texts from CBETA through API. The files will automatically be converted into DocuXML formatFootnote ¹⁶.

4. 4. THDL (http://thdl.ntu.edu.tw): Files in THDL can be exported directly into DocuSky in DocuXML format through API.

We are also preparing an API to extract files directly from Wikisource while preserving their original structure.

Once data are prepared and the database built, a number of tools can be utilized to make further analysis and visualization. Below we describe several useful tools.

Docugis, the GIS Tool in Docusky

Map visualizations using a GIS (geographic information system) is useful but hard to master for many. We have developed a GIS tool in DocuSky called DocuGIS.Footnote ¹⁷ DocuGIS has several unique features. First, given a spreadsheet of locations with coordinates, it can plot the points on the map with just one-click. In other words, anyone who knows how to prepare a spreadsheet can draw a GIS map in a few minutes. Second, it is interactive. If the spreadsheet contains a column of text, then DocuGIS allows full-text search and highlighting the results on the map. Third, DocuGIS has also collected a number of historical maps so that results can be shown on historical maps.

In the following we use the example of Xuanzang's (玄奘) seventh-century Pilgrimage to the West during the Tang Dynasty (大唐西域記). We first obtain the full-text of the book from ctext.org. Since the book was written in the order of cities (countries) that Xuanzang visited, we divide the text into segments accordingly. We then create an Excel spreadsheet, with each of the rows containing a numerical id (in the order of appearance in the book), CompilationName (section header in the book), PlaceName (name of place in the book), Coordinates (y, x), Text (the text of the segment), as shown in Figure 8.

Figure 8.

spreadsheet with content from Pilgrimage to the West during Tang Dynasty

The data in the Excel sheet can be easily converted into csv format,Footnote ¹⁸ which is then fed into DocuGIS (Figure 9) though the “upload csv” option.

Figure 9.

Uploading csv to DocuGIS

This action immediately creates a map with all locations in the book highlighted. Clicking on any of the points shows the corresponding text in that entry (Figure 10).

Figure 10.

Resulting interactive GIS system

Figure 11 showcases three additional features. First, we added a map of Tang administrative divisions.Footnote ¹⁹ Second, by using the id to specify the sequence of points, the locations become a sequence which indicate the route of Xuanzang. (This can be easily done with a couple of clicks in DocuGIS.) Third, the balloons in the same figure also show the query result with keyword Brahman (婆羅門).

Figure 11.

GIS with Tang Map, route plotted, and query issued

The ease with which DocuGIS creates a map from full-text, and the ability to query in a GIS environment, make it unique among GIS tools. DocuGIS can be used in conjunction with DocuSky or independently, without first building a database.

Tools for Term Analysis

DocuSky also provides a number of text analysis tools, including word clipping for discovering new terms,Footnote ²⁰ n-gram analysis, term frequency analysis, text style analysis,Footnote ²¹ and relevant document discovery.Footnote ²² In the following we show an example using Hsieh's TermStat analysis tool. Given a group of texts and a list of terms, TermStat returns the frequencies of terms appearing in each text. Our example wants to see what types of disease were mentioned in the Standard Histories of the Northern and Southern Dynasties (420CE~589CE). There are nine Standard Histories of that era, four for each of the dynasties in the south, three for dynasties in the north, and the two books Southern History and Northern History. We first obtain the texts of the nine histories from ctext.org through its ctext API. After producing, within minutes, the full-text database with nine documents, one for each Standard History, we export the corpus to the TermStat tool together with a list of diseases.Footnote ²³ The output from TermStat (nine lists of disease names with their frequencies) is then exported in csv form and uploaded to Palladio,Footnote ²⁴ a visualization tool developed at Stanford University. The result is shown in Figure 12. An edge in Figure 12 connects a disease name and the history in which it appears. It is therefore very easy to see which diseases are shared by which histories. Observe that the five books of the Southern Dynasties (left hand side) and the four of the Northern Dynasties are clearly clustered separately.

Figure 12.

Graph shows relationships between diseases and Standard Histories

With this DocuSky-TermStat-Palladio scenario one can conduct term analysis and visualize the outcome. Stanley-Baker's work on ancient Chinese medical texts is a particularly good example.Footnote ²⁵

Connecting Humanities Scholars and it Researchers Via Docuxml

An important goal of DocuSky is to enable digital humanities research without the direct help of an IT expert. On the other hand IT developers need to know what scholars are trying to do if they are to build suitable tools. We need a way to connect scholars and IT researchers so that the technical needs of the scholar, present and future, can be advertised and interested IT developers can heed the call. Tool building should not be limited to the DocuSky development team. Tools may be applied to different contents or the same content may make use of different tools. One of our principles is also to utilize existing tools as much as possible. Markus and Palladio are two such examples. They were developed at Leiden and Stanford Universities respectively, but they have been incorporated into the process flow of DocuSky and play important roles.

Thus we need to establish a standard for communicating between content and tools. In XML format we call this standard DocuXML.Footnote ²⁶ DocuXML defines two classes of objects, corpus and document, where a corpus may contain several documents. DocuXML defines attributes of Corpus Metadata and tags for annotating documents. Users can also define their own tags. Any file in DocuXML format can be easily turned into a database through one click using db_builder in DocuSky. In addition to full-text search, metadata contexts and tagging contexts are realized through the built-in post-query classification features. In order to utilize an existing annotation tool that uses another format, a converter may be necessary to convert that format into DocuXML.Footnote ²⁷ Currently we have written converters for plain text, Excel, Markus output in html, and csv. The same is true when a file in DocuXML format uses other tools for analysis and visualization. For example, in order to use Palladio to observe the co-occurrence relation of terms in several documents, we implemented the TermStat tool to generate the document/term tuples needed in Palladio and to convert the result into csv, the input format that Palladio accepts. Similarly, to utilize Web resources such as ctext.org or CBETA, we need to create an API for each of the resources so that their content can be downloaded and converted into DocuXML format. The successful implementation of CBETA API and the ensuing conversion of the downloaded files from TEI (the standard used by CBETA) to DocuXML, done by the DILA team led by Jenjou Hong, testifies to the simplicity and functionality of DocuXML.

If an IT researcher wishes to implement a tool that uses content through DocuSky, she needs to use the DocuSky API to access content. To simplify this work, we have developed a suite of widgets (DocuWidgets). Instead of understanding the communication details between DocuSky API and DocuSky, the implementer only needs to know how to use DocuWidgets to access data in a personal database generated by DocuSky. This way the implementer can spend more time on developing the tool, instead of understanding the inner workings of DocuSky.Footnote ²⁸

While DocuXML is still evolving, the experiment with using DocuXML as the link to turn a scholar's needs into working tools has been satisfactory so far. Except for the overall architectural design and the implementation of the core such as db_builder and DocuWidgets and some converters which are built by full-time staff, most of the tools, including the entire DocuGIS, were built by students and external IT researchers.

Final Remarks and Future Work

Figure 13 illustrates a simple research cycle of historical research. Given a research objective, the scholar collects relevant research data, processes them, organizes them, analyzes and sometimes visualizes the outcome, then interprets the findings. Interpretation sometimes leads to new research objectives, in which case the research cycle restarts. Eventually the cycle closes and hopefully results in publication of the findings.

Figure 13.

Naïve research cycle

In the age of digital archives and digital humanities, the data development process has emphasized building large-scale digital archive systems equipped with tools for analysis and visualization. Such systems contribute to the data collection, data process, data access (by building retrieval database systems), data analysis, and visualization parts of the research process (see Figure 14). Although some of these systems can be very sophisticated, they are usually built around specific data collections and cannot deal with the varying needs of individual scholars.

Figure 14.

research cycle with DH archives

DocuSky advocates the separation of content and tools. In this sense, it is designed as a platform, not a system. That is, it allows content and tools both to be freely added by anyone. DocuSky maintains the standard format DocuXML through which content and tools can interact. It also provides the core database building facility which transforms any file in DocuXML form into a full-text searchable database. Another important aspect of DocuSky is that it does not build its own tools and contents unless necessary. In other words, it facilitates open contents over the Web and open source tools. Figure 15 shows how we envision the way DocuSky fits into the research cycle.

Figure 15.

The research cycle with DocuSky

Note that we see a division of work between scholars and DocuSky. Scholars define their research objectives, collect research material that is unique to their work, and interpret what emerged from their analysis. DocuSky, on the other hand, provides tools for both such nitty-gritty work as downloading related material from the Web and processing and enriching the data, and for such technically demanding work as building a searchable database from processed data and creating analysis and visualization of the discoveries.

Since DocuSky is designed as an open platform, it is ever-evolving in scale and functionality. Although there are already more than 1,000 users of DocuSky, and it has been used in scores of projects, we welcome more. Moreover, the interface to enable a smooth exchange between scholar and IT developer—allowing the former to articulate her needs and the latter to respond to the request—has not been developed fully. Both human and technical aspects need to be carefully considered in order to tackle this problem properly. Better ways to receive and implement user feedback are also needed.

Although DocuSky currently works mainly on Chinese-language material, the principle of separating content and tools and the need to have a personal DH platform to empower scholars are certainly universal. We plan to put more language supports into DocuSky.

The contents that people have tried with DocuSky range from Shang oracle inscriptions to Taiwanese primary school textbooks. As new genres are tried and new requests for tools emerge constantly, we need more effective ways to engage IT researchers and to incorporate existing tools.