- 1 The ODD files currently available can be downloaded from GitHub: https://github.com/CoPhi. The TEI (...)
- 2 Facelets XML templates are used under the Java Server Faces 2.0 specification.
1The TeiCoPhiLib library is a collection of components currently implemented in Java (JSR 270), which parses documents encoded according to a basic subset of TEI tags defined in an ODD file1 and creates an object-oriented data structure in order to handle the textual content and its processing. The overall architecture is based on the well known Model-View-Controller (MVC) pattern, which separates the representation of data from the rendering and management of the content for the sake of flexibility and reusability. TeiCoPhiLib maps the structured document onto an aggregation of objects. The library enables the visualization through a web browser by instantiating a collection of widgets rendered on the client through standard web technologies. Specifically, the server-side environment jointly processes data and visualization templates,2 and generates HTML pages rendered on the client. Special components are devoted to monitoring the behavior and interactions among the objects generated from the input TEI documents.
2In distributed and collaborative environments, the maintenance of links and relations among editable XML documents is a challenging task (Di Iorio et al. 2009; Peroni, Poggi, and Vitali 2013; Schmidt and Colomb 2009). Indeed, this kind of environment must preserve the referential integrity among interrelated textual units and the consistency among interlinked contents (Schmidt 2014; Barabucci et al. 2013; Ronnau, Philipp, and Borgho 2009). It is worth noting that both texts and related annotations may change asynchronously. Accordingly, the problems with maintenance have an increasing relevance in social editing (Siemens et al. 2012) and in general in collaborative philology. This emerging field concerns the social activity of scholars focused on shared philological tasks (such as scholarly editing and collaborative annotation) through a cyberinfrastructure (Terras and Crane 2010; Crane, Seales, and Terras 2009). In order to face this challenge, our approach exploits software engineering techniques illustrated in section 3.3, which explains the TeiCoPhiLib design patterns.
3For this reason, the design of TeiCoPhiLib widely leverages the stand-off approaches provided by the TEI Guidelines, that is, both the reference to plain text offsets and the reference to nodes denoted by the @xml:id unique identifiers (TEI Consortium 2015; Wittern, Ciula, and Conal 2009; Pierazzo 2015). Consequently, the design of the components aims at the separation of concerns through four distinct layers: (1) textual structure; (2) semantics; (3) style; and (4) behavior, in order to ensure modularity, scalability, and flexibility.
4The main benefit of XML, and especially of the TEI Guidelines (TEI Consortium 2015), resides in simplicity, flexibility, readability, and customizability, with the assurance of a formal approach for validating the marked-up data. Consequently, XML provides a standard way to define a set of tags (vocabulary) for specific purposes. Moreover, the cluster of technologies associated with XML3 allows us to process, query and publish structured documents.
5Several frameworks and initiatives have been developed over the years for handling XML, achieving great results and benefits for both scholars and developers. Among others, the open-source general-purpose framework Cocoon4 and the native XML database eXist-db5 deserve to be mentioned. Specifically for TEI-annotated documents, TUSTEP,6 TEIBoilerplate,7 TXM,8 and TAPAS9 are prominent projects.
6For all of these initiatives, the transformation from an XML document structure to another format by XSLT can be considered the focal point.
7A document-oriented approach can be complemented by an application/API-oriented approach for the development of textual analysis tools. We are adopting a top-down design integrated with bottom-up processes (Del Grosso and Boschetti 2013), which allows us to generalize, extend, and refactor the overall architecture as new requirements and common issues emerge from use cases under development. The design is top-down because we are defining both the general abstract framework and the mechanisms that allow us to implement new functionalities according to emerging needs. On the other hand, our library also adopts a bottom-up approach because we apply refactoring strategies to adapt existing components implemented in our previous projects to the general framework, extending the framework.
8The library of components is designed by exploiting object-oriented methods and processes such as analysis of requirements, definition of the domain entities, separation of concerns, information hiding, and software reusability and extensibility (Fowler 1996). Extensive use of design patterns (i.e., recurring solutions to common problems within a given context [Gamma et al. 1995]) facilitates the achievement of these goals.
9Agile software development10 and use case–driven modeling (Rosenberg and Stephens 2007) ensure the progressive enhancement of old functionalities and the development of new ones. The main principles of agile software development that we adopt are: (1) individuals and communication are more important than processes and tools; (2) documentation and design must be accessible to everybody all the time; (3) software development starts as soon as possible; (4) changes and refactoring are part of the design and the development process; (5) all lab team members participate in all presentations; (6) software is organized in short releases and divided into short iterations; (7) results are validated by domain expert collaborations and test-driven development (both unit tests and acceptance tests). The continuous integration and release are supported by open source Integrated Development Environments (IDEs) like Eclipse or NetBeans and by a software configuration management tool such as SVN or Git for versioning and revision control.
10The aforementioned paradigm is applied in the TeiCoPhiLib library by (1) the implementation of a flexible importing and normalization module in the pre-processing phase, which ensures a coherent abstraction model of the resources; (2) the definition of the functional specification by designing the objects and by declaring the application interfaces; (3) the export of the information encapsulated in the objects into different data formats, in order to enable data integration and data exchange.
11First of all, objects that represent the whole document or interrelated documents are initialized by parsing the original TEI document(s) and by creating a new data structure, which decouples the orthogonal information conveyed by the XML elements: (1) textual structure, (2) semantics, (3) style, and (4) behavior. It is important to point out that the new data structure is the result of transformations (by XSLT DOM transformations or SAX event-driven transformations) managed during the parsing process. Thus, the current implementation of the TeiCoPhiLib exposes methods that parse the XML file and create Java objects. The resources are stored and maintained in a native XML database management system (i.e., eXist-db). The APIs and services provided by Lucene, a software library developed and hosted by the Apache Foundation, have been used for indexing the textual data.
12For instance, the information conveyed by the following TEI snippet is distributed among the appropriate Java objects that handle the four levels described above:
<div type="chapter" n="1" style="font-variant:normal">
<lb n="1"/>Io nacqui veneziano ai
18 ottobre del 1775, giorno
<lb n="2"/>dell’evangelista san Luca;
e morrò per la grazia di Dio
<lb n="3"/>italiano quando lo vorrà
quella Provvidenza che governa
<lb n="4"/>misteriosamente il mondo.</p>
[...]<milestone type="page" n="2"/>[...]
13The parsing process concerns the following aspects:
Textual structure. The same document originally structured paragraph by paragraph for literary analysis can easily restructured page by page for layout analysis and for comparison with the original page image. Semantics, style, and behavior are represented by objects separated from (but linked to) the nodes of the DOM tree.
Semantics. At the semantic level, both attributes (such as @type) and tag names (such as <p>) are processed in the same way and linked to the related DOM node.
Style. The style is managed by separated renderers, which point to textual positions affected by stylistic features. For instance, the information extracted from the @style attribute is used to instantiate the Java objects devoted to managing the rendering information.
Behavior. Behaviors are handled by objects that process textual resources according to the current state of the data structure and the rules to manage such a state. For example, a hyphenator performs its tasks according to the language of the textual data (encoded in the original TEI file, e.g., by @xml:lang="it") and the related hyphenation rules (such as the hyphenation rules for the Italian language, managed by the hyphenator bundles).
14The object-oriented representation of the document allows data to be processed dynamically, taking account of its physical and logical structure, in an attempt to overcome the multiple hierarchies issue. This means that the data model of the library has a decoupled and abstract structure which can be serialized in any available file format, including all standard TEI-compliant approaches. Consequently, the TeiCoPhiLib document entity keeps structure and logical information in an independent but related aggregate of objects (figure 3). Furthermore, output modules or visitors (figure 2) can traverse and serialize the object representation of the document to a file. In particular, modules for TEI files define the marshalling process, which encompasses the operation to obtain the encoded XML files. In this way the system provides the actual representation of the document.
15The overall architecture of the library is based on several design patterns, according to the object-oriented paradigm (Gamma et al. 1995; Buschmann, Henney, and Schmidt 2007).
16Design patterns were introduced in software engineering in order to provide a common solution for a recurring problem in a specific context. A design pattern defines the instantiation policies, the structure, or the behavior for an aggregate of objects that cooperate to provide a complex but recurrent functionality, such as the creation of polymorphic entities, the management of decoupled modules, or the selection at run time of the most suitable algorithm for the current task. The general idea of object-oriented patterns is to encapsulate functionality and data inside an efficient and flexible collection of classes. The current implementation of the prototype exploits the Java programming language technologies.
The Model-View-Controller (MVC) pattern (Burbeck 1992) determines the architecture of the library by separating the internal representation of the data from the rendering and the behavioral purposes.
The Factory pattern allows designers to enhance the object creation procedure by means of special classes (i.e., the factory classes) that guarantee abstract coupling among system modules (i.e., the object relationships do not reference implementing classes). As a matter of fact, the foregoing design makes it possible to programmatically reference abstract objects independently from the run-time instances which actually perform the task. This design pattern lets applications change the implementation of a class in a flexible way. In our case, for instance, a document object has textual content maintained in a specific DOM data structure transparent to the user. The client agent is able to manipulate and process the document independently of its internal DOM representation. The algorithms and processing keep the state of the data structure coherent by updating the DOM representation in a transparent way.
The Builder pattern is used to initialize and populate the document data structure. Together with the factory pattern, it hides the real type of the objects from the user agents and maintains the state consistency of the interconnected information. In this way, in the library initialization process, it is possible to create different data structures for different aims, in a way that is completely transparent to the user agents. For instance, a Builder oriented to the layout analysis can restructure the information parsed from the TEI input document.
The Composite pattern is the core of the data structure (figure 3). The document object is defined as an aggregation of hierarchical entities with the same data type. The hierarchy maps either the DOM structure of the original XML-TEI document or the structure of one of its transformations based on an XSLT input parameter. Thanks to this pattern, an efficient object-oriented structure, sketched through the UML class diagram on the right of figure 3, represents the whole/part relationships among the objects in the data structures.
The Strategy pattern implements different operations in different ways based on the object type or on specified parameters. For example, the building process uses different strategies, which are driven from specific features given through property files. In the previous example, the original TEI page milestone is represented by an element node in the DOM internal structure; conversely, the original TEI element for paragraph can be represented by a milestone. Furthermore, the Strategy pattern is useful for rendering the same data in multiple views in different contexts or processing the same data with different algorithms.
The Observer pattern provides a mechanism for handling dependencies among interrelated objects. This ensures that when a change occurs, the overall state is synchronized and updated. For example, if an edit operation deletes some values in the document, all related structures are notified and updated accordingly. The library organizes the entities derived from the original document information through the stand-off approach. In this way the document structure is separate from its semantics, style, and behavior.
Figure 1: Class diagram of the Observer pattern designed for the TeiCoPhiLib
Several modules of the library need synchronized data. In particular, annotations and comments need to be notified if an event occurs that changes the state of the referenced object. This problem is well known in the context of graphical user interfaces, where embodied components communicate by message exchange according to a standard protocol. The Observer pattern, which is a behavioral technique, solves the aforementioned problem in an elegant and easy way. TeiCoPhiLib exploits the Observer pattern to manage communication among the object-oriented representation of the document and the encapsulated data that point to it. Figure 1 shows the UML class diagram of the mechanism designed for the library. The pattern involves two interfaces: (6a) Subject and (6b) Observer. These two entities provide the flexibility to implement a decoupled notification mechanism. The Subject provides a registration procedure for the Observer object, and the Observer object provides a standard method allowing the Subject to notify it. TeiCoPhiLib defines objects that can change, such as the Document data type, and objects that need to be notified, such as the Annotation or the Comment data types. Consequently, the Document class implements the Subject interface, whereas the Annotation and Comment classes implement the Observer interface. The following simplified Java snippet illustrates this concept programmatically.
Observer annotation = new Annotation();
Observer comment = new Comment();
Subject teiDocument = new Document();
// some processing that modifies the teiDocument object
// accordingly the observers have to be notified
The pattern Visitor allows the user agents to gather data which are stored in different object data fields and enables the reconstruction of a consistent and homogeneous view. The stand-off mechanism implies a distributed and interrelated structure where information is maintained across various objects. For example, during the exporting phase the object-oriented representation of the document is processed in order to produce a valid TEI XML document according to a schema selected by the user.
Figure 2: Class diagram of the Visitor pattern designed for the TeiCoPhiLib
The hierarchical nature of the document representation facilitates the data structure traversal in a flexible and customizable way. The Visitor pattern provides a mechanism to extend the functionality of the TeiCoPhiLib by allowing components to perform a client-supplied operation on each node of the document hierarchy. Figure 2 shows how a client of the data model can traverse the document tree in order to write its textual content. The mechanism supplies the actual visitor instance by means of a custom operation. The current implementation of the Visitor interface is then used by the entities of the hierarchy which are unaware of the actual behavior of the visitor instance. In this way, the Visitor pattern provides a procedure to operate on the representation of the document by making available to the clients a point of extensibility (Martin 2000).
17An example should clarify the aforementioned architecture. The client application that uses TeiCoPhiLib APIs invokes the building method of the abstract Builder class. Moreover, the resulting document object is a concretization of an abstract class representing the current structure of the TEI-encoded resource, as illustrated in the following Java statement:
Document teiDocument = AbstractBuilderFactory.buildDocument(new File("features.properties"),new File("teiDocument.xml"));
18The Builder object needs two input files: (1) a property file containing the suitable configuration for the instantiation of the concrete objects; (2) the TEI XML file to parse. The state of the internal representation of the document is composite-component–based. In this way each single element is handled by the TEIComposite object shown on the right-hand side of figure 1, which represents each node of the hierarchical DOM structure. Leaves are instances of the StrippedTextChunk class (figure 3). Methods in such a structure make it possible to manipulate the content and the structure of the resources.
Figure 3: Parsed TEI document (left), mapped into a Composite pattern structure (right)
19Finally, the data structure is an object-oriented representation of the entities in the real domain of the digital document, but the storage platform/paradigm could actually be relational, hierarchical, semi-structural (XML), or network/graph–structured (Hohpe and Woolf 2004).
20The UML diagram on the left side of figure 1 shows the structure of a TEI document DOM representation. Objects, in this case, exactly map the tags of the TEI-encoded file format. The UML diagram on the right side illustrates the composite-components design. This diagram maps the same aforementioned information and structure, but using a flexible and recursive design. Each TEIComposite object reflects a TEI element and each child is recursively a TEIComposite or a textual element.
21The case studies illustrated below have been implemented with the components already developed for our library.
22Euporia is a project aimed at visualizing, editing, and annotating bilingual texts displayed in parallel. The original digital resources are stored and maintained in authoritative digital libraries available online, such as the Biblioteca Italiana and the Perseus Digital Library, or they are downloaded from social proofreading websites, such as WikiSource, and subsequently processed and marked up in TEI. Some examples of Greek and Latin texts potentially alignable or actually aligned with their Italian translations are shown in table 1.
23As mentioned in section 3, different subcollections of texts that must be aligned may provide or omit some extratextual information (such as line number or page number) and they may organize texts in different ways (for instance, lines can be grouped or not inside <lg> elements). For this reason, the XSD schema (which is expected to be a subset of the general TEI schemas) is generated a posteriori from the actual text subcollections. This approach can be considered complementary to the TEI Roma approach, a kind of reverse engineering, which also allows us to generate the ODD file. Studying the schemas, XSLT transformations are created in order to deal only with relevant information and canonical formats processed by the appropriate Aligner. Currently only the SpeechAligner for dramatic texts has been implemented: correspondences between the Italian translation and the original Greek text are automatically injected with the @corresp attribute (see row B in table 1) and misalignments must be manually corrected.
Table 1: Parallel texts
|A) Perseus Digital Library
||Digital Library of Biblioteca Italiana |
<!-- www.perseus.tufts.edu -->
<div1 type="Book" n="2" org="uniform" sample="complete">
<milestone ed="p" n="1" unit="card"/>
<l>Conticuere omnes, intentique ora tenebant.</l>
<l>Inde toro pater Aeneas sic orsus ab alto:</l>
<l>Infandum, regina, iubes renovare dolorem,</l>
<l>Troianas ut opes et lamentabile regnum</l>
<l n="5">eruerint Danai; quaeque ipse miserrima vidi,</l>
<l>et quorum pars magna fui. Quis talia fando</l>
<l>Myrmidonum Dolopumve aut duri miles Ulixi</l>
<l>temperet a lacrimis? Et iam nox umida caelo</l>
<l>praecipitat, suadentque cadentia sidera somnos.</l>
<l n="10">Sed si tantus amor casus cognoscere nostros</l>
<l>et breviter Troiae supremum audire laborem,</l>
<l>quamquam animus meminisse horret, luctuque refugit,</l>
<l>incipiam.<milestone ed="P" unit="para"/>Fracti bello fatisque repulsi</l> [...]
<!-- www.bibliotecaitaliana.it -->
<div1 sample="complete"> [...] <lg sample="complete">
<l>Tutti ammutiro e s'affisaro. Enea</l>
<l>Da l'alto seggio incominciava. Infando,</l>
<l>O Regina, è il dolor cui tu m'imponi</l>
<l>Ch'io rinnovelli. I' dovrò dir da' Greci</l>
<l>I teucri averi e il miserando regno</l>
<l>Come fosser diserti; io dire i casi</l>
<l>Tristissimi dovrò, cui vidi io stesso</l>
<l>E di che fui gran parte. E qual potrebbe</l>
<l>O Mirmidone o Dolope o seguace</l>
<l>Del fero Ulisse rattenere il pianto</l>
<l>Tai cose in ragionando? E già dal cielo</l>
<l>Precipita la notte umida, e gli astri</l>
<l>Vanno in cader persuadendo il sonno.</l>
<l>Ma se tanto desio nel cor ti siede</l>
<l>De l'ascoltar de' nostri casi e in breve</l>
<l>Udir di Troia l'ultima sciaura,</l>
<l>Benchè pur del pensiero io mi spaventi,</l>
<l>Comincerò. Dopo tant'anni infine</l> [...] </lg> [...] </div1>
|B) Perseus Digital Library
||Euporia Resources |
<lb/>τί δ, ὦ ταλαίπωρ’ — ὅστις ὤν μ’ ἀπεστράφης
<lb/>καὶ ταῖς ἐκείνης συμφοραῖς ἐμὲ στυγεῖς;
<sp id="sp_ita_0002" corresp="#sp_grc_0002">
<lb/>Perché, qual che tu sia, misero, gli occhi
<lb/>torci da me, pei falli altrui m'aborri?
24Parallel texts are visualized and managed through EuporiaWebApp (figure 4), which is a server-side Java web application compliant with the JSR 314 specification intended for educational purposes. Students, the end users of Euporia, are allowed to query texts, both jointly and independently, through multilingual or monolingual keywords.
Figure 4: Euporia
25Annotations can also be associated with linked chunks of text (such as a sentence and its translation: see figure 5) or with single, independent chunks of text (such as a single word of the original text). Annotations and their pointers to the text are stored as stand-off markup. Currently, the adopted stand-off linking mechanism uses only @xml:id, but the use of XPointer is under evaluation.
Figure 5: Annotation in Euporia
26Aporia is the enhanced version of Euporia, intended for research purposes. Accordingly, the Parallel Text framework has been adapted and extended to meet specific scientific requirements (Bozzi 2013). An experimental case study has been performed on Theodor Mommsen’s edition of the Res Gestae Divi Augusti (1883), in order to examine the complementarity of attested fragments of text in the Latin inscription of the RGDA and the related attested fragments in the Greek translation (Lamé 2012). Attested and conjectural parts from Mommsen’s critical edition have been marked. As shown in figure 6, the feature “status” (attested / partially attested / conjectural) is not only visualized in different colors (a CSS stylesheet is enough for this task), but also available in query masks to filter the results of a query (for instance, “find only attested or partially attested words”) and in tables of results.
27Unlimited stand-off layers of analysis can be added (such as morphological, syntactic, and semantic analysis), at different levels of granularity (for example, at the level of words for morphological analysis and at the level of sentences for semantic analysis). The layers of annotation are added through the web application and stored in the XML database. Moreover, they can be manually encoded and visualized through the web application.
Figure 6: Aporia
28The Saussure Project exploits the flexibility of Aporia in order to adapt the system to the study of Saussurean autographs, making author’s variants searchable and creating multilingual indexes of ancient terms studied by the linguist.
29Instead of showing linked texts in parallel, the system shows the image of the manuscript and the related transcription (figure 7).
Figure 7: Saussure Project
30The TeiCoPhiLib is a work in progress focused on the creation of a library of software components aimed at managing a limited subset of TEI tags used in the domain of collaborative philology. Because of the increasing complexity of annotations and the multiple usages of the same texts in collaborative environments, stand-off annotation and dense mark-up make it challenging to keep annotated documents readable and manageable. While annotation focuses on types of texts (such as poetic, dramatic, and with or without critical apparatus), software development focuses on abstraction of data structures and behaviors related to those texts (such as searching them in parallel, filtering by morphological features, and comparing text and image).
31Reusable software components promote the management of stand-off annotation at any level (such as editing, searching, or visualizing), improving the experience of the annotation and use of TEI documents.
32The document parsing in the current Java implementation takes place on the server side, where the Java virtual machine runs within the web application environment.
33The marshalling and unmarshalling process handles the serialization of the object representation of the TEI document, in order to store and retrieve data on the filesystem or in native XML databases, such as eXist-db.
34Performance measurement tools such as JMeter will help to optimize the performance of the library components.
35Software currently under development will be available on GitHub at https://github.com/CoPhi/cophilib.