- 1 Envisioned as an extension of our existing XML codebase, such editions are already planned for our (...)
1One of the major challenges facing TEI encoders of older documents (ancient and medieval manuscripts, early print, manuscripts transcribed in modern print editions) is the range of special characters and abbreviations that they contain. This issue is especially critical for online documentary editions, where the goal is to present the digital facsimile of a manuscript alongside its electronic transcription. Many of the characters in the facsimile will have no exact correlatives even in Unicode, let alone in standard fonts (e.g. Times New Roman, Garamond, Palatino). In response, scholars have fallen back on deeply problematic strategies, such as substituting images for glyphs or using rough, modern equivalents that are usually historically and/or linguistically inaccurate. A "normalized" transcription, which translates all such characters to some modern equivalent, does have its uses—for example, as a reader edition and/or as an accessible edition for the visually impaired (both of which would otherwise be complicated by medieval abbreviation)1—and should, therefore, be part of any scholarly edition presented on the Web. However, such a transcription is far less useful to researchers interested in examining special characters as part of the dialectical and paleographical studies important to early periods; students and non-students alike also deserve a full documentary transcription that preserves the historical character of the original manuscript. And, quite simply, for those of us who work on texts surviving in unique copies, the ideal of reproducing in XML a text with all of its artefactual manifestations (so that the encoded description could substitute for the artefact if the latter is lost) is particularly resonant.
2The stated goal of TEI—to develop and maintain "a standard for the representation of texts in digital form"—sets up a double challenge: machine-readable code on one side; its visual display on the other. Both forms of representation are crucial to translating and preserving texts in digital form. Nonetheless, our experience with TEI has shown a dramatic emphasis on the former part of the challenge—developing code that works seamlessly for XML—and surprisingly little support for the latter—developing code that guarantees accurate displays of complex characters, displays that users can identify easily with equivalent glyphs in the facsimile.2 Reinforcing this imbalanced emphasis is the TEI P5 standard for representation of non-standard characters and glyphs, which sets up a complex methodology for representing non-Unicode characters that is largely unconcerned—and at odds—with the practical issue of how these characters might actually be displayed (TEI Consortium 2012, chap. 5).3 Still, as more and more editions of texts move to the Web the relationship between XML and HTML, between coded representation and visual display, has become increasingly important. In this respect the TEI Guidelines are not ready to serve documentary sites trying to go live now, and to provide accurate display for transcriptions of their source manuscripts. This problem, crucial for scholars of texts from pre-modern and less-dominant language groups, we view as an ongoing challenge deserving wider attention among TEI members. In the interim, we offer some immediate technical solutions using Unicode Private Use Area (PUA) characters. We derive our PUA characters from the Medieval Unicode Font Initiative (MUFI), but our strategies for displaying these characters remain interchangeable with similar Unicode recommendations or character sets across disciplines and thus, we believe, offer hope to a broad range of projects struggling with issues of display left unresolved by Unicode and TEI.
3Those of us working with ancient and medieval texts face a set of Unicode charts where most of the punctuation, abbreviation marks, and other characters typical in manuscript production are missing or sequestered in PUAs ignored by virtually all fonts. Our Web-based project currently features a complete digital facsimile of the sole manuscript for The Book of Margery Kempe. As we refine our planned parallel diplomatic edition, one of our areas of focus has been developing strategies not only to encode special characters in compliance with the TEI Guidelines and best practice, but also to ensure that those characters display properly. We will not accept a single blank box in our electronic transcriptions or force visitors to our site to download and install special fonts before they can access these transcriptions fully.
4Part of the imbalance in TEI between representation and display solutions for special characters may be a consequence of print bias: many scholarly teams working on texts from the print era have not faced substantial bodies of characters that are non-standard for modern fonts. Consequently, these scholars may not mind the occasional blank box popping up in their displays of online text, particularly if the blank box (which frequently indicates the user's browser's failure to find a character among those fonts installed on his/her computer) can be resolved by the user's installation of a font downloadable from a project's Web site. As print artefacts massively outnumber surviving works in manuscript form, scholars working with print-era texts dominate the humanities and, most likely, membership in TEI.4 Thus TEI members from this latter group may see text to be coded (especially when the characters that they use have Unicode counterparts) as having a more direct relationship between initial representation in XML and ultimate display in HTML.5 Of course, one intent of TEI is to facilitate (blind) interchange and interoperability of humanities data with support for a wide variety of use cases, not only HTML Web pages. The foremost challenge in this respect is generating metadata for systems such as COinS or RDF that can be used to render large bodies of data across multiple projects into interoperable and searchable assets: text strings in these metadata formats cannot tolerate markup outside Unicode, so standardization for these systems is essential. Consequently, newly-recognized special characters cannot simply remain in PUAs indefinitely, but must be put into the Unicode pipeline for review, consideration, and eventual acceptance if texts using what are now non-standard characters are to be included in larger databases.
- 6 On these many punctuation characters and their uses in medieval England, see Parkes 1992.
5Nonetheless, scholars working on ancient and medieval manuscripts face two more immediate problems. First, though many non-alphabetic characters are included in Unicode, these characters will display only in combination with fonts that include a glyph for the appropriate codepoint—and such support is far from guaranteed. Among users worldwide, choices of software vary widely, and even different versions of the same operating system, office suite, or Web browser can affect which fonts are available to the user and how (if at all) these fonts will render a character. Second, vast numbers of special characters are not included in Unicode nor have they been proposed for inclusion in Unicode. Among medievalists, for instance, one standard resource for interpreting frequent abbreviations by scribes, Adriano Capelli's Dizionario di abbreviature latine ed italiane (1996), includes thousands of examples of such abbreviations. Some of these abbreviations can be recreated with superscripted letters or combining diacritical marks, but many cannot, and those that can often play fast and loose with the semantics of the characters drafted. Editors often simply present their expanded versions of the abbreviations without the original scribal elements, but this strategy introduces a large mediating disjunction from the visual object. A similar problem exists with medieval punctuation, which uses both characters and pauses quite differently from their modern counterparts and can only be roughly reconstructed with Unicode characters.6 Critical editions in traditional book form, restricted by the limits of print technology, regularly normalize on both counts by introducing expansions and modern punctuation equivalents; yet falling back on these old accommodations abandons one of the most important features of Web editions: visuality.
- 7 One example of this trend is the Canterbury Tales Project. This ambitious project, overseen by Pete (...)
- 8 Kevin Kiernan (2006) all but admits this, but does not explore the issue of display beyond a stated (...)
6As Kathryn Sutherland has observed, the "only aspect of the book-bound text that the computer appears to simulate with any high degree of success is the visual" (Sutherland 2009, 20). Digital facsimiles are an inevitable and highly attractive consequence of scholarly editions moving to the Web. One of the principal forms in which medieval editions are migrating to the Web is diplomatic editions of single manuscripts, in the digital form now called "documentary editions" (Pierazzo 2011). In large part this trend is practical: the edition itself is straightforward transcription rather than complex creation of a critical edition out of many texts. Rather than force us to depend entirely on the old print mechanism of the scholarly apparatus when transcribing works that survive in multiple witnesses, the Web enables us to reproduce every witness. But this opportunity is not without its challenges: in Chaucer's case, we must consider eighty-four separate manuscript witnesses (pre-1500) for the Canterbury Tales. As a result a number of early projects for digital critical editions have either stalled or moved to Web-based facsimiles and documentary editions for individual manuscripts.7 Hence the problem of display has become an immediate concern.8
- 9 The site, http://english.selu.edu/kempe/, currently has a full working facsimile and some text.
- 10 The Book of Margery Kempe has become a major text in medieval studies, now second only to Chaucer i (...)
7In our case, we are encoding for a Web site, currently in prototype, that offers a high-resolution facsimile of the manuscript for The Book of Margery Kempe and a facing diplomatic transcription.9 The Book of Margery Kempe survives in a single manuscript, but one for which a diplomatic edition was never published—the only standard edition is a hybrid: a critical edition with many silent editorial interventions, such as normalized text.10 Our first objective, then, must be a production of the diplomatic edition that has never existed. Furthermore, as a mystical text, The Book of Margery Kempe is of great interest not just to academics and medievalists but to a wide body of general users; consequently our transcriptions could not simply represent specialized graphemes such as abbreviations and medieval punctuation with direct representations or with the standard scholarly approximations. We wanted all users working with a variety of platforms and browsers to have immediate and transparent access to the transcription; to have the ability to see the text representing as exactly as possible the characters in the manuscript as they were presented there; to be able to switch between this direct representation with abbreviations to an expanded version to ease the reading process for non-specialists; and to be able to separate the original text from several layers of commentary added by later medieval hands.
8Clearly facsimiles with diplomatic editions are a good fit for the Web, but our global culture's move away from print consciousness is far from complete, and TEI P5 offers some valid, but incomplete, solutions to the problems of display faced by projects such as The Book of Margery Kempe. The charge that TEI is not well designed to address visuality, including bibliographic codes (in the term familiar from McGann 2001, 56) such as the many graphemes unique to the practice of medieval scribes, has been leveled by a number of critics, including some deeply sympathetic with the goals of TEI. Katherine Hayles has summarized this line of criticism well, citing the origins of TEI in the structuralist assumptions of OHCO—the text as an ordered hierarchy of content objects—assumptions that can be seen as a kind of New Critical desire for a platonic ideal of the text freed from the vagaries of its material manifestations (Hayles 2005, 89–96). James Cummings, while quoting Hayles's criticisms approvingly, points out that TEI P5 in some respects addresses this criticism, although a longstanding principal concern for him and other scholars is the problem of coding competing physical hierarchies such as page breaks in a system designed to encode semantic hierarchies such as chapters and paragraphs (Cummings 2008; Renear, Mylonas, and Durand 1993). Remaining undiscussed are issues of transformation both theoretical and practical: how can (and should) XML represent the specific visual manifestations of non-print graphemes?
9TEI P5 does, in fact, offer vastly improved guidance for coding abbreviations for display—introducing the <choice> tag, that can allow users to view the text in abbreviated or expanded form. Maintaining these options, particularly for side-by-side viewing of the facsimile and the manuscript, facilitates a direct transcription of the manuscript's bibliographic codes (such abbreviations are an important feature of the material culture of reading) and ease of use for non-specialist readers (who can toggle back and forth between a clear representation of what they see in the facsimile and what they can more easily understand in semantic terms). This strategy usually presents little challenge for common abbreviations such as wyth that have no special characters
- 11 Use of the abbreviation marker element (<am>) in medieval manuscripts is frequently problematized b (...)
- 12 Here the @n records the size of the drop capital D, as well as of the original space reserved for i (...)
but can occasionally lead to very long strings of code.11 For example, the six-character word "dowtyr" requires seventy-three characters to encode when utilizing a <choice> containing both abbreviation and expansion elements. This length can actually double (to 154 characters) or even quintuple (to 373 characters) when highlighting and/or multiple-line, drop-capital characters are part of the word.12
fol. 9r, "dowtyr" coded with abbreviation and expansion (72 characters):
fol. 9v, "Dowtyr" coded with highlighting, abbreviation, and expansion (154 characters):
<choice><abbr><hi rend="highlight" rendition="#RIA">D</hi>owt
fol. 44v: "DOwtyr" coded with dropcap, highlighting, abbreviation, and expansion (373 characters):
<handShift new="#SALTHOWS"/><hi rend="prompt"
hand="#RED_INK_ANNOTATOR"><hi rend="dropcap" n="3">D</hi></add><hi
rend="overwriting" hand="#RED_INK_ANNOTATOR"><hi rend="dropcap"
10Wherever possible, we have turned to automation to facilitate choice encoding—creating standardized strings of code that replace found strings and that typically complete 80% of our coding, leaving mostly second-pass tasks to our encoders.13
11Our success in encoding choices prompted us to consider the problem of display for many of the glyphs used in medieval manuscript abbreviation and punctuation. Where TEI P5 describes ways to encode these glyphs in XML, these ways are complicated and verbose—requiring 1) <gaiji> in the body of a text; 2) character declarations in the header of a text; and/or 3) entity declarations in a project's schema file with instructions on what to do with these entity declarations when they are encountered in the project's XSLT, without prescription for how these characters should/could actually be displayed in HTML. One older strategy for representing scribal glyphs, used by the editors of the Auchinleck manuscript website and many editions on CD, is to provide a specially-created font for installation by the editions' users—an invasive solution that brings with it ease-of-use issues. Other editors, such as those at The Newton Project involved in the transcription of early modern alchemical manuscripts, have the advantage that Unicode provides at least some alchemical symbols. Since these symbols are unavailable in most fonts, however, the editors elect to use image files planted in the text to represent special characters. However, in Web browsers these image files do not scale when text is resized and become distorted when the entire page is resized, and so, generally, this approach is problematic.
- 14 The problem of such semantic disjunctions is raised in Wittern 2006, though without specific displa (...)
- 15 See note 10 above for one aspect of this problem for the Kempe site. Another approach to this probl (...)
12A more recent strategy, used by the editors of the new Malory Project site, is to rely on Unicode exclusively for special characters and make do with whatever is available that displays more or less like the medieval facsimile. Again, problems across platforms emerge because a vast number of scribal characters simply are not available in Unicode. On the site, abbreviations are also left unexpanded, effectively limiting the site's use to medieval scholars. A bigger problem for non-specialist users is the site's attempt to account for scribal features for which no print equivalent has been created: the site depends on characters in Unicode not created for medieval scribal glyphs. In one case, the editors use the print character barred-l or "ƚ" (U+019A from the "Non-European and Historic" Latin Extended-B chart) to indicate an otiose hairline (a common habit for this scribe) through the letter "l," so that "all" is rendered "aƚƚ." This approach does offer a somewhat analogous correlative from print for a scribal habit. Still, the chosen character has no relationship with otiose hairlines, and the resulting code is confusing both visually for the non-expert user and semantically for search.14 We hasten to add that none of us can pretend to be pure about semantics for special characters at this point.15
- 16 A number of other projects incorporate MUFI's character recommendations in their XML encodings: The (...)
13Ideally we would have a set of Unicode characters that could represent all medieval glyphs, since authors of worldwide importance, such as Chaucer or Dante, along with hundreds of other literary figures from the period, need diplomatic transcriptions in digital documentary editions. And this is why PUAs exist: to provide a resource to scholars with which to propose the adoption of special characters (in Early Hungarian printing, for instance, or medieval manuscripts) by Unicode. As a result, some special characters have taken their place in the Unicode pipeline (http://unicode.org/alloc/Pipeline.html) and/or have become established within a PUA in the hope of eventual Unicode approval, as is the case for a cluster of characters developed by the Medieval Unicode Font Initiative (MUFI).16 The problem with such private codepoints for characters has long been that no standard fonts support them, and that their display on the Web has been impossible without a specialized font downloadable and installable by the user.
14TEI P5 guidance on the use of PUA characters strongly emphasizes the in-text use of <gaiji>, which associate XML references with XML IDs described in a text's header—and which may omit any character reference to a PUA codepoint (that might be used in HTML) entirely—ostensibly so that these special characters can be identified consistently by other XML encoders, searched, and easily replaced in the event that Unicode creates sanctioned codepoints for them. However, the approaches we have seen in practice so far, while soundly reasoned, range from the very expansive, like this:
- i Editor's note: because of the PUA character present in this example, the editors have opted to prov (...)
Figure 1: TEI approach to coding special charactersi
which utilizes character declarations, combinations of Unicode and PUA/MUFI mappings, "standardized" expansions of abbreviations, images, and gaiji to mark up and display representations of characters, to the more abbreviated, like this:
<!ENTITY aolig "">
<!--Entity declaration for Latin Small Ligature AO-->
which still utilizes entity declarations (within a project's schema file) and entities in place of Unicode/hexadecimal code (reportedly so that, if a character in the PUA becomes a Unicode character, its new code point need only be updated once, in the entity declaration in the schema file—though, arguably, there are other ways to automate this process in XML/HTML code). Nonetheless, both approaches only displayed special characters when the characters were supported by a font (or fonts) installed on the user's computer—substituting blank boxes when that font (or those fonts) was (were) not.
15For its part in the discussion, TEI seems to fall somewhere between these two approaches. First, TEI P5's instructions frown on expansive coding:
For brevity of encoding, it may be preferred to predefine internal entities such as the following:
<!ENTITY r1 '<g ref="#r1">r</g>' >
<!ENTITY r2 '<g ref="#r2">r</g>' >
which would enable the same material to be encoded as follows:
<p>Wo&r1;ds in this manusc&r2;ipt are sometimes written in a funny way.</p>
(TEI P5 at 5.3)
16Second, TEI P5's instructions do suggest that there are ways to display special characters on the Web, but focus upon their descriptive markup in XML without any consideration at all for how they might be displayed in HTML (other than as images or Unicode characters). In the process these instructions cast doubts on PUA clusters such as MUFI as a viable alternative or even a supplement to Unicode for anything more than "local processing," as in the only discussion TEI offers about creating new characters:
The creation of additional characters for use in text encoding is quite similar to the annotation of existing characters. The same element g is used to provide a link from the character instance in the text to a character definition provided within the charDecl element. This character definition takes the form of a char element. The element g itself will usually be empty, but could contain a code point from the Private Use Area (PUA) of the Unicode Standard, which is an area set aside for the very purpose of privately adding new characters to a document (TEI P5 at 5.4).
17The Guidelines go on to say that complex special characters may use pre-existing Unicode to construct the character as "a sequence of code points" in existing Unicode or "some locally-defined PUA character (say ) for local processing only." According to the Guidelines, however, neither of these approaches is desirable since "the former loses the fact that the sequence of composed characters is regarded as a single object [and] the second is not reliably portable" (TEI P5 at 5.4).
18PUA characters can be reliably portable when custom fonts can be embedded directly in a Web page, where they can be loaded and rendered automatically by the Web browser—an approach that we have proven can work across browsers (e.g. Chrome, Firefox, Internet Explorer, Opera, and Safari) in our prototype. The option to specify which font or which set of fonts—called a "font family"—the Web browser should use to display text on a Web page has long been a fixture of popular Web design software. Importantly, however, this option merely notes a designer's font preferences and, in point of fact, offers little control over how text is actually displayed by the Web browser. Theoretically, if a designer specifies a font family that includes Georgia, Times New Roman, and Times, the Web browser will first attempt to display text using Georgia and, if that font is not available—that is if that font is not present or installed on the site visitor's computer—then Times New Roman, and, if that font is not available, then Times. If Times is not available, then the Web browser is supposed to default to some standardized font. But "standardized font" is a misnomer. Even among widely-used fonts—like Times New Roman—there can exist substantial variation between different desktop operating systems and applications. And if we extend our discussion to mobile operating systems—those on tablet devices and smart phones—the problem is only further complicated.
19Neither Unicode nor MUFI directly addresses this complication. The working groups for each are tasked with deciding how, if at all, a character should be represented in their respective standards or recommendations and, if so, at which code point. Though these groups are invaluable in this regard, they are not tasked with producing—or regulating—the fonts that will actually support approved characters. That responsibility falls to software and/or font developers. Accordingly, adding to the complication, not all code points are supported by all fonts. This is especially true—and, frankly, should be expected—with Unicode, which contains nearly 250,000 assigned code points. But even were we to narrow our focus just to MUFI's character recommendations, which contain far fewer code points (just over 1,500), we would find that only four fonts currently support the latest version of that standard. The first challenge, then, in resolving the font complication is finding and selecting a font that supports all desired code points. Ensuring that that font is actually displayed—and displayed correctly—by the Web browser is only possible at this point through font embedding.
- 17 While support for CSS is user-definable—a built-in feature, it can be modified and even disabled wi (...)
20Font embedding depends on the cascading style sheet, or CSS, code for the @font-face rule:17
Figure 2: The @font-face rule
- 18 This issue changed with the release of IE 9, which does now support embedded TTFs. Unfortunately, t (...)
- 19 We have developed a tool to test—or "prototype"—special font characters and abbreviations, called P (...)
21There is nothing particularly complicated about this code. TTFs, or TrueType fonts, have been around for decades and can be installed and used across operating systems and applications. EOT, or Embedded OpenType, is a font type invented by Microsoft for use with Internet Explorer versions 4–8, which did not and do not support embedded TTFs.18 So if we want our IE users to see it, our selected font must have an EOT variant. If our selected font does not have an EOT variant—and most fonts will not—our next step might be to create one from the TTF that we selected. While there are a number of tools available to do this, we need to review our font's license to ensure that such conversion is permitted—noting that even if our font's license does permit conversion, that conversion may be flawed or fail altogether.19 Equally important is that the font's license permits embedding. The alternative to using an existing font is, of course, creating one, but that only results in a new font, one that still has to be embedded.
- 20 Tested through version 10.0.
22Embedding a font so that it renders consistently across Web browsers is actually not as simple as the @font-face code above suggests—because different Web browsers read the same CSS code in different ways. Recent security changes in Firefox, for example, have necessitated that 1) the style sheet containing the @font-face rule(s), 2) the font(s) referenced by the rule(s), and 3) the Web page(s) that will use them share the same folder and that 4), in some cases, EOTs, which are not even processed by Firefox, actually precede other embedded font types in the rule(s).20 Accordingly, to make the @font-face rule work, we have to rewrite it:
Figure 3: Rewritten @font-face rule
23We use this code to embed fonts on our project. The addition of the CSS class medieval is used to mark which text should be rendered with our embedded font, Andron Scriptor Web, on pages where both medieval and non-medieval text coexist.21 More recently, we have also begun "wrapping" the TTF version of the font that we use for "other Web browsers" in the Web Open Font Format (WOFF), now supported by the current versions of all major Web browsers (except Internet Explorer 8). This, in addition to enhancing the security of the font, which we license, enables us to compress the enclosed TTF and to reduce its file size—improving site performance.
- 22 The marginalia contain drawings and other elements which will call for other strategies. Although a (...)
- 23 A new version of an embedded font (e.g. one with enhanced character support and/or features) can be (...)
24For visitors to our site who do not have the Andron Scriptor Web font installed on their computers, the code above causes the font to be loaded from our server and rendered by their Web browsers automatically and invisibly. No GIFs, JPGs, or other image types are used to display any of the text in our electronic transcriptions, so special characters can be enlarged alongside other text without distortion and without affecting relative proportions between even large embedded characters, like our drop capitals, and the body text.22 The embedded font also offers other important advantages over its installed counterpart—including tighter control over versioning, which can and frequently does affect how special characters are rendered.23
25Once a suitable font has been embedded in a Web page, displaying a PUA character becomes as simple as encoding its character reference—for example,  (the same as it would be in XML)—in HTML. Accordingly, wrapping a character reference like  in a <gaiji> (e.g. <g ref="#ur-leminskate"></g>) or even omitting the character reference completely (e.g. <g ref="#ur-leminskate"/>) in XML for later processing through XSLT (which will, effectively, restore the element to a simple character reference), strikes us as rather paradoxical—especially since the characters referenced are now demonstrably and reliably "portable" using embedded fonts.
26No doubt, the objection that will be raised here is that our definition of portability does not actually meet the portability test. Our counter-objection would be that the definition of portability found in TEI P5 does not always meet the portability test itself. Respecting characters, the portability test in TEI P5 is really the Unicode test: does this character exist in Unicode? If it does, TEI P5 allows the encoder simply to record a character reference: & for an ampersand, ‒ for an en dash, — for an em dash, and so on. For common characters, this reference is, generally, reliably portable: often, even the XML editor can display such a reference as a familiar glyph. But for less common characters and especially for new or uncommon ones, display may not be possible (for the reasons we discuss above), and the uninitiated encoder may be left scratching his/her head trying to figure out what ⎡ represents. Perhaps the character reference is defined somewhere in the XML document's header or in a schema. Perhaps not. In either case, the likelihood is that the encoder will have to look the character reference up in Unicode to understand its visual significance. Putting aside reliance upon an external source, to say that this approach constitutes portability depends on the continued existence and accessibility of Unicode. If Unicode ceases to exist (replaced by a new standard, for instance), undergoes significant revision resulting in the reassignment of code points, institutes changes in how it is accessed, or becomes inaccessible (for whatever reason), the Unicode character reference ceases to be portable.
27 Admittedly, these are only hypotheticals, and all are highly unlikely. But they do illustrate a double standard in TEI P5's definition of portability: if Unicode character references—which implicitly reference an external source—are considered portable, then why can't the character references from an alternative and/or emerging standard, like MUFI, also be considered portable? The question strikes us as particularly pertinent given that MUFI is largely a collection of Unicode character recommendations, updated as medieval characters are adopted by Unicode. Further, by adopting MUFI and the Andron Scriptor Web font, we have been able to ensure that every character reference included in our XML is defined (through consensus by an international body of academics, scholars, graphic designers, and information technology professionals) and displayable in one of three ways: 1) by visiting our site; 2) by saving site pages for offline viewing; or 3) by installing the Andron Scriptor Web font.
28TEI P5 recommends using the empty <gaiji> in 1) the creation of a combined Unicode entity that has no semantic association with a specific historical character and/or 2) coding nonstandard characters, including characters for which there is reasonable hope of future inclusion in Unicode. The first recommendation clearly does not live up to the ideal of reproducing in XML a text complete with its artefactual manifestations so that the encoded description could substitute for the artefact if the latter is lost. And the second demands effectively that XML wrap with <gaiji> every character not absolutely standard—surely a difficult task dependent to some extent on guesswork.
29Of the use of the <gaiji> to tag single PUA characters, the TEI Guidelines suggest:
In the fullness of time, a character may become standardized, and thus assigned a specific code point outside the PUA. Documents which have been encoded using the mechanism must at the least ensure that this changed code point is recorded within the relevant char element; it will however normally be simpler to remove the char element and replace all occurrences of g elements which reference it by occurrences of the newly coded character (TEI P5 at 5.5).
30As we have previously argued, other mechanisms exist to find occurrences of any text string and replace it with another in XML. Moreover, both suggestions seem to proceed from the assumption that PUA characters are useful for "local processing" (e.g. processing by fonts installed on the user's computer) only—which is certainly no longer the case.
- 24 Mandell, in an e-mail message to the authors (January 25, 2012), notes that she uses the decimal ch (...)
31While we do concur that, for maximum transparency, these PUA characters (and their character references) should be defined somewhere, we wonder if, given the realities of font embedding, there is not some other way to define the character references themselves without having to resort to what seems unnecessary—or unnecessarily verbose—tagging. In her tagging of special characters for the Robert Southey Edition at Romantic Circles, Laura Mandell suggests what may well be such an alternative in the editorial declarations of texts' TEI headers:24
<p>& has been used for the ampersand sign.</p>
<p>£ has been used for £, the pound sign</p>
<p>All other characters, those with accents, non-breaking
spaces, etc., have been encoded in HTML entity decimals.</p>
32No doubt some in TEI will balk at this method of defining character references—preferring, at a minimum, the more verbose method of gaiji tags and character declarations. However, Mandell's method does, in fact, define these character references—and, arguably, in a way that is much more transparent (e.g. to non-encoders) than the methods suggested in the TEI P5 Guidelines on special characters (TEI P5 at 5). which, again, we would argue were based on a different reality at the time that they were devised.
33For the purposes of combined characters—be they PUA or those with assigned Unicode points—meant to represent a single "composed" character and for Unicode characters that have not been employed as prescribed by Unicode, we agree that additional tagging should be considered. For example, <g ref="#ur-leminskate"></g>, where ur-leminskate is a defined XML ID elsewhere in the project's code (perhaps in a text's header within a <charDecl>) and/or its documentation. Alternately, a project might opt to use named entities and entity declarations—or some combination of all of the above—in such cases.
34In summary, TEI P5 does not appear to prefer one methodology over the other for "representation of non-standard characters and glyphs" (or for representation of characters and glyphs used in non-standard ways). It does not offer clear use cases for the gaiji-character declaration or named entity-entity declaration, both of which seem to have been devised at a time when broad support for PUA characters was not possible (especially on the Web). We have demonstrated that this limitation is no longer insoluble. A project can, in fact, now embed fonts supporting such characters directly in its Web pages with very high degrees of cross-browser support to display PUA characters. Wrapping or replacing a character reference processable in both XML and HTML with TEI's <gaiji> for later processing through XSLT, which will restore the element to a character reference, makes little sense to us—especially when alternatives exist for defining characters and character references in XML without necessitating <gaiji> or elaborate header declarations.
35While the case can be made that XML and HTML are descriptive markup languages with differing goals, as the Web becomes a more centralized social and cultural technology (not to mention the preferred way that scholarly editions are presented to audiences worldwide), the interchange between these two languages, as well as the opportunities and the consequences that arise from that interchange, must become more central to TEI. Digital editions that have embraced TEI and XML are far too often limited by print models for textual representation in the absence of guidelines, recommendations, or even exemplars for display. Since XML and HTML are frequently partnered in achieving a common goal, and since a fundamental goal of HTML is display, the problems of and solutions for display must become more central to TEI. It is our hope that TEI will soon embrace encoding solutions that will make possible new levels of accuracy and transparency in presenting the graphic features of texts as they are witnessed in their material artefacts—coding that respects the original purposes and meanings of the thousands of characters for which print and Unicode have never offered equivalents.