{"id":163,"date":"2008-03-24T15:25:00","date_gmt":"2008-03-24T20:25:00","guid":{"rendered":"http:\/\/2d823b65bb.nxcli.io\/2008\/03\/ooxmls-out-of-control-characters.html"},"modified":"2008-03-24T15:25:00","modified_gmt":"2008-03-24T20:25:00","slug":"ooxmls-out-of-control-characters","status":"publish","type":"post","link":"https:\/\/www.robweir.com\/blog\/2008\/03\/ooxmls-out-of-control-characters.html","title":{"rendered":"OOXML’s (Out of) Control Characters"},"content":{"rendered":"

Let’s start with the concepts of \u201clexical\u201d and \u201cvalue\u201d spaces in XML, as well as the mechanism of \u201cderivation by restriction\u201d in XML Schema. Any engineer can understand the basics here, even if you don’t eat and drink XML for breakfast.<\/p>\n

The value space for an XML data item comprises the set of all allowed values. So the value space for the \u201cfloat\u201d data type would be all floating point numbers, such as 12.34<\/b> or 43.21<\/b>. The lexical space comprises all ways of expressing these values in the character stream of an XML document. So lexical representations of the value 12.34<\/b> include \u201c12.34\u201d, \u201c12.340\u201d and ‘1.234E1\u201d. For ease of illustration I will indicate value space items in bold, and lexical space items in quotes. In general there are multiple lexical representations that may represent the same value.<\/p>\n

Character data in XML also permits more than one lexical representation of the same value. For example, \u201cA\u201d and \u201c&#65;\u201d both represent the value A<\/span>. The \u201cnumerical character reference\u201d approach allows an XML author to easily encode the occasional Unicode character which is not part of the author’s native editing environment, e.g., adding the copyright character or occasional foreign character. The value space allowed by XML includes most of Unicode, including all of the major writing systems of the world, current and historical.<\/p>\n

The concern I have with DIS 29500 concerns Ecma’s introduction of a ST_XString (Escaped String) datatype. This new type is defined via the following XML Schema definition:<\/p>\n

<simpletype name=”ST_Xstring”>
<restriction base=”xsd:string”>
<\/simpletype><\/p>\n

This uses the \u201cderivation by restriction\u201d facility of XML Schema to define a new type, derived from the standard xsd:string schema type. The xsd:string type is defined to allow only character values that are also allowed in the XML standard.<\/p>\n

The use of derivation by restriction implies a clear relationship between the ST_Xstring type and the base type xsd:string. This is stated in XML Schema Part 1, clause 2.2.1.1<\/a>:<\/p>\n

A type definition whose declarations or facets are in a one-to-one relation with those of another specified type definition, with each in turn restricting the possibilities of the one it corresponds to, is said to be a restriction<\/b>.<\/p>\n

The specific restrictions might include narrowed ranges or reduced alternatives. Members of a type, A, whose definition is a restriction of the definition of another type, B, are always members of type B as well.<\/p><\/blockquote>\n

The latest sentence can be taken as a restatement of the Liskov Substitution Principle<\/a>, a fundamental principle of interface design, that a subtype should be usable (substitutable) wherever a base type is usable. It is this principle that ensures interoperability. A type derived by restriction limits, restricts, constrains, reduces the permitted value space of its base type, but it cannot increase the value space beyond that permitted by its base type.<\/p>\n

So, with that background, let’s now look at how OOXML defines the semantics of its ST_Xstring type:<\/p>\n

ST_Xstring (Escaped String)<\/p>\n

String of characters with support for escaped invalid-XML characters.<\/p>\n

For all characters which cannot be represented in XML as defined by the XML 1.0 specification, the characters are escaped using the Unicode numerical character representation escape character format _xHHHH_, where H represents a hexadecimal character in the character’s value. [Example: The Unicode character 8 is invalid in an XML 1.0 document, so it shall be escaped as _x0008_. end example]<\/p>\n

This simple type’s contents are a restriction of the XML Schema string datatype.<\/p><\/blockquote>\n

In other words, although ST_Xstring is declared to be a restriction of xsd:string it is, via a proprietary escape notation, in fact expanding the semantics of xsd:string to create a value space that includes additional characters, including characters that are invalid in XML.<\/p>\n

Let’s review some of the problems it introduces.<\/p>\n

First, the semantics of XML strings that contain invalid XML-characters is undefined by this or any other standard. For example, OOXML uses ST_Xstring in Part 4, Clause 3.3.1.30 to store the error message which should be displayed when a data validation formula fails. But what should an OOXML-supporting application do when given a display string which contains control characters from the C0 control range, characters forbidden in XML 1.0?<\/p>\n