An Antic Disposition

Spreadsheet file format performance

noreply@blogger.com (Rob) — Tue, 13 May 2008 19:30:00 +0000

I've been doing some performance timings of file format support, comparing MS Office and OpenOffice. Most of the results are as expected, but some are surprising, and one in particular is quite disappointing.

But first, a little details of my setup. All timings, done by stopwatch, were from Office 2003 and OpenOffice 2.4.0 running on Windows XP, with all current service packs and patches. The machine is a Lenova T60p, dual-core Intel 2.16 Ghz and 2 GB of RAM. I took all the standard precautions -- disk was defragmented, and test files were confirmed as defragmented using contig. No other applications were running and background tasks were all shut down.

For test files, I went back to an old favorite, George Ou's (at the time with ZDNet) monster 50MB XLS file from his series of tests back in 2005. This file, although very large, is very simple. There are no formulas, indeed no formatting or styles. It is just text and numbers, treating a spreadsheet like a giant data table. So tests of this file will emphasize the raw throughput of the applications. Real world spreadsheets will typically be worse than this due to additional overhead from process styles, formulas, etc.

A test of a single file is not really that interesting. We want to see trends, see patterns. So I made a set of variations on George's original file, converting it into ODF, XLS and OOXML formats, as well as making scaled down versions of it. In total I made 12 different sized subsets of the original file, ranging down to a 437KB version, and created each file in all three formats. I then tested how long it took to load each file in each of the applications. In the case of MS Office, I installed the current versions of the translators for those formats, the Compatibility Pack for OOXML, and the ODF Add-in for the ODF support.

I find it convenient to report numbers per 100,000 spreadsheet cells. You could equally well use the original XLS spreadsheet size, or the number of rows of data, or any other correlated variable as the ordinate, but values per 100K cells is simple for anyone to understand.

I'll spare you all the pretty picture. If you want to make some, here is the raw data (CSV format). But I will give some summary observations.

For document sizes, the results are as follows:

Binary XLS format = 1,503 KB per 100K cells
OOXML format = 491 KB per 100K cells
ODF format = 117 KB per 100K cells

So the XML formats are far smaller than the legacy binary format. This is due to the added Zip compression that both XML formats use. Also, note that the ODF files are significantly smaller than the OOXML files, less than 1/4 the size on average. Upon further examination, the XML document representing the ODF content is larger than the corresponding XML in OOXML, as expected, due to its use of longer, more descriptive markup tags. However the ODF XML compresses far better than the OOXML version, enough to overcome its greater verbosity and result in files smaller than OOXML. The compression ratio (original/zipped) for ODF's content.xml is 87, whereas the compression ratio for OOXML's sheet1.xml is only 12. We could just mumble something about entropy and walk away, but I think this area could bear further investigation.

Any ideas?

For load time, the times for processing the binary XLS files were:

Microsoft Office 2003 = 0.03 seconds per 100K cells
OpenOffice 2.4.0 = 0.4 seconds per 100K cells

Not too surprising. These binary formats are optimized for the guts of MS Office. We would expect them to load faster in their native application.

So what about the new XML formats? There has been recent talk about the "Angle Bracket Tax" for XML formats. How bad is it?

Microsoft Office 2003 with OOXML = 1.5 seconds per 100K cells
OpenOffice 2.4.0 with ODF = 2.7 seconds per 100K cells

For typical sized documents, you probably will not notice the difference. However with the largest documents, like the 16-page, 3-million cells monster sheet, the OOXML document took 40 seconds to load in Office, the ODF sheet took 90 seconds to load in OpenOffice, whereas the XLS binary took less than 2 seconds to load in MS Office.

OK. So what are we missing. Ah, yes, ODF format in MS Office, using their ODF Add-in.

Microsoft Office 2003 with ODF, using the ODF Add-in = 74.6 seconds per 100K cells

Yup. You read that right. To put this in perspective, let's look at a single test file, a 600K cells file, as we load it in the various formats and editors:

Microsoft Office 2003 in XLS format = 0.75 seconds
OpenOffice 2.4.0 in XLS format = 3.03 seconds
Microsoft Office 2003 in OOXML format = 8.28 seconds
OpenOffice 2.4.0 in ODF format = 14.09 seconds
Microsoft Office 2003 in ODF format = 515.60 seconds

Can someone explain to me why Microsoft Office needs almost 10 minutes to load an ODF file that OpenOffice can load in 14 seconds?

(I was not able to test files larger than this using the ODF Add-in since they all crashed .)

(Update: Since it is the question everyone wants to know, the beta OpenOffice 3.0 opens the OOXML version of that file in 49.4 seconds, over 10x faster than MS Office loads the ODF document.)

This is one reason why I think file format translation is a poor engineering approach to interoperability. When OpenOffice wants to read an legacy XLS file, it does not approach the problem by translating the XLS into an ODF document and then loading the ODF file. Instead they simply load the XLS file, via a file filter, into the internal memory model of OpenOffice.

What is a file filter? It is like 1/2 of a translator. Instead of translating from one disk format to another disk format, it simply loads the disk format and maps it into an application-specific memory model that the application logic can operate directly on. This is far more efficient than translation. This is the untold truth that the layperson does not know. But this is how everyone does it. That is how we support formats in SmartSuite. That is how OpenOffice does it. And that is how MS Office does it for the file formats they care about. In fact, that is the way that Novell is now doing it now, since they discovered that the Microsoft approach is doomed to performance hell.

So it is with some amusement that I watch Microsoft and others propose translation as a solution to interoperability, creating reports about translation, even a proposal for a new work item in JTC1/SC34 concerning file format translation, when the single concrete attempt at translation is such an abysmal failure. It may look great on paper, but it is an engineering disaster. What customers need is direct, internal support for ODF in MS Office, via native code, in a file filter, not a translator that takes 10 minutes to load a file.

The astute engineer will agree with the above, but will also feel some discomfort at the numbers. There is more here than can be explained simply by the use of translators versus import filters. That choice might explain a 2x difference in performance. A particularly poor implementation might explain a 5x difference. But none of this explains why MS Office is almost 40x slower in processing ODF files. Being that much slower is hard to do accidentally. Other forces must be at play.

Any ideas?

Berry Good, Berry Bad

noreply@blogger.com (Rob) — Thu, 08 May 2008 17:30:00 +0000

It has been an interesting Spring here in Westford, weather-wise. April dipped below freezing on the 3rd, 8th, 9th, 15th and 16th. Then we got a warm spell, a week of days that reached 75 °F (23 °C) and even one day that reached 87.4 °F (30.8 °C) (April 23rd). Then it struck, on May 1st, an overnight low of 28.7 °F (-1.8 °C).

The vulnerability, when a late frost like this occurs, is in bud development. If the plant, by warm sunny days, has been tricked into bud development, and then a freeze occurs, the bud will be injured or killed. Strawberries are particularly prone to this problem.

Because of interactions of thermal inversions at the ground, humidity levels, etc., a simple temperate reading is not an accurate indicator of whether damage actually occurred. For example, if humidity is high, the temperate can dip to freezing, but in the act of freezing water vapor (creating frost) energy is released (latent heat of fusion, as the chemists call it). So you have a few degrees of tolerance if humidity is high, if you have a fog, etc. In fact, commercial strawberry growers will handle this problem by running sprinklers when a freeze threatens, to increase the amount of water around the plants available to freeze, as a buffer to protect the plants. Every degree helps.

But I wasn't so lucky. The extent of damage was not clear until the strawberry plants started blooming this week. Here's what I am seeing.

Above is an example of a normal, healthy strawberry blossom. You see the ring of stames, the male organs of the flower, each with a filiament stalk tipped with an anther containing the pollen. In the center is the receptacle with the many carpels, which are the female side of the equation.

But in the picture below, we see a blossom from my garden that shows injury. Although the plant is sound, and it did flower, the carpels are dead. This blossom will not yield a berry.

From the looks of it, 40-50% of buds are damaged in this way. So no strawberry wine this year. I'll only have enough for fresh eating and ice cream.

Achieving the impossible

noreply@blogger.com (Rob) — Wed, 07 May 2008 22:45:00 +0000

Unadulterated copy of James Clark's Relax NG validator jing. Unadulterated copy of Kohsuke Kawaguchi's Sun Multi-Schema Validator msv. Unadulterated copy of the ODF 1.0 Relax NG schema. Unadulterated copy of the ODF 1.0 Standard, in ODF format.

No errors from either validator.

msv is so good as to tell us "the document is valid". But jing indicates success with only silence. So will I.

Standards Words

noreply@blogger.com (Rob) — Tue, 06 May 2008 19:15:00 +0000

Introduction

There are several words, more widely used than understood, that recur frequently when discussing standards. Specification and standardization requires us precisely to describe technology in such a way that practitioners in that field can achieve the goals set out in the standard. But this precision is only perfectly intelligible to those who share the same code words. What follows is a handful of the more important ones, what they mean, and how they are unintentionally confused or intentionally misused. You are at a distinct disadvantage when reading (or writing) a news article, a blog post, or evaluating an argument if you do not know the correct meaning of the following words.

Standard

Take the definition from ISO/IEC Guide 2:2004, definition 3.2:

[A] document, established by consensus and approved by a recognized body, that provides, for common and repeated use, rules, guidelines or characteristics for activities or their results, aimed at the achievement of the optimum degree of order in a given context.

NOTE Standards should be based on the consolidated results of science, technology and experience, and aimed at the promotion of optimum community benefits.

So, it is a document, a written description, not an embodiment in the form of a product, that is standardized. Its aims are the "achievement of optimum degree of order" and "promotion of optimum community benefits", and it is achieved through consensus and consolidation.

international standard

According to ISO/IEC Guide 2:2004, definition 3.2.1.1:

[A] standard that is adopted by an international standardizing/standards organization and made available to the public.

International Standard

[An] international standard where the international standards organization is ISO or IEC

Note the distinction. With capital letters only ISO or IEC standards apply. With lowercase, other standards are included. This is a bit self-serving. ISO and IEC Standards are the only International Standards, because ISO says so. Sorry ITU, sorry CEN, sorry W3C.

So think of "International Standards" as a controlled mark of ISO, like "parmigiano reggiano" is a controlled mark of the Northern Italian Cheese Consorzio.

Normative

The normative parts of a standard are those which set out the scope and provisions of the standard. See ISO Directives, Part 2, section 3.8.

Provisions

The provisions of a standard consist of:

Requirements that must be met for conformance
Recommendations
Statements of permissible, possible or capable actions

See ISO Directives, Part 2, section 3.12.

Note that standards have specific words which denote and distinguish requirements, recommendations and capabilities. Different standards organizations have different vocabulary for this, so a W3C Recommendation, an IETF RFC and an ISO Standard may have different ways of stating the same provision. For ISO Standards, the conventions are:

"shall" and "shall not" are the normal terms for expressing requirements.
"should" and "should not" are the normal terms for expressing recommendations.
"may" and "need not" are the normal terms for expressing permission.
"can" and "cannot" are the normal terms for expressing possibility and capability.

This is necessary because of the extreme ambiguity of the English language in the area of modality. Consider the following sentences, using the word "must":

(On hearing the doorbell ring), "Oh, that must be the mailman!" [expressing likelihood]
(To a misbehaving child) "You must obey your mother" [expressing obligation]

Or the following exchange with a teenage daughter:

Teen: "I shall return by 11pm" [simple future]
Parent: "No, you shall return by 10pm" [expressing a command]

We can be loose and still be understood, in context, in normal conversation, but in standards work we try to be precise and uniform in the use of our control vocabulary.

Conformance

This simply is a question of whether something meets the requirements of the standard. However, for many standards, there are multiple levels, perhaps even multiple classes of conformance. So you need to be very specific about what you are saying.

For example, you should not ask "Does Excel 2007 conform to OOXML?" You should ask "Is Excel 2007 a conforming transitional class SpreadsheetML Producer?" If you count it all up, OOXML probably has at least 18 distinct conformance classes, by various combinations of applications, documents, readers/writers and transitional/strict conformance classes.

Not in particular that conformance does not mean that an application implements the entire standard.

[My definition above is not very satisfactory. Anyone have something better? Is there an ISO definition of conformance?]

Compliance

This is not a typical standards term. The more typical term is "conformance". Best to avoid it unless you are talking in regulatory or legal context. See ISO Directives, Part 2, section 6.6.1.1:

A document does not in itself impose any obligation upon anyone to follow it. However, such an obligation may be imposed, for example, by legislation or by a contract. In order to be able to claim compliance with a document, the user needs to be able to identify the requirements he/she is obliged to satisfy. The user also needs to be able to distinguish these requirements from other provisions where there is a certain freedom of choice.

Validity

This is an XML term, referring to the relationship between an XML document instance (an XML file) and a schema (the definition of the syntax of the markup language). Generally, an XML document instance is valid if it adheres to the constraints defined in the schema. The precise definition of validity will depend on the schema definition language used.

I'd welcome any suggestions for other words or definitions that should be included here.

The Challenge

noreply@blogger.com (Rob) — Tue, 06 May 2008 01:30:00 +0000

<?xml version="1.0" encoding="UTF-8"?>
<office:document-content
xmlns:office="urn:oasis:names:tc:opendocument:xmlns:office:1.0"
xmlns:text="urn:oasis:names:tc:opendocument:xmlns:text:1.0"
office:version="1.0">
<office:body>
<office:text>
<text:p>Dear Alex Brown. Please prove that I am invalid ODF 1.0 (ISO 26300:2006). I do not think that I am. In fact I think that your statement that there are no valid ISO ODF documents in the world, and that there cannot be, is a brash, irresponsible and indefensible piece of bombast that you should retract.</text:p>
<text:p>(Please note that this document contains no ID, IDREF or IDREFS attributes. Nor does it contain custom content.)</text:p>
</office:text>
</office:body>
</office:document-content>

Release the OOXML final DIS text now !

noreply@blogger.com (Rob) — Sun, 04 May 2008 17:30:00 +0000

The JTC1 Directives [pdf] are quite clear on this point. After a Ballot Resolution Meeting (BRM), if the text is approved, the edited, final version of the text is to be distributed to NB's within 1 month. This requirement is in the Fast Track part of JTC1 Directives, specifically in 13.12:

13.12 The time period for post ballot activities by the respective responsible parties shall be as follows:
.
.
.
In not more than one month after the ballot resolution group meeting the SC Secretariat shall distribute the final report of the meeting and final DIS text in case of acceptance.

The OOXML BRM ended on February 29th. One month after February 29th, if my course work in scientific computing does not fail me, is... let's see, carry the 3, multiply, convert to sidereal time, account for proper nutation of the solar mean, subtract the perihelion distance at first point of Aries, OK. Got it. Simple. One month later is approximately March 29th +/- 3 days.

So the SC34 Secretariat should have distributed the "final DIS text" by March 29th, or at the very least, when the final ballot results on OOXML were known a few days later.

But that didn't happen. Nothing. Silence. What is the hang up? I note that when NB's said that the Fast Track schedule did not give sufficient time to review OOXML, the response from ISO/IEC was "There is nothing we can do. The Directives only permit 5 months". And when NB's protested at the arbitrary 5 day length of the OOXML BRM, the response was similarly dismissive. But when Microsoft needs more time to edit OOXML, well that appears to be something entirely different. "Directives, Schmerectives. You don't worry yourself about no stinkin' Directives. Take whatever time you need, Sir."

It makes you wonder who ISO/IEC bureaucracy is working for? The rights and prerogatives of NB's? Or of large corporations? Almost every decision they made in the OOXML processing was to the the detriment of NB prerogatives.

This delay has practical implications as well. Consider the following:

We are currently approaching a two month period where NB's can lodge an appeal against OOXML. Ordinarily, one of the grounds for appeal would be if the Project Editor did not faithfully carry out the editing instructions approved at the BRM. For example, if he failed to make approved changes, made changes that were not authorized, or introduced new errors when applying the approved changes. But with no final DIS text, the NB's are unable to make any appeals on those grounds. By delaying the release of the final DIS text, JTC1 is preventing NB's from exercising their rights.
Law suits, such as the recent one in the UK, are alleging process irregularities, including (if I read it correctly) that BSI approved OOXML without seeing the final text. I imagine that having the final DIS text in hand and being able to point to particular flaws in that text that should have justified disapproval would bolster their case. But if JTC1 withholds the text, then they cannot make that point as effectively.
There are obvious anti-competitive effects at play here. Microsoft has the final DIS version of the ISO/IEC 29500:2008 standard, and by JTC1 delaying release to NB's, Microsoft is able to have 2+ extra months, free of competition, to produce a fix pack to bring their products in line with the final standard, while other competitors like Sun or Corel are left behind. So much for transparency. So much for open standards. How can this can considered open if some competitors are given a significant time and access advantage?

Note that I'm not talking about the publication of the IS here. I'm talking about the requirements of 13.12 and the release of the final DIS text. Obviously ITTF will have a lot of work to do prepping OOXML for publication. For ODF it took 6 months. For OOXML I would expect it to take at least that long. But that does not prevent adhearance to the Directives, in particular the requirement to distribute the final DIS text.

JTC1/SC34, noticing the delay in the release of this text, adopted the following Resolution at their Plenary in early April:

Resolution 8: Distribution of Final text of DIS 29500

SC 34 requests the ITTF and the SC34 secretariat to distribute the already received final text of DIS 29500 to the SC 34 members in accordance with JTC 1 directives section 13.12 as soon as possible, but not later than May 1st 2008. Access to this document is important for the success of various ISO/IEC 29500 maintenance activities.

This indicates that the final DIS text had already been received by SC34 (but not distributed) as of that date (April 9th).

Well, here we are, May 4th, over two months since the final DIS text was due, and past the date requested by the SC34 Plenary (who by they way have no authority to extend the deadline required by JTC1 Directives, but that is another story). We have nothing.

So, I'll make my own personal appeal. JTC1 has the text. The Directives are clear. The delay is unnecessary and harmful in the ways I outlined above. Release the final DIS text now. Not next month. Not next week. Release it now.

ODF Validation for Dummies

noreply@blogger.com (Rob) — Fri, 02 May 2008 16:15:00 +0000

[Updated 4 May 2008, with additional rebuttal at the end]

Alex Brown has a problem. He can't figure out how to validate ODF documents. Unfortunately, when he couldn't figure it out, he didn't ask the OASIS ODF TC for help, which would have been the normal thing to do. Indeed, the ODF TC passed a resolution back in February 2007 that said, in part:

That the ODF TC welcomes any questions from ISO/IEC JTC1/SC34 and
member NB's regarding OpenDocument Format, the functionality it
describes, the planned evolution of this standard, and its relationship
to other work on the technical agenda of JTC1/SC34. Questions and
comments can be directed to the TC chair and secretary whose email
addresses are given at

http://www.oasis-open.org/committees/tc_home.php?wg_abbrev=office

or through the comments facility at

http://www.oasis-open.org/committees/tc_home.php?wg_abbrev=office

So it is rather uncollegial of Alex to refuse such an open, transparent way of getting his questions answered. But Alex didn't avail himself of that avenue. He just assumed if he couldn't figure out how to validate ODF then it simply couldn't be done, and that ODF was to blame. This is presumptuous. Does he think that in the three years since ODF 1.0 became a standard, that no one has tried to validate a document?

Alex is so sure of himself that he publicly exults on the claimed significance of his findings:

For ISO/IEC 26300:2006 (ODF) in general, we can say that the standard itself has a defect which prevents any document claiming validity from being actually valid. Consequently, there are no XML documents in existence which are valid to ISO ODF.
Even if the schema is fixed, we can see that OpenOffice.org 2.4.0 does not produce valid XML documents. This is to be expected and is a mirror-case of what was found for MS Office 2007: while MS Office has not caught up with the ISO standard, OpenOffice has rather bypassed it (it aims at its consortium standard, just as MS Office does).

I think you agree that these are bold pronouncements, especially coming from someone so prominent in SC34, the Convenor of the ill-fated OOXML BRM, someone who is currently arguing that SC34 should own the maintenance of OOXML and ODF, indeed someone who would be well served if he could show that all consortia standards are junk, and that only SC34 (and he himself) could make them good.

Of course, I've been known to pontificate as well. There is nothing necessarily wrong with that. The difference here is that Alex Brown is totally wrong.

But let's see if we can help show Alex, or anyone else similarly confused, the correct way to validate an ODF document.

First start with an ODF document. When Alex tested OOXML, he used the Ecma-376 OOXML specification. Let's do the analogous test and validate the ODF 1.0 text. You can download it from the OASIS ODF web site. You'll want this version of the text, ODF 1.0 (second edition), which is the source document for the ISO version of ODF.

You'll also want to download the Relax NG schema files for OASIS ODF 1.0, which you can download in two pieces: the main schema, and the manifest schema.

Next you'll need to get a Relax NG validator. Alex recommends James Clark's jing, so we'll use that. I downloaded jing-20030619.zip the main distribution for use with the Java Runtime Environment. Unzip that to a directory and we're almost there.

Since jing operates on XML files and knows nothing about the Zip package structure of an ODF file, you'll need to extract the XML contents of the ODF file. There are many ways to do this. My preference, on Windows, is to associate WinZip with the ODF file extensions (ODT, ODS and ODP) so I can right-click on these files unzip them. When you unzip you will have the following XML files, along with directories for images files and other non-XML resources you can ignore:

content.xml
styles.xml
meta.xml
settings.xml
META-INF/manifest.xml

So now we're ready to validate! Let's start with content.xml. The command line for me was:

java -jar c:/jing/bin/jing.jar OpenDocument-schema-v1.0-os.rng content.xml

(Your command may vary, depending on where you put jing, the ODF schema files and the unzipped ODF files)

The result is a whole slew of error messages:

C:\temp\odf\OpenDocument-schema-v1.0-os.rng:17658:18: error: conflicting ID-types for attribute "targetElement" from namespace "urn:oasis:names:tc:opendocument:xmlns:smil-compatible:1.0" of element "command" from namespace "urn:oasis:names:tc:opendocument:xmlns:animation:1.0" C:\temp\odf\OpenDocument-schema-v1.0-os.rng:10294:22: error: conflicting ID-types for attribute "targetElement" from namespace "urn:oasis:names:tc:opendocument:xmlns:smil-compatible:1.0" of element "command" from namespace "urn:oasis:names:tc:opendocument:xmlns:animation:1.0"

Oh no! Emergency, emergency, everyone to get from street!

I wonder if this is one of the things that tripped Alex up? Take a deep breath. These in fact are not Relax NG (ISO/IEC 19757-2) errors at all, but errors generated by jing's default validation of a different set of constraints, defined in the Relax NG DTD Compatibility specification which has the status of a Committee Specification in OASIS. It is not part of ISO/IEC 19757-2.

Relax NG DTD Compatibility provides three extensions to Relax NG: default attribute values, ID/IDREF constaints and a documentation element. The Relax NG DTD Compatibility specification is quite clear in section 2 that "Conformance is defined separately for each feature. A conformant implementation can support any combination of features." And in fact, ODF 1.0, in section 1.2 does just that: "The schema language used within this specification is Relax-NG (see [RNG]). The attribute default value feature specified in [RNG-Compat] is used to provide attribute default values".

It is best to simple disable the checking of Relax NG DTD Compatibility constraints by using the documented "-i" flag in jing. If you want to validate ID/IDREF cross-references, then you'll need to do that in application code, and not using jing in Relax NG DTD Compatibility mode. Note that jing was not complaining about any actual ID/IDREF problem in the ODF document.

So, false alarm. You can walk safely on the streets now.

(That said, if we can make some simple changes to the ODF schemas that will allow it to work better with the default settings of jing, or other popular tools, then I'm certainly in favor of that. Alex's proposed changes to the schema are reasonable and should be considered.)

So, let's repeat the validation with the -i flag:

java -jar c:/jing/bin/jing.jar -i OpenDocument-schema-v1.0-os.rng content.xml

Zero errors, zero warnings.

java -jar c:/jing/bin/jing.jar -i OpenDocument-schema-v1.0-os.rng styles.xml

Zero errors, zero warnings.

java -jar c:/jing/bin/jing.jar -i OpenDocument-schema-v1.0-os.rng meta.xml

Zero errors, zero warnings.

java -jar c:/jing/bin/jing.jar -i OpenDocument-schema-v1.0-os.rng settings.xml

Zero errors, zero warnings.

java -jar c:/jing/bin/jing.jar -i OpenDocument-manifest-schema-v1.0-os.rng META-INF/manifest.xml

Zero errors, zero warnings.

So, there you have it, an example that shows that there is at least one document in the universe that is valid to the ODF 1.0 schema, disproving Alex's statement that "there are no XML documents in existence which are valid to ISO ODF."

The directions are complete and should allow anyone to validate the ODF 1.0 specification, or any other ODF 1.0 document. Now that we have the basics down, let's work on some more advanced topics.

First, the reader should note that there are two versions of the ODF schema, the original 1.0 from 2005, and the updated 1.1 from 2007. (This is also a third version underway, ODF 1.2, but that needn't concern us here.)

An application, when it creates an ODF document, indicates which version of the ODF standard it is targeting. You can find this indication if you look at the office:version attribute on the root element of any ODF XML file. The only values I would expect to see in use today would be "1.0" and "1.1". Eventually we'll also see "1.2".

It is important to use the appropriate version of the ODF schema to validate a particular document. Our goal, as we evolve ODF, is that an application that knows only about ODF 1.0 should be able to adapt and "degrade gracefully" when given an ODF 1.1 document, by ignoring the features it does not understand. But an application written to understand ODF 1.1 should be able to fully understand ODF 1.0 documents without any additional accommodation.

Put differently, from the document perspective, a document that conforms to ODF 1.0 should also conform to ODF 1.1. But the reverse direction is not true.

To accomplish this, as we evolve ODF, within the 1.x family of revisions, we try to limit ourselves to changes that widen the schema constraints, by adding new optional elements, or new attribute values, or expanding the range of values permitted. Constraint changes that are logically narrowing, like removing elements, making optional elements mandatory, or reducing the range of allowed values, would break this kind of document compatibility.

Now of course, at some point we may want to make bolder changes to the schema, but this would be in a major release, like a 2.0 version. But within the ODF 1.x family we want this kind of compatibility.

The net of this is, an ODF 1.1 document should only be expected to be valid to the ODF 1.1 schema, but an ODF 1.0 document should be valid to the ODF 1.0 and the ODF 1.1 schemas.

That's enough theory! Let's take a look now at the test that Alex actually ran. It is a rather curious, strangely biased kind of test, but the bad thinking is interesting enough to devote some time to examine in some detail.

When he earlier tested OOXML, Alex used the OOXML standard itself, a text on which Microsoft engineers had lavished many person-years of attention for the past 18 months, and he validated it with the current version of the OOXML schema. That is pretty much the best case, testing a document that has never been out of Microsoft's sight for 18 months and testing it with the current version of the schema. I would expect that this document would have been a regular test case for Microsoft internally, and that its validity has been repeatedly and exhaustively tested over the past 18 months. I know that I personally tested it when Ecma-376 was first released, since it was the only significant OOXML document around. So, essentially Alex gave OOXML the softest of all soft pitches.

I think Microsoft's response, that the validity errors detected by Alex are due to changes made to the schema at the BRM, is a reasonable and accurate explanation. The real story on OOXML standardization is not how many changes were made that were incompatible with Office 2007, but how few. It appears that very few changes, perhaps only one, will be required to make Office 2007's output be valid OOXML.

So when testing ODF, what did Alex do? Did he use the ODF 1.0 specification as a test case, a document that the OASIS TC might have had the opportunity to give a similar level of attention to? No, he did not, although that would have validated perfectly, as I've demonstrated above. Instead, Alex uses the OOXML specification, a document which by his own testing is not valid OOXML, then converts it into the proprietary .DOC binary format, then translates that binary format into ODF and then tries to validate the results with the ODF 1.0 schema (i.e., the wrong version of the ODF schema since OpenOffice 2.4.0's output is clearly declared as ODF 1.1), and then applies a non-applicable, non-standard DTD Compatibility constraint test during the Relax NG validation.

Does anyone see something else wrong with this testing methodology?

Aside from the obvious bias of using an input document that Microsoft has spent 18 months perfecting, and using the wrong schemas and validator settings, there is another, more subtle problem.

Alex's test of OOXML and ODF are testing entirely different things. With OOXML, he took a version N (Ecma-376) OOXML document and tried to validate it with a version N+1 (ISO/IEC 29500) version of the OOXML schema.

But what he did with ODF was take a version N+1 (ODF 1.1) document and tried to validate it with an version N (ODF 1.0) of the ODF schema.

These are entirely different operations. One test is testing the backwards compatibility of the schema, the other is testing the backwards compatibility of document instances. It takes no genius to figure out that if ODF 1.1 adds new elements, then an ODF 1.1 document instance will not validate with the ODF 1.0 schema. We don't ordinarily expect backwardly compatible validity of document instances. Again, Alex's tests are biased in OOXML's favor, giving ODF a much more difficult, even impossible task, compared the the versions ran for OOXML.

If we want to compare apples to apples, it is quite easy to perform the equivalent test with ODF. I gave it a try, taking a version N document (the ODF 1.0 standard itself, per above) and validated it with the version N+1 schema (ODF 1.1 in this case). It worked perfectly. No warnings, no errors.

In any case, in his backwards test Alex reports 7,525 errors, "mostly of the same type (use of an undeclared soft-page-break element)" when validating the OOXML text with ODF 1.0 schema. Indeed, all but 39 of these errors are reports of soft-page-break.

Soft page breaks are a new feature introduced in ODF 1.1. It has two primary advantages for accessibility. First it allows easier collaboration between people using different technologies to read a document. Not all documents are deeply structured, with formal divisions like section 3.2.1, etc. Most business documents are loosely structured, and collaboration occurs by referring to "2nd paragraph on page 23" or "the bottom of page 18". But when using different assistive technologies, from larger fonts, to braille, to audio renderings, the page breaks (if the assistive technology even has the concept of a page break) are usually located differently from the page breaks in the original authoring tool. This makes collaboration difficult. So, ODF 1.1 added the ability for applications to write out "soft" page breaks, indicating where the page breaks occurred when the original source document was saved.

Although this feature was added for accessibility reasons, like curb cuts, its likely future applications are more general. We will all benefit. For example, a convertor for translating from ODF to HTML would ordinarily only be able to calculate the original page breaks by undertaking complex layout calculations. But with soft page breaks recorded, even a simple XSLT script can use this information to insert indications of page breaks, or to generate accurate page numbering, etc. Although the addition of this feature hinders Alex's idiosyncratic attempt to validate ODF 1.1 documents with the ODF 1.0 schema, I think the fact that this feature helps blind and visually impaired users, and generally improves collaboration makes it a fair trade-off.

Wouldn't you agree?

That leaves 39 validation errors in Alex's test. 12 of them are reports of invalid values in an xlink:href attribute value. This appears to be an error in the original DOCX file. Garbage In, Garbage Out. For example, in one case the original document has HYPERLINK field that contains a link to content in Microsoft's proprietary CHM format (Compiled HTML). The link provided in the original document does not match the syntax rules required for an XML Schema anyURI (the URL ends with "##" rather than "#") Maybe it is correct for markup like this, with non-standard, non-interoperable URI's, to give validation errors. This is not the first time that OOXML has been found polluting XML with proprietary extensions. But realize that OpenOffice 2.4.0 did not create this error. OpenOffice is just passing the error along, as Office 2007 saved it. It is interesting to note that this error was not caught in MS Office, and indeed is undetectable with OOXML's lax schema. But the error was caught with the ODF schema. This is a good thing, yes? It might be a good idea for OpenOffice to add an optional validation step after importing Microsoft Office documents, to filter out such data pollution.

For the remaining validation errors, they are 27 instances of style:with-tab. Honestly, I have no explanation for this. This attribute does not exist in ODF 1.0 or ODF 1.1. That it is written out appears to be a bug in OpenOffice. Maybe someone there can tell us why the story is on this? But I don't see this problem in all documents, or even most documents.

For fun I tried processing this OOXML document another way. Instead of the multi-hop OOXML-to-DOC-to-ODF conversion Alex did, why not go directly from OOXML to ODF in one step, using the convertor that Microsoft/CleverAge created? This should be much cleaner, since it doesn't have all the legacy code or messiness of the binary formats or legacy application code. It is just a mapping from one markup to another markup, written from scratch. Getting the output to be valid should be trivial.

So I download the "OpenXML/ODF Translator Command Line Tools" from SourceForge. According to their web page, this tool targets ODF 1.0, so we'll be validating against the ODF 1.0 schemas.

This tool is very easy to use once you have the .NET prerequisites installed. The command line was:

odfconvertor /I "Office Open XML Part 4 - Markup Language Reference.docx"

The convertor then chugs along for a long, long, long time. I mean a long time. The conversion from OOXML to ODF eventually finished, after 11 hours, 10 minutes and 41 seconds! And this was on a Thinkpad T60p with dual-core Intel 2.16Ghz processor and 2.0 GB of RAM.

I then rang jing, using the validation command lines from above. It reported 376 validation errors, which fell into several categories:

text:s element not allowed in this context
bad value for text:style:name
bad value for text:outline-level
bad value for svg:x
bad value for svg:y
element tetx:tracked-changes not allowed in this context
"text not allowed here"

In any case, not a lot of errors, but a handful of errors repeated. But it is surprising to see that this single-purpose tool, written from scratch, had more validation errors in it than OpenOffice 2.4.0 does.

In the end we should put this in perspective. Can OpenOffice produce valid ODF documents? Yes, it can, and I have given an example. Can OpenOffice produce invalid documents? Yes, of course. For example when it writes out a .DOC binary file, it is not even well-formed XML. And we've seen one example, where via a conversion from OOXML, it wrote out an ODF 1.1 document that failed validation. But conformance for an application does not require that it is incapable of writing out an invalid document. Conformance requires that it is capable of writing out a valid document. And of course, success for an ODF implementation requires that its conformance to the standard is sufficient to deliver on the promises of the standard, for interoperability.

It is interesting to recall the study that Dagfinn Parnas did a few years ago. He analyzed 2.5 million web pages. He found that only 0.7% of them were valid markup. Depending on how you write the headlines, this is either an alarming statement on the low formal quality of web content, or a reassuring thought on the robustness of well-designed applications and systems. Certainly the web seems to have thrived in spite of the fact that almost every web page is in error according to the appropriate web standards. In fact I promise you that the page you are reading now is not valid, and neither is Alex Brown's, nor SC34's, nor JTC1's, nor Ecma's, nor ISO's, nor the IEC's.

So I suggest that ODF has a far better validation record than HTML and the web have, and that is an encouraging statement. In any case, Alex Brown's dire pronouncements on ODF validity have been weighed in the balance and found wanting.

4 May 2008

Alex has responded on his blog with "ODF validation for cognoscneti". He deals purely with the ID/IDREF/IDREFS questions in XML. He does not justify his biased and faulty testing methodology, not does he reiterate his bold claims that there are no valid ODF 1.0 documents in existence.

Since Alex's blog does not seem to be allowing me to comment, I'll put here what I would have put there. I'll be brief because I have other fish to fry today.

Alex, no one doubts that ID/IDREF/IDREFS constraints must be respected by valid ODF document instances. I never suggested otherwise. But what I do state is that this is not a concern of a Relax NG validator. You can read James Clark saying the same thing in his 2001 "Guidelines for using W3C XML Schema Datatypes with RELAX NG", which says in part:

The semantics defined by [W3C XML Schema Datatypes] for the ID, IDREF and IDREFS datatypes are purely lexical and do not include the cross-reference semantics of the corresponding [XML 1.0] datatypes. The cross-reference semantics of these datatypes in XML Schema comes from XML Schema Part 1. Furthermore, the [XML 1.0] cross-reference semantics of these datatypes do not fit into the RELAX NG model of what a datatype is. Therefore, RELAX NG validation will only validate the lexical aspects of these datatypes as defined in [W3C XML Schema Datatypes].

Validation of ID/IDREF/IDREFS cross-reference semantics is not the job of Relax NG, and you are incorrect to suggest otherwise. Your logic is also deficient when you take my statement of that fact and derive the false statement that I believe that ID/IDREF semantics do not apply to ODF. One does not follow from the other.

You know, as much as anyone, that conformance is a complex topic. One does not ordinarily expect, except in trivial XML formats, that the complete set of conformance constraints will be expressed in the schema. Typically a multi-layered approach is used, with some syntax and structural constraints expressed in XML Schema or Relax NG, some business constraints in Schematron, and maybe even some deeper semantic constraints that are expressed only in the text of the standard and can only be tested by application logic.

For example, a document that defines a cryptographic algorithm might need to store a prime number. The schema might define this as an integer. The fact that the schema does not state or guarantee that it is a prime number is not the fault of the schema. And the inability of a Relax NG validator to test primality is not a defect in Relax NG. The primality test would simply need to be carried out at another level, with application logic. But the requirement for primality in document instances can still be a conformance requirement and it is still testable, albeit with some computational effort, in application logic.

I believe that is the source of your confusion. The initial errors you saw when running jing with the Relax NG DTD Compatibility flag enabled were not errors in the ODF document instances. What you saw was jing reporting that it could not apply the Relax NG DTD Compatibility ID/IDREF/IDREFS constraint checks using the ODF 1.0 schema. That in no way means that the constraints defined in XML 1.0 are not required on ODF document instances. It simply indicates that you would need to verify these constraints using means other than Relax NG DTD Compatibility.

So I wonder, have you actually found ODF document instances, say written from OpenOffice 2.4.0, which have ID/IDREF/IDREFS usage which violates the constraints expressed in ODF 1.0?

Finally, in your professional judgment, do you maintain that this is a accurate statement: "For ISO/IEC 26300:2006 (ODF) in general, we can say that the standard itself has a defect which prevents any document claiming validity from being actually valid. Consequently, there are no XML documents in existence which are valid to ISO ODF."

Embrace the Reality and Logic of Choice

noreply@blogger.com (Rob) — Wed, 30 Apr 2008 15:14:00 +0000

Another neo-colonialist press release from Microsoft's CompTIA lobbying arm, this time inveighing against South Africa's adoption of ODF as a national standard. One way to point out the absurdity of their logic is to replace the reference to ODF with references to any other useful standard that a government might adopt, like electrical standards.

When we do this, we end up with the following.

South Africa Electrical Current Adoption Outdated

South Africa’s recent adoption of the 230V/50Hz residential electrical standard represents a tact that will blunt innovation, much needed for their developing economy. The policy choice – which actually reduces electrical current choice – runs contrary to worldwide policy trends, where multiple electrical standards rule, thus threatening to separate South Africa from the wealth creating abilities of the global electrical industry.

For MonPrevAss, the Monopoly Preservation Association, the overall concern for the global electrical industry is to ensure that lawmakers adopt flexible policies and set policy targets rather than deciding on fixed rules, technologies and different national standards to achieve these targets. Such rigid approaches pull the global electrical market apart rather than getting markets to work together and boost innovation for consumers and taxpayers. “The adoption sends a negative signal to a highly innovative sector” says I.M. Atool, MonPrevAss's Group Director, Public Policy EMEA.

The “South African Bureau of Standards” (SABS) approved the 230V/50Hz residential electrical standard on Friday 18 April as an official national standard. This adoption, if implemented, will reduce choice, decrease the benefits of open competition and thwart innovation. The irony here is that South Africa is moving in a direction which stands in stark relief to the reality of the highly dynamic market, with some 40 different electrical current conventions available today.

“Multiple co-existing electrical standards as opposed to only one standard should be favoured in the interest of users. The markets are the most efficient in creating electrical standards and it should stay within the exclusive hands of the market”, I.M. Atool explains.

In light of the recent ISO/IEC adoption of the Microsoft 240V/55Hz electrical standard, the South African decision will not lead to improvements in the electrical sector. MonPrevAss urges Governments to allow consumers and users to decide which electrical standards are best. We fear that the choice of just one electrical standard runs the risk of being outdated before it is even implemented, as well as being prohibitively costly to public budgets and taxpayers.

Governments should not restrict themselves to working with one electrical standard, and should urge legislators to refrain from any kind of mandatory regulation and discriminatory interventions in the market. The global electrical industry recommends Governments to embrace the reality and logic of choice and to devote their energies to ensuring interoperability through this choice.

Of course, this is just a rehash of an old logical fallacy, related to the old "Broken Windows" fallacy. It is like saying heart disease is a good thing because you have such a wide choice of therapies to treat it. We would all agree that it is far preferable to be healthy and have a wide choice of activities that you want to do, rather than a wide choice of solutions to a problem that you never asked for and don't want.

Consumers don't want a bag of adapters to convert between different formats and protocols. That is giving consumers a choice in a solution to a interoperability problem they didn't ask for and they don't want. Consumers want a choice of goods and services.

Observe the recent standards war with Blu-ray and HD DVD. Ask yourself:

Did consumers want a choice in formats, or did they want a wider choice in players and high definition movies?
Did movie studios want a choice in formats and either the uncertainty over choosing the winner, or the expense of supporting both formats? Or did they really just want a single format that would allow them to reach all consumers?
Did the uncertainty around the existence of two competing high definition formats help or hurt the adoption of high definition technologies in general?
Did consumers who make the early choice to go with HD DVD, say Microsoft XBox owners, benefit from having this choice?

If every private individual, and every private business has the right to adopt technology standards according to their needs, why should governments be denied that same right? Why should they be forced to take the only certain losing side of every standards war -- implementing all standards indiscriminately -- a choice that no rational business owner would make?

How many spreadsheet formats does Microsoft use internally for running their business on? Why should governments be denied choice in the same field that Microsoft itself exerts its right to chose?

Sinclair's Syndrome

noreply@blogger.com (Rob) — Fri, 18 Apr 2008 04:15:00 +0000

A curious FAQ put up by an unnamed ISO staffer on MS-OOXML. Question #1 expresses concerns about Fast Tracking a 6,000 page specification, a concern which a large number of NB's also expressed during the DIS process. Rather than deal honestly with this question, the ISO FAQ says:

The number of pages of a document is not a criterion cited in the JTC 1 Directives for refusal. It should be noted that it is not unusual for IT standards to run to several hundred, or even several thousand pages.

Now certainly there are standards that are several pages long. For example, Microsoft likes to bring up the example of ISO 14496, MPEG 4, at over 4,000 pages in length. But that wasn't a Fast Track. And as Arnaud Lehors reminded us earlier, MPEG 4 was standardized in 17 parts over 6 years.

So any answer in the FAQ which attempts to consider what is usual and what is unusual must take account of past practice JTC1 Fast Track submissions. That, after all, was the question the FAQ purports to address.

Ecma claims (PowerPoint presentation here) that there have been around 300 Fast Tracked standards since 1987 and Ecma has done around 80% of them. So looking at Ecma Fast Tracks is a reasonable sample. Luckily Ecma has posted all of their standards, from 1991 at least, in a nice table that allows us to examine this question more closely. Since we're only concerned with JTC1 Fast Tracks, not ISO Fast Tracks or standards that received no approval beyond Ecma, we should look at only those which have ISO/IEC designations. "ISO/IEC" indicates that the standard was approved by JTC1.

So where did things stand on the eve of Microsoft's submission of OOXML to Ecma?

At that point there had been 187 JTC1 Fast Tracks from Ecma since 1991, with basic descriptive statistics as follows:

mean = 103 pages
median = 82 pages
min = 12 pages
max = 767 pages
standard deviation = 102 pages

A histogram of the page lengths looks like this:

So the ISO statement that "it is not unusual for IT standards to run to several hundred, or even several thousand pages" does not seem to ring true in the case of JTC1 Fast Tracks. A good question to ask anyone who says otherwise is, "In the time since JTC1 was founded, how many JTC1 Fast Tracks have been submitted greater than 1,000 pages in length". Let me know if you get a straight answer.

Let's look at one more chart. This shows the length of Ecma Fast Tracks over time, from the 28-page Ecma-6 in 1991 to the 6,045 page Ecma-376 in 2006.

Let's consider the question of usual and unusual again, the question that ISO is trying to inform the public on. Do you see anything unusual in the above chart? Take a few minutes. It is a little tricky to spot at first, but with some study you will see that one of the standards plotted in the above chart is atypical. Keep looking for it. Focus on the center of the chart, let your eyes relax, clear your mind of extraneous thoughts.

If you don't see it after 10 minutes or so, don't feel bad. Some people and even whole companies are not capable of seeing this anomaly. As best as I can tell it is a novel cognitive disorder caused by taking money from Microsoft. I call it "Sinclair's Syndrome" after Upton Sinclair who gave an early description of the condition, writing in 1935: "It is difficult to get a man to understand something when his salary depends upon his not understanding it."

To put it in more approachable terms, observe that Ecma-376, OOXML, at 6,045 pages in length, was 58 standard deviations above the mean for Ecma Fast Tracks. Consider also that the average adult American male is 5′ 9″ (175 cm) tall, with a standard deviation of 3″ (8 cm). For a man to be as tall, relative to the average height, as OOXML is to the average Fast Track, he would need to be 20′ 3″ (6.2 m) tall !

For ISO, in a public relations pitch, to blithely suggest that several thousand page Fast Tracks are "not unusual" shows an audacious disregard for the truth and a lack of respect for a public that is looking for ISO to correct its errors, not blow smoke at them in a revisionist attempt to portray the DIS 29500 approval process as normal, acceptable or even legitimate. We should expect better from ISO and we should express disappointment in them when they let us down in our reasonable expectations of honesty. We don't expect this from Ecma. We don't expect this from Microsoft. But we should expect this from ISO.

Suggesting ODF Enhancements

noreply@blogger.com (Rob) — Wed, 16 Apr 2008 18:00:00 +0000

There is a good post by Mathias Bauer on Sun Hamburg's GullFOSS blog. He deals with the practical importance of OASIS's "Feedback License" that governs any public feedback OASIS receives from non-TC members.

The ODF TC receives ideas for new features from many places. Many of the ideas come from our TC members themselves, where we have representation from most of the major ODF vendors, from open source projects, interest groups, as well as from individual contributors.

Other ideas come from other vendors or open source projects, from organizations that the TC has a liaison relationship with (like ISO/IEC JTC1/SC34), or individual members of the public.

Contributions from OASIS TC members are already covered by the OASIS IPR Policy. The TC member who contributes written proposals to the TC is obliged from the time of contribution. And other TC members are obliged if they have been TC members for at least 60 days and remain a member 7 days after approval of any Committee Draft. You can see the participation status of TC members here.

For everyone else, those who are not members of the ODF TC, the rules require that proposals, feedback, comments, ideas, etc., come through our comment mailing list. But before you can post to the comment list you must first accept the terms of the Feedback License.

Is this extra step annoying? Yes, it is. But this pain is what is necessary to keep our IP pedigree clean and protect the rights of everyone to implement and use ODF. It is part of the price we pay for open standards. Free does not mean free from vigilance.

One of my responsibilities on the ODF TC is to monitor and process the public comments we receive. Regretfully this is a duty which I've neglected for too long. So I spent some time this week getting caught up on the comments, entering them all into a tracking spreadsheet. We have a total of 180 public comments since ODF 1.0 was approved by OASIS, covering everything from new feature proposals to reports of typographical errors.

The largest single source of comments is from the Japanese JTC1/SC34 mirror committee, where they have been translating the ODF 1.0 standard into Japanese. As you know, you will get no closer reading of a text than when attempting translation, so we're glad to receive this scrutiny. I'll look forward to adding the Japanese translation of ODF along side the existing Russian and Chinese translations soon.

For comments that are in the nature of a defect report, i.e., reporting an editorial or technical error in the standard, we will include a fix in the ODF 1.0 errata document we are preparing. For comments that are in the nature of a new feature proposal, we will discuss on a TC call, and decide whether or not to include it in ODF 1.2.

A sample of some of the feature proposals from the comment list are:

A request to support embedded fonts in ODF documents
A request to support multiple versions of the same document in the same file
A request to allow vertical text justification
A proposal for enhanced string processing spreadsheet functions
A proposal for spreadsheet values to allow units, which would help prevent calculation errors due to mixing units, i.e., adding mm to kg would be flagged as an error.
A proposal for allowing spreadsheet named ranges to have namespaces, with each sheet in a workbook having its own namespace.
A proposal to allow a document to have a "portable" flag to allow it to self-identify that it contains only portable ODF content with no proprietary extensions.
Proposal for adding FFT support to spreadsheet
Proposal for adding overline text attribute

If you have any other ideas for ODF enhancements, or thoughts on the above proposals, please don't post a response to this blog! Remember, you need to use the comment list for your feedback to be considered by the OASIS ODF TC.

Of course, general comments are always welcome on this blog.

New Paths in Standardization

noreply@blogger.com (Rob) — Wed, 02 Apr 2008 13:00:00 +0000

The world should be pleased to note, that with the approval of ISO/IEC 29500, Microsoft's Vector Markup Language (VML), after failing to be approved by the W3C in 1998 and after being neglected for the better part of a decade, is now also ISO-approved. Thus VML becomes the first and only standard that Microsoft Internet Explorer fully supports.

Congratulations are due to the Internet Explorer team for reaching this milestone!

Now that it has been demonstrated that pushing proprietary interfaces, protocols and formats through ISO is cheaper and faster than writing code to implement existing open standards, one assumes that the future is bright for more such boutique standards from Redmond. Open HTML, anyone?

Seeking Open Standards Activists

noreply@blogger.com (Rob) — Wed, 26 Mar 2008 03:45:00 +0000

Some thoughts for Document Freedom Day 2008.

Back a few weeks ago in Geneva, OpenForum Europe hosted an evening of mini-talks and a discussion panel with various well-known personalities in our field: Vint Cerf, Bob Sutor, Andy Updegrove and Håkon Lie. I wasn't able to comment on the event at the time, due to my self-imposed blog silence that week, but I'd like to take the opportunity today to carry forward one of the topics discussed then.

I'd like to take as my launching point the theme of Andy Updegrove's talk, which was "Civil ICT Standards". Andy treats this subject more fully on his blog, and also speaks to the topic in his taped interview with Groklaw's Sean Daly.

Thus spake Updegrove:

But as the world becomes more interconnected, more virtual, and more dependent on ICT, public policy relating to ICT will become as important, if not more, than existing policies that relate to freedom of travel (often now being replaced by virtual experiences), freedom of speech (increasingly expressed on line), freedom of access (affordable broadband or otherwise), and freedom to create (open versus closed systems, the ability to create mashups under Creative Commons licenses, and so on.

This is where standards enter the picture, because standards are where policy and technology touch at the most intimate level.

Much as a constitution establishes and balances the basic rights of an individual in civil society, standards codify the points where proprietary technologies touch each other, and where the passage of information is negotiated.

In this way, standards can protect – or not – the rights of the individual to fully participate in the highly technical environment into which the world is now evolving. Among other rights, standards can guarantee:
That any citizen can use any product or service, proprietary or open, that she desires when interacting with her government.
That any citizen can use any product or service when interacting with any other citizen, and to exercise every civil right.
That any entrepreneur can have equal access to marketplace opportunities at the technical level, independent of the market power of existing incumbents.
That any person, advantaged or disadvantaged, and anywhere in the world, can have equal access to the Internet and the Web in the most available and inexpensive method possible.
That any owner of data can have the freedom to create, store, and move that data anywhere, any time, throughout her lifetime, without risk of capture, abandonment or loss due to dependence upon a single vendor.

Let us call these “Civil ICT Rights,” and pause a moment to ask: what will life be like in the future if Civil ICT Rights are not recognized and protected, as paper and other fixed media disappear, as information becomes available exclusively on line, and as history itself becomes hostage to technology?

This rings true to me. Technology, computer technology in particular, now permeates our lives. We interact with it daily, from the moment the internet-radio alarm clock goes off, until days end, when we check our email "one last time" before going to bed.

Similarly, the standards that define the interfaces between these devices are also of increasing importance. There was once a time when standards dealt only with the "infrastructure", the stuff in the walls and under the panel floor, or in that funny little locked door off the hallway, with all the cables and flashing lights, where strange men with clipboards would occasionally emerge, accompanied by a poof of cold air and the buzzing of machines.

But today, the technology and the standards that mediate the technology are now directly in front of your face. Think MP3 players. Think DVD's. Think DRM. Think cellular phones. Think web pages. Think encryption. Think privacy. Think documents. Think documents-privacy-security-DRM, your data and what you are allowed to do with it, and what others are allowed to do with it, and whether you control any bit of this in this mad world of ours.

Between you and the tasks that want to do today stands technology and the standards that mediate that technology. Standards are damn important.

Now, although the reach of technology and ICT standards has progressed over the years, the organizations and the processes that create these standards have not always kept up. In many cases standardization remains the creature of big industry with little or no consumer input. It is back-room discussions, where companies connive to see how many patents of their own portfolio they can encumber the standard with. A successful standard is one where no major company is left hungry. Consensus means everyone at the table has been fed. That is the traditional world of technology standards. It brings to mind the famous line from Adam Smith:

People of the same trade seldom meet together, even for merriment and diversion, but the conversation ends in a conspiracy against the public, or in some contrivance to raise prices — The Wealth of Nations (I.x.c.27)

Luckily, there is some hope. The proponents of "open standards" seek standards based on principles of open participation, consensus decision making, non-profit stewardship, royalty-free IP, and free access to standards. The web itself, with the underlying network protocol stack, HTML family of formats with DOM and scripting API's is a shining example of what open standards can accomplish. Tim Berners-Lee says it best, in his FAQ's:

Q: Do you have had mixed emotions about "cashing in" on the Web?

A: Not really. It was simply that had the technology been proprietary, and in my total control, it would probably not have taken off. The decision to make the Web an open system was necessary for it to be universal. You can't propose that something be a universal space and at the same time keep control of it.

But it is important to realize that "control" mechanisms in standards go well beyond IP and organization issues. There are other important factors at play, and we need to address these as well. Knut Blind discusses some of these issues a section called "Anti-Competitive Effects of Standards" from his The Economics of Standards (2004).

The negative impact of standards for competition are mostly caused by a biased endowment with resources available for the standardization process itself. Therefor, even when the consensus rule is applied, dominant large companies are able to manipulate the outcomes of the process, the specification of the standard, into a direction which leads to skewed distribution of benefits or costs in favor of their interests.

In other words, participation in standardization activities is time consuming and expensive, and large companies are much more able to make this kind of commitment than small companies, organizations or individuals. So ,large companies rule the world.

This is especially true with standardization at the international level, where decisions are often made at meetings in very expensive international locations. JTC1 is still discussing what technologies would be required to allow participation in meetings without travel. (Hint — its called a "telephone") To put this in perspective, my week in Geneva cost $3687.52. I flew coach, ate most of my meals on the cheap, often just grabbing hors d'oeuvres at receptions, and I received negotiated IBM corporate rates for air and hotel. This is one JTC1 meeting. What if I wanted to be really active? Add in two SC34 Plenary meeting (Norway/Kyoto). Add in JTC1 Plenary meetings. Add in US NB meetings. Add in US NB membership fees, consortium fees, conferences, etc. This starts adding up, around $40,000/year to participate actively in tech standards, and this doesn't include the cost of my time.

How many small companies are going to pay this amount? How many non-profit organizations? How many individuals? Not many.

But in spite of the expense, in spite of the large company bias of the international standardization system, I saw reason for hope at the Geneva BRM. I saw younger participants, with fire in their bellies. I saw FOSS supports from developing countries. I saw Linux on laptops. I saw participants from FOSSFA, SIUG, EFFI, ODF Alliance Brazil, COSS, etc. They joined their NB's, participated in their NB debates and were appointed to represent their countries in the BRM.

Sure, it is only a foot in the door. One in five BRM participants were Microsoft employees. But it was a hopeful sign. We've planted the seed. We must plant more. And we must see that they grow.

Strength in standards participation comes with time, with participation, with networking, with learning the rules (written and unwritten) learning from others, etc. Just as we have FOSS experts in the software engineering, in law, in business, in training/education, we also need experts in standardization. Certainly the bread and butter participation will be from individual engineers, participating for the duration of a particular proposal or group of proposals. But we also need the institutional linchpin participants, those who have taken on leadership positions within standards organizations, and whose influence is broad and deep.

FOSS also needs a standards agenda. In a world of patent encumbered standards controlling the central networks, open source software dies, and dies quickly. We must protect and grow the open standards, for without them we cease to exist.

What standards are important? Which demand FOSS representation? Remember just a few weeks ago, when there was a lot of concern about how the DIS 29500 BRM added explicit mention of the patent-encumbered MP3 standard, but failed to mention Ogg Vorbis at all? Although I sympathize with this concern, the fact is the BRM could not have added Ogg Vorbis at all, because it is not a standard. Are we willing to do more than lament about this? I tell you that if Ogg Vorbis had been an ISO standard it would have been explicitly added to OOXML at the BRM. Are we willing to do something about it?

What are the standards critical to FOSS, and what are we doing about it? What standards, existing or potential, should we be focusing on? I suggest the following for a start:

Ogg Vorbis
Ogg Theora
PNG, ISO/IEC 15948
ODF, ISO/IEC 26300
PDF, ISO 3200
Linux Standard Base (LSB), ISO/IEC 23360
Most of the W3C Recommendations
Most of the IETF RFC's

I'm sure you can suggest many others.

Let's put it all together. Some ICT standards directly impact what we can do with our data and our digital lives. These are the Civil ICT Standards. We need to ensure that these standards remain open standards, so anyone can implement them freely. However, the standardization system, both at the national and international levels is biased in favor of those large corporations best able to afford dedicated staff to work within those organizations and develop personal effectiveness and influence in the process. Showing up once a year is not going to work. If FOSS is going to maintain any level of influence in formal standardization world, especially at the high-stakes international level, it needs to find a way to identify, nurture and support participation of "Open Standards Activists". The GNOME Foundation's joining of Ecma, or KDE's membership in OASIS are examples how this could work. Umbrella organizations like Digistan also are critical and can be a nucleus for standards activists. But what about taking this to the next level, to NB membership? Another example is the Linux Foundation's Travel Fund, designed to sponsor attendance of FOSS developers at technical conferences. Imagine what could be done with a similar fund for attendance at standards meetings?

So that is my challenge to you on this first Document Freedom Day. We're near the end of what promises to be one of many battles. The virtual networks of the future are just as lucrative as the railroad and telephone networks of the last century were. These include the network of compatible audio formats, or the network of IM users using a compatible protocol, or the network of users using a single open document format. If FOSS projects and organizations want to secure the value for their users that comes from being part of these networks, then FOSS projects must encourage the use of open standards, and must also encourage and nurture new talent for the next generation of open standards activists.

I'm looking forward to the day, soon, when I can search Google for "open standards activist" and not find a paid Microsoft shill among the listings on the first page.

OOXML's (Out of) Control Characters

noreply@blogger.com (Rob) — Mon, 24 Mar 2008 20:25:00 +0000

Let's start with the concepts of “lexical” and “value” spaces in XML, as well as the mechanism of “derivation by restriction” in XML Schema. Any engineer can understand the basics here, even if you don't eat and drink XML for breakfast.

The value space for an XML data item comprises the set of all allowed values. So the value space for the “float” data type would be all floating point numbers, such as 12.34 or 43.21. The lexical space comprises all ways of expressing these values in the character stream of an XML document. So lexical representations of the value 12.34 include “12.34”, “12.340” and '1.234E1”. For ease of illustration I will indicate value space items in bold, and lexical space items in quotes. In general there are multiple lexical representations that may represent the same value.

Character data in XML also permits more than one lexical representation of the same value. For example, “A” and “A” both represent the value A. The “numerical character reference” approach allows an XML author to easily encode the occasional Unicode character which is not part of the author's native editing environment, e.g., adding the copyright character or occasional foreign character. The value space allowed by XML includes most of Unicode, including all of the major writing systems of the world, current and historical.

The concern I have with DIS 29500 concerns Ecma's introduction of a ST_XString (Escaped String) datatype. This new type is defined via the following XML Schema definition:

<simpletype name="ST_Xstring">
<restriction base="xsd:string">
</simpletype>

This uses the “derivation by restriction” facility of XML Schema to define a new type, derived from the standard xsd:string schema type. The xsd:string type is defined to allow only character values that are also allowed in the XML standard.

The use of derivation by restriction implies a clear relationship between the ST_Xstring type and the base type xsd:string. This is stated in XML Schema Part 1, clause 2.2.1.1:

A type definition whose declarations or facets are in a one-to-one relation with those of another specified type definition, with each in turn restricting the possibilities of the one it corresponds to, is said to be a restriction.

The specific restrictions might include narrowed ranges or reduced alternatives. Members of a type, A, whose definition is a restriction of the definition of another type, B, are always members of type B as well.

The latest sentence can be taken as a restatement of the Liskov Substitution Principle, a fundamental principle of interface design, that a subtype should be usable (substitutable) wherever a base type is usable. It is this principle that ensures interoperability. A type derived by restriction limits, restricts, constrains, reduces the permitted value space of its base type, but it cannot increase the value space beyond that permitted by its base type.

So, with that background, let's now look at how OOXML defines the semantics of its ST_Xstring type:

ST_Xstring (Escaped String)

String of characters with support for escaped invalid-XML characters.

For all characters which cannot be represented in XML as defined by the XML 1.0 specification, the characters are escaped using the Unicode numerical character representation escape character format _xHHHH_, where H represents a hexadecimal character in the character's value. [Example: The Unicode character 8 is invalid in an XML 1.0 document, so it shall be escaped as _x0008_. end example]

This simple type's contents are a restriction of the XML Schema string datatype.

In other words, although ST_Xstring is declared to be a restriction of xsd:string it is, via a proprietary escape notation, in fact expanding the semantics of xsd:string to create a value space that includes additional characters, including characters that are invalid in XML.

Let's review some of the problems it introduces.

First, the semantics of XML strings that contain invalid XML-characters is undefined by this or any other standard. For example, OOXML uses ST_Xstring in Part 4, Clause 3.3.1.30 to store the error message which should be displayed when a data validation formula fails. But what should an OOXML-supporting application do when given a display string which contains control characters from the C0 control range, characters forbidden in XML 1.0?

U+0004 END OF TRANSMISSION
U+0006 ACKNOWLEDGE
U+0007 BELL
U+0008 BACKSPACE
U+0017 SYNCHRONOUS IDLE

How should these characters be displayed?

There is a reason XML excludes these dumb terminal control codes. They are neither desired nor necessary in XML.

Elliotte Rusty Harold explains the rationale for this prohibition in his book Effective XML:

The first 32 Unicode characters with code points 0 to 31 are known as the C0 controls. They were originally defined in ASCII to control teletypes and other monospace dumb terminals. Aside from the tab, carriage return, and line feed they have no obvious meaning in text. Since XML is text, it does not include binary characters such as NULL (#x00), BEL (#x07), DC1 (#x11) through DC4 (#x14), and so forth. These noncharacters are historic relics. XML 1.0 does not allow them.

This is a good thing. Although dumb terminals and binary-hostile gateways are far less common today than they were twenty years ago, they are still used, and passing these characters through equipment that expects to see plain text can have nasty consequences, including disabling the screen.

Further, since these characters are undefined in XML, they are unlikely to work well with existing accessibility interfaces and devices. At best these characters will be ignored and introduce subtle errors. For example, what does “$10,[BS]000” become if one system processes the backspace and another does not? Worst case, the accessibility interface expecting a certain range of characters as defined by the xsd:string type will crash when presented with values beyond the expected range.

Interfaces with existing programming languages are also harmed by ST_Xstring. How does a C or C++ XML parser deal with XML that now can allow a U+0000 (NULL) character in the middle of a string, something which is illegal in that programming language?

What about XML database interfaces that take XML data and store it in relational tables? If they are schema-aware and see that ST_Xstring is merely a restriction of xsd:string, they will assume the normal range of characters can be stored wherever an xsd:string can be stored. But since the value space is expanded, there is no guarantee that this will still be true. These characters may cause validation errors in the database.

By now, the observant reader may be accusing me of pulling a fast one. "But Rob, none of the above is a problem if the application simply leaves the ST_Xstring encoded and does not try to decode or interpret the non-XML character," you might say.

OK. Fair enough. Let's follow that approach and see where it leads us.

Let's look at interoperability with other XML-based standards. Imagine you do a DOM parse of an OOXML document that contains “strings” of type ST_Xstring. Either your parser/application is OOXML-aware, or it isn't. In other words, either it is able to interpret the non-standard _xHHHH_ instructions, or it isn't.

If it doesn't understand them, then any other code that operates on the DOM nodes with ST_Xstring data is at risk of returning the wrong answer. For example, what is the length of the string “ABC”? Three-characters, of course. But what is the length of the string “_x0041_BC” ? These two strings both have the same values according to OOXML. But an XML application might return 9 or return 3, depending on whether it is OOXML-aware or not. Since most (all) XML parsers are unaware of the non-standard escape mechanism proposed by OOXML, they will typically calculate things such as string lengths, string comparisons, string sorting, etc., incorrectly.

But suppose the parser/application is OOXML-aware and correctly decodes these character references into the correct Unicode values, then what? Assuming the host language doesn't crash from the existence of this control characters, we then are presented with problems at the interface with any other code that operates on the DOM. Suppose we try to transform the DOM via XSLT to XHTML. Will the XSLT engine properly handle the existence of these forbidden character values? The XSLT engine may just crash. But suppose it doesn't. How does it write out these control characters into XHTML? It can't. These values are not permitted in XHTML. Dead end. What about DocBook? DITA? OpenDocument Format? Not possible. Since these characters are not permitted in XML 1.0 at all, they will be forbidden in all other markup languages that are based on XML 1.0, or even XML 1.1 for that matter (XML 1.1 allows some but not all of these characters, in particular the NULL character is excluded).

Note further that with XML pipelining and with mashups, the application that writes XML output typically does not have direct knowledge of the application that originally produced the XML values. This decoupling of producers and consumers is an essential aspect of modern systems integration, include Web Services. By corrupting XML string values in the way that it does, DIS 29500 breaks the ability to have loosely coupled systems. Once the value space is polluted by these aberrant control characters, every application, every process that touches this data must be aware of their non-standard idiosyncrasies lest they crash or return incorrect answers. In this way, one standard perverts the entire XML universe, forcing them all to contend with the poor hygiene of a single vendor.

The reader might think that I exaggerate the importance of this, that surely ST_Xstring is only used in OOXML in edge cases, in rare, compatibility modes. We wish that this were true. However, a look at the DIS 29500 shows that ST_Xstring is pervasive, and in fact is the predominant data type in SpreadsheetML, used to express the vast majority of spreadsheet content, including cell contents, headers, footers, displays strings, error strings, tooltip help, range names, etc. Any application that operates on an OOXML spreadsheet will need to deal with this mess.

For example, here are some uses of ST_Xstring in DIS 29500, Part 4:

Clause 3.2.3 for the name of a custom view in a spreadsheet
Clause 3.2.5 for the name of a spreadsheet named range, for the descriptive comment, for the name description, for the
help topic, the keyboard shortcut, the status bar text and for the menu item text
Clause 3.2.14 for the name of a spreadsheet function group
Clause 3.2.19 for the name of a sheet in a workbook
Clause 3.2.22 for the name of a smart tag as well as for the URL of a smart tag.
Clause 3.2.25 for the destination file name and title when publishing spreadsheet to the web.
Clause 3.3.1.10 for the value of a conditional formatting object, e.g., a gradient
Clause 3.3.1.20 for the name of a custom property
Clause 3.3.1.28 for sheet and range names
Clause 3.3.1.30 for error message string, error message title, prompt string and prompt title in a spreadsheet data validation definition.
Clause 3.3.1.35 for the value of a footer for even numbered pages.
Clause 3.3.1.36 for the value of a header for even numbered pages.
Clause 3.3.1.38 for the content of the first page footer
Clause 3.3.1.39 for the content of the first page header
Clause 3.3.1.44 for the display string for a hyperlink, the tooltip help for the link, also the anchor target if the hyperlink is to an HTML page
Clause 3.3.1.49 for values of input cells in a scenario
Clause 3.3.1.50 for cell inline text values
Clause 3.3.1.55 for the value of a footer for odd numbered pages.
Clause 3.3.1.56 for the value of a header for odd numbered pages.
Clause 3.3.1.73, in scenarios for the comment text, the scenario name and the name of the person who last changed the scenario.
Clause 3.3.1.88 when defining sort condition, for the values of a the custom sort list
Clause 3.3.1.93 for the value contained within a cell
Clause 3.3.1.94 for information associated with items published to the web, including the destination file and the title of the output HTML file
Clause 3.3.2.2 for expressing the criteria values in a filter
Clause 3.3.15 for the key/values for smart tag properties
Clause 3.4.4 for expressing the contents of a rich text run
Clause 3.4.5 for expressing the name of a font
Clause 3.4.6 for expressing the text of a phonetic hint for East Asian text
Clause 3.4.8 for expressing a text item in the shared string table
Clause 3.4.12 for the text content shown as part of a string
Clause 3.5.1.2 for a table, expressing a textual comment, a display name as well as style names.
Clause 3.5.1.3 for a table column, expressing cell and row style names, column name
Clause 3.5.1.7 for column properties created from an XML mapping, for expressing the associated XPath.
Clause 3.5.2.4 for the XPath associated with column properties for XML tables
Clause 3.7.1-3.7.6 for specifying content of tracked comments, including the text of the comments as well as the authors of the comments
Clause 3.8.29 expressing the name of a font

There are hundreds of additional uses. A search of DIS 29500 Part 4 for “ST_Xstring” returns 467 hits. OOXML also defines two additional types, “lptsr” (7.4.2.8) and “bstr” (7.4.2.4) that have the same flaw as ST_Xstring.

The reader might further argue that, although the type allows characters that are forbidden by XML, the actual occurrence of these values in real legacy documents is likely to be rare. This might be true, but this is cause for even greater concern. If every document contained these control characters, then we would immediately be aware of any interoperability problems when integrating OOXML data with other systems. But if these characters are permitted, but occur rarely and randomly, then the integration errors will also occur rarely and randomly, allowing data corruption and other problems to occur and propagate further before detection.

In summary, we are concerned that the ST_Xstring type in OOXML opens us up to problems such as:

Introducing accessibility problems
Breaking unaware C/C++ XML parsers
Breaking XML databases
Breaking interoperability with other XML languages
Breaking application logic related to string searching, sorting, comparisons, etc.
Introducing errors that will be hard to detect and resolve

Possible remedies include:

Use xsd:string uniformly instead of ST_Xstring, with no use of forbidden XML characters. This would require that applications that read legacy binary documents containing such characters eliminate them at this point, perhaps replacing them with licit characters or with whitespace. No application will be more able to devise the original meaning and intent of these characters than the original vendor. So they should be responsible for cleaning up these strings to make them XML-ready.
Use a non-string type such as the binary xsd:hexBinary or xsd:base64Binary to represent these data items.
Use a mixed content encoding, where the licit characters are represented by xsd:string data, and the forbidden characters are denoted by specially-defined elements. So “A_x0008_BC” would become: <text>A<backspace/>BC </text>. In this case the semantics of the <backspace> element would need to be documented in the DIS 29500 specification, including its effect on searching, sorting, length calculations, etc.

Five (Bad) Reasons to Approve OOXML

noreply@blogger.com (Rob) — Mon, 24 Mar 2008 19:00:00 +0000

If you don't approve OOXML, Microsoft will walk away, and you'll never hear from them again. Forget the fact that OOXML is already an Ecma standard (Ecma-376), and cannot be taken away. Forget the fact that Microsoft has other formats lined up for ISO approval in the near future, like XPS or HD Photo. Microsoft wants you to think that if you don't give them exactly what they want, now, they will walk away from ISO and you will be the worse from it. We need to encourage Microsoft for their abuse of the standardization process, in hopes that their participation will evolve in line with our hopes, and not our fears, that they will improve on the standardization side, while curbing the abuse side. Of course, the encouragement could be misinterpreted to mean the opposite, and we could get more abuse, and even lower quality standards. I guess that's the risk we'll just need to take. By similar abuses of logic small children hold their breath until their faces turn blue, thinking they can scare adults into giving them what they want. It doesn't work there either.
If you approve OOXML, you can have the privilege of spending the next 5 years in the glorious work of fixing thousands of defects in the text. You can get a seat at the table, fixing bugs that should have been fixed in Ecma before OOXML was even submitted to JTC1. Forget the fact that maintenance in JTC1 is a ponderous, time consuming activity, where individual defects are enumerated, changes proposed, discussed, voted on, etc. Forget the fact that the recent BRM showed that you can't really get through more than 60 defects in a week-long meeting. Forget the fact that fixing defects in Ecma, not JTC1, would be far faster and easier due to the lighter-weight process Ecma imposes on their TC's. Forget that Fast Track is intended for mature, adopted standards not for ones that will require a "Perpetual BRM". Forget all that. You want a seat at the bug fixing table? You got it.
Billions and Billions of legacy documents. Well, actually these legacy documents are not in OOXML format; they are in the legacy binary format. And no mapping has been provided from the legacy formats to OOXML. But there are billions and billions of these legacy documents. That must be important. So vote Yes for OOXML because there are billions and billions of documents in some other format that is nebulously related to it.
More standards are better. More standards means more choice, means more decisions, means more consultants, means more money paid to XML experts. You'll sooner find the American Dairy Council recommending less milk consumption than a standards professional calling for fewer standards. So ignore quality, maturity and need. More standards are a good thing. Like Blue-ray and HD DVD.
ODF will be better if OOXML is approved. In OASIS we're too stupid to look up legacy features or Excel spreadsheet formulas in Ecma-376. We would have never thought of that. We believe the only way to make ODF better is to make it more like OOXML. That is why we would like to encourage nice little JTC1 countries like Kazakhstan to vote YES for OOXML. As soon as OOXML is approved, then magically, it becomes useful to us. But the exactly same text, not approved by Kazakhstan and JTC1, is not useful to us at all. It is all or nothing. There is nothing in the middle. Rather than taking a useful, high quality text, and approving it on its merits, we are asked to approve a specification with thousands of defects, and by our approval we transform it into something useful to ODF.