I have seen some conformance work in OASIS, though as a separate TC. For example there was an XSLT Conformance TC a few years ago that developed an XSLT/XPath test suite. And the W3C does have its online validators, which is great.

A test suite is a daunting task. Some work was started at the University of Central Florida and picked up by the OpenDocument Fellowship but it has only a few hundred test cases. ODF, a 700 page specification probably has on the order of 5 testable statements per page, each one of which could require 4 test cases to test the main and edge conditions, positive and negative tests. So we’re talking 14,000 test cases. Even if I’m off by a factor of 2 or 4, this is clearly a large undertaking. Project this out to OOXML’s 6,000 pages and you would need 120,000 test cases.

One approach to taming this complexity is what we’re doing on the ODF Formula Subcommittee. As we define spreadsheet functions, we’re putting in plenty of test cases, 3, 4 or more per function. This is done in a structured way, with a special named style in the document. We then have a tool that post-processes the ODF XML of the specification, extracts the formula test cases and creates an ODF spreadsheet document that executes each one of the test cases. So we in essence have a self-testing specification, at least in this area.

Hi Ben,

The ODF specification explicitly handles the case of what to do when you encounter an extension that you don’t understand. You simply ignore it. OOXML has another way of handling this. They allow the application writing the document to encode a feature multiple ways. For example, you could describe a new bullet list feature using a markup extension understood only by your application and then also provide an alternative rendering in standard OOXML 1.0 to be used if the extension is not understood. The application reading the document processes this section like it was a switch/case/default statement in C. This is a clever approach and probably worth doing something similar in ODF.

For the case of functional subsets, I still think profiles are the way to go. This pushes the decision making of how to degrade gracefully into the application doing the writing rather than the one doing the reading. Of course you don’t always know in advance whether you are targeting a partial implementation, but when you do, that will give you the best results.

Anonymous #1, you raise a good point. I’d note that a test suite or a certification program is as much a tool for the consumer or evaluator/purchaser of tools as it is to the vendor.

Anonymous #2, thanks. What you describe is one of my bigger complaints against OOXML. They have many duplicative ways of describing the same thing, due to their legacy support. So graphics can be done in DrawingML or in VML. A document can contain section described in WordProcessingML, or HTML or RTF or legacy versions of WordProcessingML. Sure, some of this is deprecated or oprional, but a consumer, needs to be able to handle anything. Even the most trivial of consumers, a tool to extract text for indexing for a search engine will need to handle these many alternatives.

With XML we at least have a good set of parsers that hide us from the complexities of the lexical space and give us a nice clean InfoSet. But we don’t yet have similar levels of abstraction for ODF or OOXML. Of course there is the ODF Toolkit that Sun is starting up at OpenOffice.org. That is another factor that can increase interoperability: wrap the complexity of the format in a tool. If the tool is coded to handle the full complexity of the format and to emit only valid ODF, then those who use the tool will experience enhanced interoperability compared to those who write code from scratch.

One of the issues in dealing with a format is the generator-consumer relationship. The generator is basically free to choose the subset of features from a specification that best suites its needs. The consumer on the other hand, is typically expected to handle input from any generator. This requires handling every aspect of the specification correctly.

An example might be the choice of encoding of an XML file. Let’s leave alone the way you’re supposed to detect encodings given that the encoding is encoded in that encoding :). A generator could simply choose an encoding of convenience. The consumer must handle any potential encoding. Given the broad set of possible choices, the consumer ends up having to pull in support for a broad array of encodings.

I guess what I’m trying to say is that the freedom of choice for the generator comes at a significant cost to the consumer.

This is a pervasive theme for XML. An XML consumer needs to deal with dozens of oddities in the XML specification. Reserved character handling, the issue of whitespace significance, schemas, binary data nonsupport, etc. In the end, consumers end up pulling monster libraries and then have to add custom logic on top just to read the format. Sometimes.

The ODF spec is very open to multiple interpreting of it parts and thing like the “office-settings” and the extensibility allow for a wide variety in ODF files.
I it very difficult to get agreement on a set op interoperability guidelines and consent about testsets if large parties will not agree to those and they are not a formal part of the standard specifications.

It is very well possible that in the a year or two an enourmous amount ODF files will be created using plugins for MS Office. Mayby even more then with OOo. As MS is harresed severly over ODF they might not be persuaded very easily to commit to interoperability. Showing the poor interoperability that the ODF spec allows would confirm why they could not use it as their Office format.

The key here is that if the implementation degrades gracefully, and even better if the standard defines a way for the implementation to degrade gracefully, interoperability is well served. Take HTML as an example. If an unknown tag is excountered, should the implmentation fail, ignore the tag and everything in it, ignore the tag but not child tags within it? Those are very important issues with a standard and extensions beyond the standard, and there are ways to define how both subsets and supersets should be handled.

1. you can have a formal spec written in a formal language like Z, this should give you a rigorous spec without the need to implement all the code. But it is very hard to enumerate all possible failure modes, as that is so implementation specific.

2. test suites. First. I have found incredible resistance to test-driven development in the OASIS and W3C groups, which I think is partially driven by the fact that many of the people who participate havent coded for a while, and are not current with modern test-driven processes. There is also something about a standards body, where
the timely 1.0 bit of paper is deemed important, and the implementation an afterthought.

A big issue here is politics. There was a lot of pressure to ship ODF in a timely way, and now there is even more pressure from MS to catch up. But without test suites, there is no formal way to verify compliant implementations of either spec.

There is another way to do interop: the open source way. You dont need multiple implementations -you have an implementation that everyone uses. The UC Berkeley TCP stack is the best example of this. For a long time it was the TCP/IP specification.