Interoperability

This is a follow up to some work we did at the ODF Interoperability Workshop in Beijing last November. We had good participation there: IBM, Sun, Google, Novell and Redflag from the big vendor side, as well as a good number of users. It was a full-day workshop and we covered a number of topics. One of them was spreadsheet formulas. I gave a short presentation on spreadsheet interoperability, specifically on the work we’ve done on OpenFormula for ODF 1.2. We also did a short exercise to look for spreadsheet formula bugs.

As many of you know, neither ODF 1.0 nor ODF 1.1 defines a spreadsheet formula language. They leave it implementation-defined. The specification makes only a few broad statements, such as a recommendation that formula attributes be qualified by namespace, that formulas begin with ‘=’ , that cell addresses be surrounded by ‘[‘ and ‘]’ and that formula parameters be delimited by ‘;’. So in theory, this is a mess. But in practice it has worked out quite well, since implementations have played “follow the leader” and have nearly converged on interoperable spreadsheet formulas. With ODF 1.2, we’ll standardize the consensus on spreadsheet formulas, giving even greater certainties.

Let’s see how this works in practice. I created a simple spreadsheet document in several ODF-supporting applications, including Microsoft Office using the various plugins. Here is what I tested:

1. Microsoft Office 2003 with the Microsoft-sponsored CleverAge Add-in version 2.5
2. Google Spreadsheets
3. KOffice’s KSpread 1.6.3
4. Lotus Symphony 1.1
5. OpenOffice 2.4
6. Microsoft Office 2003 with Sun’s ODF Plugin

I used what I had installed on my two machines, Windows and Ubuntu. There may be updates to some of these applications that do even better.

I created the same basic spreadsheet from scratch in each editor and saved it as ODF format. I then looked at each document to see how formulas were being stored in the XML:

1. CleverAge stores it in the OpenOffice namespace (xmns:oooc=”http://openoffice.org/2004/calc”)
2. Google also uses the OpenOffice namespace.
3. KSpread doesn’t use namespace-qualified formula attributes.
4. Symphony also doesn’t use namespace-qualified formula attributes.
5. OpenOffice uses the OpenOffice namespace.
6. Sun’s Plugin also uses the OpenOffice namespace.

OK. So there is some variation in how the formulas are stored, with two approaches in use. How does this then impact interoperability? In theory it is horrible. In practice it works out pretty well.

I took each of the 6 spreadsheet documents and opened each one in each of the other 5 applications — 30 interoperability tests — to see whether the formulas were loaded and calculated correctly. Here is what I saw:

So the formulas came through OK, in almost all instances. The only exception was the CleverAge add-in, which failed to process formulas from KSpread and Symphony. For example, loading the Symphony spreadsheet into Office 2003 results in cells with contents containing errors such as “=#REF!+#REF!-#REF!” which is tantamount to data loss.

I think we can do better than this with a few simple changes.

The Law of Robustness as stated in RFC 1122 is “Be liberal in what you accept, and conservative in what you send.” Adapting that principle to ODF spreadsheets, I recommend the following practice for ensuring interoperability using ODF 1.0 and ODF 1.1:

1. When writing ODF 1.0 or ODF 1.1 spreadsheet documents, write formula attribute values using the OpenOffice namespace prefix: “http://openoffice.org/2004/calc”. All ODF spreadsheet applications I have tested accept and correctly process formulas in that namespace. Note that the CleverAge add-in is not doing the namespace checks in a XML-correct fashion. They are comparing only the text of the prefix, not resolving it to a namespace URI and comparing the URI’s. So you should be sure to also use “oooc” as the namespace prefix.
2. When reading ODF 1.0 or ODF 1.1 spreadsheet documents, be prepared to handle formulas with no namespace qualification as well as those with the OpenOffice namespace.

Specifically, Symphony and KSpread should consider making changes to accommodate #1 and CleverAge should consider changes needed to do #2. In the CleverAge case, a trivial, one-line change to OdfConditionalPostProcessor.cs will quickly restore compatibility with Symphony and KSpread documents.

Now, if you are entirely satisfied with what I have said above, and have no lingering doubts, then you are not thinking enough. It is not enough to merely bring the spreadsheet formulas over intact. Interoperability also requires that we interpret the formulas in the same way.

So let’s look at that side of the equation (no pun intended). Fortunately, we are all quite close to what is being defined in ODF 1.2’s OpenFormula specification. This is not so surprising, since OpenFormula was based on actual spreadsheet practice, looking at a variety of spreadsheet applications. I did a quick test of the 6 ODF spreadsheet applications to see how well they fared against a test suite of 509 core tests that OpenFormula defines for spreadsheet functions. The results were:

• CleverAge 455/509 = 89%
• Google 457/509 = 90%
• KSpread 472/509 = 93%
• Symphony 487/509 = 96%
• OpenOffice 493/509 = 97%
• Sun Plugin 500/509 – 98%

So, we’re not yet perfect, but we’re getting pretty close. Interestingly, the lowest scores (CleverAge) and highest scores (Sun Plugin) are both for the same calculation engine (Excel).

Looking forward, we’ll continue to edit and refine OpenFormula and its test cases. You might look for it when it comes out for public review, hopefully in a couple of months. Unlike other parts of ODF 1.2, OpenFormula is essentially XML-free. It is a mini-expression language, defined by a BNF grammar and accompanied by hundreds of spreadsheet functions from mathematics, finance, engineering, statistics, etc. So review by subject matter experts in these disciplines is especially needed, even if they have zero XML experience. If you want to see the current OpenFormula Working Draft, currently in its 71st revision, take a look. Comments may be submitted to the ODF TC’s comment list.

I’m also looking forward to testing Office 2007 SP2’s ODF support when it comes out, to see how their ODF support is improving. Anything less than the 500/509 results that Excel 2003 gives with the Sun Plugin will be a disappointment. KOffice has a 2.0 version in beta I should look at. OpenOffice has their 3.0 update. Sun also has an updated ODF Plugin. I’ll lean on the Symphony team as well, and see if we can beat 500/509. Game on!

A short tale, all true, to relate. The names have been changed to protect the guilty.

Years ago, but not so very long ago, when XML still had that new car smell, two companies, let’s call them Red and Blue, decided to make a new XML-based standard. This new standard would be, they claimed, a huge step forward and would increase interoperability, especially in complex heterogeneous environments, with multiple operating systems, multiple vendors and applications, etc. Their activities received much fanfare in the press. Everyone was pleased that Red and Blue were cooperating together to make this new standard.

This wonderful new standard was eventually completed, and Red and Blue both went and implemented the standard in two implementations which I’ll call RedLib and BlueLib. But when they tried running their RedLib and BlueLib implementations against each other, to demonstrate interoperability, it didn’t work. It was a total failure. There was zero interoperability.

So what did Red and Blue do? They realized that interoperability is not guaranteed merely by the existence of a standard. You also need high quality implementations, implementations that accurately and completely implement the standard. For any non-trivial standard, implementation errors will dominate the list of causes of interoperability problems. So Red and Blue worked together, with other vendors, to create an interoperability lab for the new standard, and created test suites to test interoperability, and held interoperability demonstrations at conferences, and tested and iterated on this until the implementations provided a high level of interoperability.

Today billions of dollars are transacted every day using this XML-based standard.

With ODF we find ourselves in a similar, though more complex, situation. There are more vendors involved than just Red and Blue. We are starting with many commercial and open source implementations. In some cases, with some editors, interoperability is quite good. In other cases it is rather poor. But when a user loads a document, which they may have downloaded on the web, or received via email, they have no idea where that document came from, what application, what operating system. And when you create an ODF document, you may not know who will eventually read it. It isn’t enough to have good interoperability between some ODF implementations. We need good interoperability among all ODF implementations.

From a technical perspective, this is a goal we all know how to achieve. It has been done over and over again throughout the history of technology standards, especially network standards. You develop test suites, you test your implementations against these test suites, you have interoperability workshops (or plug-fests as they are sometimes called). You iterate until you have a high level of interoperability.

For the past 6 months I’ve been talking to my peers at a number of ODF vendor companies, to fellow standards professionals in OASIS, to ODF adopters, as well as to people who have gone through interoperability efforts like this before. I’ve given a few presentations on ODF interoperability conferences and led a workshop on the topic. I led a 90-day mailing list discussion on the ODF interoperability. Generally, I’ve been trying to find the best place and set of activities needed to bring the interested parties together and achieve the high level of interoperability we all want to see with ODF.

The culmination of these efforts is the creation of a new Technical Committee in OASIS, called the ODF Interoperability and Conformance TC, or OIC TC for short. The official 30-day OASIS Call for Participation went out last Friday. You can read the full charter there, but you can get a good idea by just reading the “Scope of Work”:

Initially and periodically thereafter, to review the current state of conformance and interoperability among a number of ODF implementations; To produce reports on overall trends in conformance and interoperability that note areas of accomplishment as well as areas needing improvement, and to recommend prioritized activities for advancing the state of conformance and interoperability among ODF implementations in general without identifying or commenting on particular implementations;

To collect the provisions of the ODF standard, and of standards normatively referenced by the ODF standard, and to produce a comprehensive conformity assessment methodology specification which enumerates all collected provisions, as well as specific actions recommended to test each provision, including definition of preconditions, expected results, scoring and reporting;

To select a corpus of ODF interoperability test documents, such documents to be created by the OIC TC, or received as member or public contributions; To publish the ODF interoperability test corpus and promote its use in interoperability workshops and similar events;

To define profiles of ODF which will increase interoperability among implementations in the same vertical domain, for example, ODF/A for archiving;

To define profiles of ODF which will increase interoperability among implementations in the same horizontal domain, for example ODF Mobile for pervasive devices, or ODF Web for browser-based editors.

To provide feedback, where necessary, to the OASIS Open Document Format for Office Applications (OpenDocument) TC on changes to ODF that might improve interoperability;

To coordinate, in conjunction with the ODF Adoption TC, Interop Workshops and OASIS InterOp Demonstrations related to ODF;

To liaise on conformance and interoperability topics with other TC’s and bodies whose work is leveraged in present or future ODF specifications, and with committees dealing with conformance and interoperability in general.

We have a broad set of co-proposers of this new TC, representing ODF vendors, ODF adopters, private sector and government:

• Robert Weir, IBM
• Bart Hanssens, Individual
• Dennis E. Hamilton, Individual
• Zaheda Bhorat, Google
• Charles-H. Schulz, Ars Aperta
• Michael Brauer, Sun Microsystems
• Donald Harbison, IBM
• Alan Clark, Novell
• Jerry Smith, US Department of Defense
• Aslam Raffee, South Africa Department of Science and Technology

The OIC TC will have its first meeting, via teleconference, on October 22nd. At that point members will elect their chairman.

I’d like to see broader representation in this TC’s important work. In particular, I’d like to see:

1. Additional vendors that support ODF, such as Corel and Microsoft (and yes, before you ask, I have already extended a direct and person invitation to Doug Mahugh at Microsoft)
2. A representative from KOffice
3. A representative from the OpenDocument Fellowship, which has already done some work on an ODF test suite. Wouldn’t it be good to combine our efforts?
4. Representatives from non-desktop ODF implementations, e.g., web-based and device-based.
5. Broader geographic participation.
6. Participation with specialized skills to help define and review test cases in areas such as: Accessibility, East Asian languages, Bidi text, etc.
7. People with an interest in archiving, to help to define an ODF/A profile.

So, if you fall into one of those categories, I hope you’ll consider joining the new TC. Heck, even if you are outside of those categories you are welcome to join. The only prerequisite is that you are an OASIS member. OASIS membership is $300 for individuals, and for companies has a sliding scale according to company size. More information on OASIS membership is here.

We have a lot of work to do, but now we finally have a place where we can get the work done. This is big. This is important, both for ODF vendors and ODF users. I hope you’ll join us as we all work to improve interoperability among ODF implementations!

[Update: On 12 November Doug Mahugh accepted my invite and announced that Microsoft would join the TC.]

You must admire their tenacity. Gary Edwards, Sam Hiser, and Paul E. Merrell (aka “Marbux”) . The mythology of Silicon Valley is filled with stories of three guys and a garage founding great enterprises. And here we have three guys, and through blogs, interviews, and constant attendance at conferences, they have become some of the most-heard voices on ODF. Maybe it is partly due to the power of the name? The “OpenDocument Foundation” sounds so official. Although it has no official role in the ODF standard, this name opens doors. The ODF Alliance , the ODF Fellowship, the OASIS ODF TC, ODF Adoption TC (and many other groups without “ODF” in their name) have done far more to promote and improve ODF, yet the OpenDocument Foundation, Inc. seems to score the panel invites. Not bad for three guys without a garage.

However, in recent months the OpenDocument Foundation has found itself more and more isolated, outside of the mainstream debate. How far they have fallen can be seen in the fact that Microsoft has gone from ridiculing their conspiracy theories to using them to support their arguments. At the same time the Foundation’s membership has dwindled to the point where only a small number remain. Former members have disassociated themselves from the Foundation as it turned increasingly to strident rhetoric. Whereas in the early days, the Foundation had a large membership that participated fully in the OASIS TC’s, now their “contributions” are mainly that of heckling and haranguing the other members. Finally, the Foundation has recently announced its intent to abandon constructive work within OASIS, to actively lobby against adoption of ODF 1.2 in ISO and to push for an alternative format, CDF, based on XHTML, CSS 3.0 and RDF. This is an odd stance for a non-profit whose charter was:

The OpenDocument Foundation, Inc. is a 501c(3) non profit chartered to work in the public interest to support, promote and develop the OASIS OpenDocument File Format affectionately known as “ODf”.

So it is against this backdrop that I read with interest in Linux Today the latest correspondence from the Foundation. You can read it yourself, or take the following 8 points from me as a condensed summary of their main points:

1. “The commercialization of interoperability remains a key driver in both big vendor deals and big vendor consortia FOSS is left on the outside looking in.”
2. The conversion to XML [document formats] must be nondisruptive” meaning it fits into existing business processes which are increasingly dominated by Microsoft middleware. This implies a requirement for high-fidelity, loss-less round-trip conversions.
3. The alternative is “rip and replace” and that is too costly and disruptive.
4. Microsoft is moving toward a “grand convergence” of their services, desktop, device and servers, with OOXML at the core. “MS-OOXML is the primary transport, the document/data container of interop-integration preference.”
5. ODF was not designed as a response to these problems.
6. Microsoft/Sun/Novell are working “to limit ODF interoperability and usefulness” because of some patent deals. (Sorry I can’t summarize this one better — I just don’t understand it.)
7. IBM/Oracle/Google are working to “limit ODF interop” because “they want a total ripout and replace of MS Office.”
8. The Open Document Foundation is in “the middle area of trying to perfect the conversion to XML”.

Let me take these points one-by-one:

1. The OpenDocument Foundation seems to try to clothe themselves in the mantle of the open source community and pontificate on how the big bad vendors treat interoperability. But are they speaking as a non-profit or as a vendor? Take their DaVinci plugin, for example. Where is the source code? Why isn’t this open source? Are we to follow the Foundation’s claim of 100% interoperability, based on blind faith, without seeing some proof in the form of working code? I’ve been working on document conversions and document file formats of one kind or another for almost 20 years. I’ve never seen 100% fidelity conversions of anything but trivial formats. Extraordinary claims require extraordinary evidence. But we have nothing here, just white papers.
2. I would not claim a priori that all customers require lossless, 100% fidelity conversions. Remember, we do not see 100% fidelity even when upgrading from Office 2003 to Office 2007, but this appears to be adequate. What is required is that the total return from changing document formats exceeds any other profitable use of capital available to the enterprise. In other words, to a business this is an investment, and will be judged as an investment. Very few businesses will take a dogmatic, ideologically pure view of this. Ask yourself, would you accept 1% loss in fidelity if I gave you a billion dollars? Yes,of course you would. There are no purists in business who will remain in business. We’re just haggling over what price/fidelity combination is needed to make a prudent investment.However, there is a notable exception to this rule, and that is where access to open document formats are mandated as a public right, not as a business investment. Think of the last 20 years or so of enabling public buildings with ramps for the disabled, bathrooms to accommodate wheelchairs, braille lettering in elevators. This was done by legislation and regulation, as a matter of public policy, to ensure that all of the public has access to public facilities. There was no requirement that an access ramp post a net profit. Similarly, today we see some movements to ODF are based on open-access principles.
3. This is what we call the “fallacy of the excluded middle.” You are either with us, or against us, etc. It is false to suggest that the only two approaches to interoperability are to either blindly follow the OpenDocument Foundation’s mysterious DaVinci plugin, or to ignore interoperability altogether and advocate rip and replace. There are today two other other ODF plugins available, one from Microsoft and one from Sun. This is real, running code, open source even in the case of the first plugin. So why should we be taking exclusive direction from the Foundation on how we achieve interoperability? Oh right, they are claiming that their program achieves 100% round-trip fidelity. Extraordinary claims…
4. Gary is in the ballpark when he suspects that Microsoft is seeking some sort of “grand convergence” around protocols and formats. However, I disagree with his impression that OOXML sits at the center of this. In my opinion, OOXML is a rushed, transitional format, intended purely to disrupt ODF adoption. Just as the Office 2000, Office XP, and Office 2003 markup formats were abandoned by Microsoft, I predict that OOXML will soon be cast aside. The problem is that OOXML is such a poorly-engineered format that not even Microsoft wants to build upon this. If I had to divine the future of Microsoft’s file formats, I’d look more in the XAML/XPS/Silverlight space. I believe that future MS Office document formats will look more like that than like OOXML.
5. I find this observation amusing. ODF, which started its standards track late in 2002, was not designed to be 100% compatible with Office 2007. Mercy me, how did we manage to drop the ball on this one?! Remember, in 2002 there was no publicly available specification for Microsoft document formats. There was no Open Specification Promise or Covenant Not to Sue. So not only was 100% compatibility technically impossible, attempting it via reverse engineering was precarious from a legal standpoint. In my opinion, it still is, even in 2007.In any case I’m staunchly opposed to evolving any open standard purely for the benefit of a single vendor. Microsoft Internet Explorer is the dominate web browser. Should we then require that HTML only evolve in ways that improve interoperability with Internet Explorer? I don’t think so. Why should document formats be different?
6. This comment manages to avoid confronting a heap of contrary facts. Microsoft supports the open source ODF Translator project on SourceForge. Sun has made their own ODF Plugin 1.1 for MS Office available for download. And Novell, along with helping the Microsoft effort, has integrated that translator into their version of OpenOffice and has also started work on more powerful, next-generation support for OOXML. So these three companies are seeking to “limit ODF interoperability and usefulness”? If so, they sure have a clever way of disguising their intent. To the ordinary bystander, writing conversion and translation code to allow documents to be shared between OpenOffice and MS Office would be seen as a pro-interoperability statement. But thanks to the OpenDocument Foundation’s in-depth sleuthing, we now know that the opposite is true. Not!Although I have serious doubts as to long-term technical feasibility of some of these translation endeavors, they do have the advantage of showing real, running code working with real, running applications. They may not claim 100% fidelity, but this is first-generation work and will undoubtedly improve. But they have an important advantage over the Foundation’s DaVinci Plugin in that these other efforts demonstrably exist. Given a choice, I’ll take an open source version of a partial fidelity convertor, with a reasonable architecture, over one that claims 100% fidelity, but that I can’t see or touch.
7. The claim is that IBM/Google/Oracle also want to “limit ODF interop” because (according to Gary) we want rip & replace. Strange, but just a few weeks ago I lead an ODF Interoperability Camp in Barcelona, on behalf of the OASIS ODF Adoption TC, where we had a good selection of ODF vendors, open source projects and customers working to improve interoperability, including Sun, Novell, Google and IBM. The OpenDocument Foundation is a member of the OASIS ODF Adoption TC. So did they help in the organizing of the event? Did they participate? No, nothing, nada. Evidently it is easier to complain about interoperability than to do something about it.And again there is this fallacy of the excluded middle. You must either accept the magical DaVinci Plugin, or you are for rip & replace. There are no other alternatives considered. I’d remind the OpenDocument Foundation that interoperability was not invented yesterday, and that there are many technical approaches that can be applied to foster it. Open standards are one way, but there are others that can be applied as well, including conformance testing, test suites, plug-fests, profiles, shared code, reference implementations, etc. We should apply experience and engineering judgment to select the appropriate solution for the problem, and not fall into the trap of believing that there is only a single path to interoperability, and that this path just happens to be based on the Foundation’s product.
8. Although it sure would be nice to portray yourself as the little guy, watching out for the customer, while the big bad vendors tromp all over the flowers, the fact is that the big vendors are actively working on interoperability, with at least three major solutions available today, as well a major initiative around interoperability in the ODF Adoption TC. In particular, IBM (with SmartSuite) and Sun (with StarOffice) have 15 or so years experience each in working on document interoperability with MS Office. This isn’t rocket science, but neither is it easy. You can either stand on the sidelines and make pronouncements about how the world is out to prevent interoperability, or you can roll up your sleeves and help get the work done. I know which one I’ll be doing. What about you?

If the Foundation’s approach was technically feasible, they would just go out and do it. You don’t let a breakthrough technical innovation wait on a standards committee to act. You just go out and do it and then standardize it later, once you’ve proven it works. If the Foundation really thinks that they can achieve 100% interoperability with MS Office with just 5 simple changes to ODF, then why the heck don’t they just do it? Don’t wait for the formality of an the ODF TC ‘s approval. They should go ahead, as if the standard already had their 5 fixes, and show the world how they have achieved 100% interoperability with MS Office. If they are right, they would all become multi-millionaires in a very short period of time.

The OASIS ODF Adoption TC is organizing an ODF Camp to be held on September 20th in Barcelona, Spain. Facilities for this event are graciously provided by OpenOffice.org, which will be holding its annual conference concurrently.

The hope is that this will be the first of several such events to bring ODF vendors together to explore ways of greater technical coordination, especially in the area of interoperability. I’ve written about and presented on this topic before. Now is the time for action, and I’m extremely pleased that so many vendors will be attending.

On other occasions I’ve called interoperability “the price of success” because a standard implemented by only a single vendor and a single application need not worry about it. Only successful standards with many implementations need to rent a hall to bring the implementors together to review and perfect interoperability.

(It is like capital gains taxes. I grumble when I pay them, but take some solace in the fact that my investments were profitable. Those who make a losing investment don’t pay capital gains taxes on it.)

The focus of this first interoperability event will be on the ODF word processor format. Follow-up events will look at spreadsheets and presentations.

Please have a look at the detailed agenda for the camp and consider joining us in Barcelona.