ODF

Some Q&A, in the form of a self-interview. As with anything on this blog, these are my opinions.

Question: How are you doing today, Rob?

Rob: Very well thank you. I just finished attending a good set of working group (WG) meetings, and the Plenary meeting of ISO/IEC JTC1/SC34 in Seattle, Washington.

Question: Anything newsworthy to report?

Rob: Most of the meeting was routine, moving various documents forward, inch by inch, at the glacial pace we expect from ISO. But one noteworthy event this week was the first meeting of SC34/Ad Hoc Group 3.

Question: What is Ad Hoc Group 3?

Rob: Ad Hoc Group 3, or “AHG3,” is an temporary working group that was charged with determining how SC34 would fulfill its maintenance obligations relating to ISO/IEC 26300, the ISO/IEC transposition of the ODF standard. Francis Cave, of the UK, presided over the AHG3 meeting, which was well attended by 20 or so delegates.

Question: What do you mean by “maintenance”?

Rob: In this context maintenance refers to the activities that occur between major revisions of a standard, primarily the collection of defects, and the approval and publication of corrections.

Question: So what did Ad Hoc Group 3 do?

Rob: AGH3 met for around 8 hours over two days. Its most notable act was to orchestrate its own dissolution, by proposing that a new, permanent working group, WG 6, be created by SC34 to carry out pretty much the same charge given to AHG3. WG6 will likely also have the same Convenor and participants as AHG3.

Other than that, the primary benefit of the AHG3 meeting was to bring everyone up to speed on the current state of various ODF defect reports and errata documents, and to discuss ways in which we can better align the ISO/IEC version of ODF and keep it in alignment, with the OASIS version.

Question: So who owns ODF maintenance?

Rob: The OASIS ODF TC owns the maintenance of the OASIS ODF standard, and WG6 will own this activity for the equivalent ISO/IEC text. However, neither committee has absolute freedom of action, both being governed by applicable procedural rules of their parent organizations, as well as various joint agreements between OASIS and JTC1.

Question: Won’t this lead to a divergence of the OASIS and ISO/IEC versions?

Rob: It is a stated goal of JTC1 that ISO/IEC 26300, as the PAS transposed version of ODF, remain in sync with the OASIS version of ODF. This goal is shared with OASIS. OASIS and JTC1 have jointly approved a set of maintenance principles which call for the OASIS ODF TC to take the lead on maintenance.

Although we have not worked out all the details, a natural way to avoid divergence is to treat this as a two-phase process:

• Defect reports from SC34 are submitted to the OASIS ODF TC
• Corrections originate in the OASIS ODF TC, and are then transmitted to SC34

Question: But that doesn’t guarantee that these texts will not diverge, right?

Rob: At the very least there will be a delay between the correction of the OASIS version of the text and the publication of the ISO/IEC version of the correction. This delay will typically be around 5 months. If both OASIS and SC34 adhere the agreed maintenance principles, any divergence should amount to no more than such a delay.

Note that we already have some degree of divergence. For example, OASIS published ODF 1.0 Approved Errata 1.0 in November 2008, and submitted these changes to SC34 where they have not been acted on. And OASIS published ODF 1.1 back in early 2007 and has failed to submit these changes to SC34. Although there appears to be no adverse market effects from this degree of divergence, both sides agree that this is not an ideal situation and have agreed in principle to take steps to remedy this situation.

Also, we shouldn’t worry about the possibility of intentional or malicious divergence, since careful consideration of the ownership of the copyright and trademark for the ODF standard, as well as terms governing any associated patent claims, suggests that any rogue attempt to fork the ODF standard would be fraught with peril. ISO/IEC 26300 may lag OASIS ODF, but it cannot go beyond the corresponding OASIS text in any substantive way.

Question: So what role does WG6 have then?

Rob: SC34 has several WGs, each dealing with a different standard or group of standards. This WG structure is important for National Bodies, which often allocate their resources (personnel, travel budgets, etc.) at the level of a WG. It is also a logistic convenience for scheduling meetings.

WG6’s charter sounds impressive, “All SC 34 projects and activities relating to the maintenance of ISO/IEC 26300 OpenDocument Format”. However, since JTC1 has already assigned coordination of ODF maintenance to the OASIS ODF TC, it is not clear what, if any, residual tasks remain for WG6 to perform. But even if its official responsibilities are minor, WG6 does serve a useful purpose as a focal point for SC34 technical experts having an interest in ODF. Under the aegis of WG6, these experts might, for example, create defect reports for submission to OASIS and give feedback on OASIS draft corrigenda.

Question: But what about contributions in the other direction? How do SC34 technical experts contribute to ODF?

Rob: As stated in the joint memorandum of principles between OASIS and JTC1: “National Body input, including but not limited to the submission of defects and amendments, can best be achieved by the participation of JTC 1 experts in the ODF TC of OASIS”.

The exact answer will depend on your specific goal. If you participate via SC34, your opinion will count as one vote on your NB delegation, and in turn your delegation will have one vote among many NBs in SC34, which amounts to one source of feedback, among many others, to the OASIS ODF TC. So essentially you will have a much diluted and rather indirect way of expressing your thoughts on the technical work of the OASIS ODF TC.

Or you could join OASIS and and the ODF TC directly, as an individual member, in which case you can contribute directly in our work and your vote will be equal to mine or any other member of the ODF TC.

So if your priority is to present a national view via national body participation, then by all means participate via SC34 and WG6. This is also good if your priority is racking up frequent flier miles via attendance at international meetings. But if you are looking for the most effective way to contribute directly to the standard, then you cannot do better than joining the OASIS ODF TC. This is probably why nine members of JTC1/SC34 already are OASIS ODF TC members.

Question: So we’ve talked so far about maintenance. But what about major revisions, such as ODF 1.2, how are they handled?

Rob: Major revisions developed in OASIS may be submitted to JTC1 via the PAS transposition process. This results in a JTC1 ballot and does not directly involve SC34 technical experts, although in some cases the SC34 national “mirror committee” may be consulted by their NBs in the preparation of a national position on the PAS ballot. In any case, WG6 would have no formal role in this process, nor can I imagine an informal role for them.

I’m pleased to report that the 0.7 release of the ODF Toolkit Union’s ODFDOM library has just been released. This is an open source (Apache 2.0 license) Java toolkit for programmatically reading, writing and manipulating ODF documents. The code is 100% Java and does not require that you have OpenOffice or any other ODF editor installed. It operates directly on the document itself.

You can download the JAR and JavaDoc from the distribution here. You will want to go through J. David Eisenberg’s tutorials on ODFDOM as well.

If you sign up for a Toolkit Union account, you’ll be able to participate in the user’s mailing list (users@odfdom.odftoolkit.org) where we welcome your ideas, bug reports and patches. Or better yet, move over to the dev list (where the real fun is) and contribute actively!

The ODF Toolkit also has an active “Conformance Tools” project, including the ODF Validator, and our most-recent project, AODL, which is a .NET/C# module.

At the Toolkit Union we’re able to host more such ODF-related toolkit projects, in other programming languages. I don’t see any good reason to have two Java API’s for ODF, but I’d love to see broader coverage, especially in some of the more-widely used scripting languages like Python or Perl. I have even seen some interest in a PHP/ODF module. I think there is some advantage to getting a “critical mass” of programmers interested in working with ODF together in one place to bounce ideas off of each other. So if you are interested in joining this effort, sign up at the Toolkit Union’s web site, or send me a note if you want to talk first.

[22 July 2009 — more on ODFDOM 0.7 in this post by project lead Svante Schubert]

Although the term may be alien to some, “plugfests” have been around for around 20 years. A plugfest is when implementors of the same interface get together and test the interoperability of their products. In the beginning this was done with wired standards, USB, etc. (thus ‘plug’). Over the years the term was applied to networking at all higher levels of the protocol stack. The concept is also applicable to document exchange formats like ODF.

We had an ODF Plugfest last week in the Hague. Although we’ve had interoperability workshops and camps before that attracted a handful of vendors, this was the first one that had nearly universal participation from ODF vendors. I’m not going to recap the details of the plugfest. Others have done that already. But I will share with you some of my conclusions, based on long discussions with other participants, from whose insights I have greatly benefited.

In an ideal world, specifications would be perfect and software applications would be bug-free and users would read the manuals and we would achieve perfect interoperability instantly by anointment of the salubrious unction of standardization. But to the extent this planet’s population obdurately persists in imperfection, we are resigned to make additional efforts in pursuit of interoperability. We are not alone in this regard. The only standards that don’t need to work on interoperability are those standards that no one implements.

We should use every licit technique at our disposal to give the user the best experience with ODF we can. In a competitive market you can not get away with telling your customer, “Sorry, your spreadsheet doesn’t work because page 652, clause 23 says ‘should’ rather than ‘shall'”. If you did that you would not have that customer for long. (Unless, of course, you have a monopoly, in which case many seemingly irrational, anti-consumer actions can occur, seemingly without consequences.)

Further, I assert:

1. Users want real-world interoperability, and not excuses
2. Real-world interoperability is what users see and achieve in practice
3. Where vendors have the will to interoperate, achieving interoperability is a known technical problem, with known engineering solutions, but where the will to interoperate is lacking, there are no technical means of compelling interoperability
4. Interoperability lies at the intersection of technology, engineering standards, competition law, intellectual property and economics. There are no silver bullets, although there are a arsenal of proven techniques that can help to improve interoperability
5. Achieving interoperability is facilitated by a variety of cooperative activities, including standardization, test case creation, implementation testing, online validators, plugfests, defect collection and reporting

Going forward there is a promising constellation of efforts converging around ODF interoperability:

• The OASIS ODF Interoperability and Conformance TC, charged with creating an ODF test suite
• The OASIS ODF TC, finishing up work on ODF 1.2
• OfficeShots.org, providing a way to test the interoperability of a document in many ODF editors
• The ODF Toolkit Union, especially their open source ODF Validator
• The Plugfest participants, who continue to add information and test scenarios to the plugfest’s wiki.
• Groups such as OpenDoc Society and OpenForum Europe which lend their organizational skills and enthusiasm to the effort, and often much more.

So, we’re moving in the right direction. The key thing will be to sustain the momentum from the Plugfest and transition it into an ongoing effort, a Perpetual and Virtual Plugfest where the effort and the progress is continuous.

[6/29/09: I’ve received some emails on the photo, so here are the details:

The picture was taken at 3:30PM on the 2nd day of the workshop.

The lens was a Pentax DA 10-17mm “fisheye” zoom at 10mm. So that explains the projection distortion. The graininess and B&W was from post-processing using Nik Software’s Silver Efex Pro and Sharpener Pro.]

I used a variation of this chart at the recent ODF Plugfest in the Netherlands. But the aspect ratio of a presentation slide doesn’t suit this type of chart well, so here is a fuller version of what I showed there.

Those who are not familiar with standards development are sometimes amazed at how long it takes to develop a good standard. Perhaps the single-vendor, 6,000 page, 12-month escapade of OOXML in Ecma has skewed expectations. Fortunately, OOXML is the exception, not the rule. Achieving a multi-vendor consensus around a substantial technical standard will always be time-consuming, but it is time that is well spent.

OASIS standards go through several stages of development:

1. Working Draft (WD)
2. Committee Draft (CD)
3. Public Review Draft
4. Committee Specification
5. OASIS Standard

Progressing from one step to another is by ballot. The first 4 stages are advanced by vote of the Technical Committee (TC), while the last stage (OASIS Standard) is by a ballot of all OASIS members. As a draft advances through stages 1-4, an increasing degree of consensus is required. So, a CD requires only simple majority, whereas a Committee Specification requires 2/3 approval, with no more than 1/4 disapproval. Some of these stages allow iteration. So we can, and typically do, have several WD’s and several CD’s.

If you want more detail on the nitty-gritty details, here is a flow chart of the OASIS standards approval process.

I occasionally get a question along the lines of: “What has the ODF TC been doing for the past couple of years?” The following timeline should give you an idea. I’ve indicated the time spent developing ODF 1.0 and ODF 1.1, along with some other milestone activities, such as the PAS transposition of ISO/IEC 26300, the publication of ODF 1.0 Approved Errata 01 and the creation of the various ODF subcommittees. I’ve also indicated the dates of each of the ODF 1.2 WD’s and CD’s.

As you can see, we’ve been quite busy. After iterating on WD’s during 2007 and 2008, we’ve now moved on to CD’s. This is not a drawn out process, but simply the ODF TC working with full transparency, making all of the intermediate drafts available for public inspection.

All of the planned feature work for ODF 1.2 is now completed. The remaining work is to address the various editorial and technical comments that have been submitted to our comment list, as well comments from TC members and JTC1/SC34. The goal is to have no known defects in ODF 1.2 before we send it out for a Public Review. Of course, previously-unknown defects will likely be identified during the Public Review, and we have a process for handling these. I’ll comment more on that process, and Public Reviews in general, when we get closer to that stage.

There is an interesting disinformation campaign being waged against ODF. You won’t see this FUD splattered across the front pages of blogs or press releases. It is the kind of stuff that is spread by email and whispers, and you or I rarely will see it in the light of day. But occasionally some of it does cross my desk, and I’d like to share with you some recent examples.

First up is this instance, from a small Baltic republic, where a rather large US-based software company was recently arguing to the national standards committee for the adoption of OOXML instead of ODF. Here are some of the points made by this large company in a letter:

There is no software that currently implements ODF as approved by the ISO

(They then link to Alex Brown’s comment from Wikipedia). I think this demonstrates the triangle-trade relationship among Microsoft, Alex Brown (and other bloggers) and Wikipedia, by which Microsoft FUD is laundered via intermediaries to Wikipedia for later reference as newly minted “facts”. No wonder one of Microsoft’s first actions during their OOXML push was to seize control of the Wikipedia articles on ODF and OOXML via paid consultants. In any case, Alex’s claims were rebutted long ago.

ODF has a number (more than a hundred) of technical flaws which haven’t been addressed for 3 years despite change requests addressed to OASIS by countries such as Japan and United Kingdom. There are discussions between OASIS and ISO/IEC JTC 1 SC 34 regarding true ownership of ISO ODF, which is a reason why the flaws in ISO ODF aren’t being addressed. In a recent SC 34 meeting in Prague a new ISO ODF maintenance committee has been formed because ISO / IEC 26300: 2006 is not being presently maintained.

This is not true. First, the ODF TC has received zero defect reports from any ISO/IEC national body other than Japan. Second, we responded to the Japanese defect report last November. Amazingly, Alex Brown is implicated in this FUD one as well. It was false then and it is false now. At the exact time Alex was quoted in the press as saying the the ODF TC was not acting on defect reports (October 8th, 2008), we had in fact already sent our response to the defect report out to public review (August 7th, 2008) and then completed that reivew (August 22nd), after quite a bit of active technical discussion with the submitter of the original defect report (Murata Mokoto). How Alex translated that into “Their defect reports are being shelved” and “Oasis has not been acting on reports of defects” is beyond me. It must be particularly embarrassing that Murata-san wrote to the OASIS list, within days of Alex’s FUD, “I am happy with the way that the errata has been prepared.” How could Alex be ignorant of these facts? Why was he lying to the press? How is this conformant with his leadership role in JTC1/SC34 and his participation in BSI? Also observe the triangle-trade route of FUD in this case from Alex to Doug Mahugh to Wikipedia, this time for negative edits in the OASIS article.

IBM currently recommends not using OASIS ODF 1.1 and to instead use OASIS ODF 1.2 which is currently not complete and will not be complete and ISO certified before 2010/2011. OASIS on the other hand have started work on ODF 2.0 which will not be backwards compatible.

This is an odd one, demonstrably false. IBM Lotus Symphony supports ODF 1.1. We have no ODF 1.2 support at present. I wonder where they came up with this one? It is totally bizarre. Although we have started to gather requirements for “ODF-Next”, the contents of that version, and to what degree it will be backwards compatible, has not even been discussed by the TC, let alone determined. So this is pure FUD, trying to make ODF sound risky to adopt, and then lying about IBM’s support for it, and our position on ODF 1.2.

The list goes on, including claims that no one supports ODF 1.0 or ODF 1.1, etc., but you get the gist of it. The particulars are interesting, of course, but more so the reckless disregard for the truth, and the triangle-trade relationship between notable bloggers, Wikipedia, and Microsoft’s whisper campaign.

Another current example is part of Microsoft’s attempt to duck and cover from criticism over their interoperability-busting ODF support in Office 2007 SP2. I’ve heard variations on the following from three different people in three different countries, including from government officials. So it is getting around. It goes something like this:

We (Microsoft) wanted to be more interoperable with ODF. In fact we submitted 15 proposals to the ODF TC to improve interoperability, but IBM and Sun voted them down.

Nice story, but not true. Certainly Microsoft submitted 15 proposals. But they were never voted on by the TC, because Microsoft chose not to advance them for a vote. They opted not to have these proposals considered for ODF 1.2. It was their choice alone and their decision alone not to put these items up for a vote. I would have been fine with whatever decision Microsoft wanted to make in this situation. I’m not criticizing their decision. I’m just saying we need to be clear that the outcome was entirely due to their decision, and not to blame IBM or Sun for Microsoft’s choice in this matter.

I think I can trace this FUD back to a May 13th blog post from Doug Mahugh where he wrote:

We then continued submitting proposed solutions to specific interoperability issues, and by the time proposals for ODF 1.2 were cut off in December, we had submitted 15 proposals for consideration. The TC voted on what to include in version 1.2, and none of the proposals we had submitted made it into ODF 1.2.

This certainly is an interesting statement. There is nothing I can point to that is false here. Everything here is 100% accurate. However, it seems to be reckless in how it neglects the most relevant facts, namely that the proposals did not make it into ODF 1.2 at Microsoft’s sole election. It is as if Lee Harvey Oswald had written a note: “Went to Dallas and saw a parade today. Tried to see a movie, but had to leave early. Heard later on the radio that the President was shot”. This would have been 100% accurate as well, but not the “whole truth”. In any case, the rundown of the facts in this question are on the TC’s mailing list.

So what is one to do? You obviously can’t trust Wikipedia whatsoever in this area. This is unfortunate, since I am a big fan of Wikipedia. I want it to succeed. But since the day when Microsoft decided they needed to pay people to “improve” the ODF and OOXML articles, these articles have been a cesspool of FUD, spin and outright lies, seemingly manufactured for Microsoft’s re-use in their whisper campaign. My advice would be to seek out official information on the standards, from the relevant organizations, like OASIS, the chairs of the relevant committees, etc. Ask the questions in public places and seek a public, on-the-record response. More people are willing to lie than face the consequences of being caught lying. That is the ultimate weakness of lies. They cannot stand the light of public exposure. Sunlight is the best antiseptic.