OOXML

“Politics aside, there are 400 million users of the Office Open format, and we basically just recognized reality.” This quote by the retired Secretary General of Ecma, Jan van den Beld, explaining why it is so important to standardize OOXML.

Anyone else want to recognize reality? Maybe I can help.

Two questions to consider: 1) What is the actual state of OOXML adoption? and 2) What influence should market adoption of a technology have on its standardization?

On the first question, we should note that the 400 million users figure quoted by vdBeld in no way concerns OOXML. That figure is merely Microsoft’s estimate of the total number of Microsoft Office users, of all versions, world wide. Only a small percentage of them are using OOXML.

Let’s see if we can estimate the number.

How are Office 2007 sales? One (leaked) estimate (in September) was 70 million. But a follow-up statement makes it clear this is total Office licenses sold, of all versions. This is probably on the high end, not indicating installations, or even real end sales, since Microsoft typically reports sales into the channel. So that number must be reduced by some factor to account for real installations.

What percentage of Office users are running Office 2007? Joe Wilcox quotes Gartner, saying “Our Symposium survey showed Office at greater than 10 percent installed base…”

And not every Office 2007 will use the default OOXML formats. I’ve heard that corporate installations are often choosing to change their configuration to default to Compatibility Mode, so that Office 2007 saves in the legacy binary formats, for the increased interoperability this offers.

How does this net out? Something more than 40 million and less than 70 million seems the right neighborhood.

Let’s look for some more data points.

Take the example of OpenOffice, which has has seen over 100 million downloads, not including copies which are included already with Linux distributions. So I believe there are far more OpenOffice users than Office 2007 users. Of course, not all OpenOffice users save in ODF format. Some will change the defaults to use the legacy Microsoft binary formats.

Let’s take a look at an updated version of a chart I made back in May, with data now current through 11/27/2007.

The data here shows the number of documents reported by Google over time for ODF and OOXML documents. Hollow circles are ODF data points; solid circles are OOXML data points. (Yes, I need to figure out how to do scatterplot legends in R) The X-axis does not show the date. That would not be fair, since ODF had a significant head start in standardization and adoption. So in order have a fair comparison, both formats are shown against to the number of “days since standardization”, which is May 1st, 2005 for ODF, and December 7th, 2006 for OOXML, the days the formats were approved by OASIS and Ecma respectively.

Next week is the one year anniversary of Ecma’s approval of OOXML as an Ecma Standard. The news is not good. There are fewer than 2,000 OOXML documents on the entire internet (as indexed by Google at least) and the trend is flat.

What about ODF? Almost 160,000 and growing strongly.

Now we shouldn’t be so careless as to say that there are only 2,000 OOXML document in existence, or for that matter only 160,000 ODF documents. Not all documents are posted on the web. In fact, most of them are sitting on hard drives, in mail files, behind corporate firewalls, etc. The documents that Google sees is only a sampling of real-world documents. But this is true of both ODF and OOXML. My hard drive is loaded with ODF documents that are not included in the above sampling. But however you spin it, the minuscule number of OOXML documents and their pathetic growth rate should be a cause of concern and distress for Microsoft.

Where are all the OOXML documents? What governments have adopted OOXML? What agencies? What major companies? If there was an adoption bigger than a Cub Scout pack we would have heard it trumpeted all over the headlines. Listen. Do you hear anything? No. The silence speaks volumes.

But for sake of argument, what if the numbers were different? What if there were millions of documents on the web in OOXML format? Would that have any relevance to the JTC1 standardization process? The answer is a clear “No”. Market share, or even market domination, is not a criterion. In the US NB, INCITS, we are required to make our decision based on “objective technical factors”. Making a decision to favor a proposed standard because of the proposer’s market share would bring antitrust risks.

Consider this: In JTC1 we vote. One country one vote. We do not vote based on a nation’s GDP. Jamaica and Japan are equal in ISO. We have engineers review the standards. We do not bring in accountants to review financial statements and verify inventories. If we want to make decisions based on market share then we should scrap JTC1 altogether and hand standardization over to revenue department authorities to administer.

But that would then perpetuate a technological neo-colonialism where the developed world controls the the patents, the capital and the standards, and the rest of the world licenses, pays and obeys. There’s the rub. Where standards are open, consensually developed in a transparent process and made available to all to freely implement, there we lower barriers to implementation, level the playing field and allow all nations of the world to compete based on their native genius. But where standards are bought we end up with bad standards and a worse world for it.

Although Microsoft publicly testifies from every available pulpit of their deep longing for multiple document formats, a quick glance at reality shows that this love remains unrequited in their products. For example, what new formats does Office 2007 include out of the box? A new Microsoft XML format (OOXML), an updated Microsoft binary format, and a different new Microsoft binary format for Excel. So Microsoft clearly loves multiple Microsoft document formats! (Discuss among yourselves whether this love is amour de soi or amour propre.) But what about other, standard formats? ODF support is available only as a separate download, in their ODF Add-in for Word. However this tool is very poorly integrated into the Office user interface, making it almost impossible to use for real work.

For document exchange between different versions of MS Office, on the surface it looks a little bit better. Office 2007 provides a “compatibility mode” for users of Office 2007 who wish to create or edit documents that will remain compatible with earlier versions of Office.

That’s the theory at least.

In practice, things are rather messy. I recently received an email from Julie Watson, a project manager who has been doing enterprise deployments & migrations for 15 years. She has spent the last few months working on a plan to migrate 18,000+ workstations, trying to find a way to have a gradual rollout while still maintaining round-trip collaboration between her Office 2003 and Office 2007 users. Julie has put together a nice report showing what works and what doesn’t. Ignore the official documentation and ignore intuition, since neither will serve you well here. Take a gawk at the seedy side of reality in “[Compatibility Mode] Confusion in Office 2007.”

By now you have all seen the news. Andy Updegrove appears to have have the most coverage and analysis of that story, so I’ll try not to repeat stuff that you can easily read there.

The short of it is that DIS 29500 has failed in its attempt to be approved as an International Standard. The Microsoft spinmeisters are trying to make defeat sound like it is a good thing, that this is just the next step in the approval process.

Jason Matusow claims that “The next 6 months will be where the rubber really meets the road for the work on Open XML.” This is nonsense. The work should have been done back in Ecma, before submission to ISO. Fast Track is not a standards development process. It is intended for standards that are already completed and for which there is already industry consensus, to quickly transpose them into International Standards. Fast Track starts at the last stage, the Approval stage, of ISO’s 5 steps. By this point it is assumed that the text is complete, accurate, and has already been thoroughly reviewed. Since JTC1 NB’s have registered hundreds of technical flaws in OOXML, it is clear now that it never should have been put on Fast Track in the first place. The types of errors that are being reported now should have been found and fixed back at the committee draft stage or earlier, in Ecma. This defeat is an indictment of Ecma’s shoddy review. It is an abuse of ISO process for Microsoft to try to ram it through Fast Track in this state. They deserve the rebuke they have been given for this poor judgment.

Let’s drill into the numbers a bit and see what this all means.

First, recall JTC1’s two approval criteria for Fast Track submissions:

• 2/3 of JTC1 P-members must approve
• No more than 25% of total votes may be negative

In both cases those NB’s that Abstain or do not vote are ignored.

So, the ballot results for OOXML appear to be:

P-members: Approval: 17, Abstain: 9, Disapprove: 15. With only 53% Approval of P-members, DIS 29500 fails by the first criterion.

Overall vote: Approval: 51, Abstain: 18, No: 18. With 26% overall Disapproval, DIS 29500 also fails by the second criterion.

Microsoft highlights this second number (74%) in their press release, but does not even mention the P-member number. This is deceptive since even if they raised that number to 76% OOXML would have still failed. Only P-members can cause a Fast Track to be approved.

What is interesting is the large number of NB’s who participated in this process that have never participated in JTC1 before (at least to my knowledge). In fact, a number of them have such a strong interest in JTC1’s activities that they have joined as P-members — the highest level of participation — in some cases only in the last week or so. This is, I assume, what Tom Robertson, Microsoft’s GM of Interoperability and Standards, means when he talks of “rejuvinating” standards bodies:

Robertson dismissed criticism of Microsoft’s efforts to encourage its partners to join standards bodies. Most standards bodies are filled with “an old guard” membership that needs rejuvenation, he said. He also likened Microsoft’s recruitment efforts to a voter registration drive. “Have we been speaking to our community of companies about this issue? Yes, we have,” he said. “They needed to know. They, in many cases, decided to participate. [But] there is no basis to allegations that we are gerrymandering the process.

“Old Guard” NB’s appear to be those like Canada, France, New Zealand, Japan, Korea, Ireland, China or Norway that voted against OOXML. The new blood presumably are countries like Cote d’Ivoire, Syria, Kazakhstan and Tanzania that are participating in JTC1 for the first time, and voting in favor of OOXML.

JTC1 has historically had a rather stable membership of NB’s active in its technical agenda. There has been only a slow increase in membership, 1 NB joining in 2001, but none in 2002, 4 joining in 2003, 1 joining in 2004, none in 2005, 4 joining in 2006. But in 2007 JTC1 has been blessed with 12 new P-members, many of then joining in only the last week. There is a very clear trend in how these new P-members have voted:

In that table I’m defining “old guard” as those NB’s who were P-members of JTC1 before the OOXML process started. As you can see, the “old guard” voted overwhelmingly against OOXML by 2-to-1 margins. But the new P-members have almost all voted in favor of OOXML.

We can look at this graphically as well, showing the P-member composition of JTC1 over time and how they ultimately voted. As you see, JTC1 was overwhelmingly against OOXML until the blip at the very end, when Kazakhstan, etc. joined.

Another difference between the “old guard” and the “rejuvenated” membership is the level of public input and industry participation in their national committees. The old guard members had public forums, invited all sides to come in and speak, had all their stakeholders participate, reviewed technical comments and tried to come to a consensus. With openness like this, no wonder Microsoft believes they need rejuvenation! The new members, well… let’s just say the transparency of their decision making process is not uniformly great.

(6 Sept 2007 Update. I’ve removed the previous mention of CPI correlations to avoid confusion. The “transparency” of a national standards body’s process does not bear any necessary relationship with a country’s overall business climate. We’ve certainly seen in-depth, thorough technical reviews in the developing world, and we’ve seen suspect political dealings even in the United States. With the specific instances so damning to Microsoft, there is no need to make generalizations.)

I suppose that no one should be surprised that Microsoft, which has been stuffing committees at the national level throughout this ballot, would also attempt the same at the JTC1 level. From what I have been able to determine, NB’s, never having sat in a single JTC1 meeting and never having joined a single JTC1 technical committee, were able join as a P-members, in the last hours of the OOXML ballot, simply by sending an email to ISO.

Although this attempt to juice their results by signing up new P-members did not help Microsoft win approval for OOXML, it remains to be seen what adverse effect this will have on other JTC1 activities. We need to remember that a participation rate of 50% of JTC1 P-members is required to transact most JTC1 business. So this “rejuvenation” may very well paralyze JTC1 entirely unless the new members are earnest and participate in ballots beyond OOXML.

We’ve heard a lot of coverage of events in Sweden, Mexico, Australia, the US, etc. But we should remember that there are 150 or so countries eligible to vote. Here is a first-person account of the Microsoft medicine show in Ghana, from Kwasi at Ramblings of an African Geek:

So, the Q&A section rolls around, I asked some questions and an attempt was made by the MS reps to paint me as ill-informed and obtaining all my information from blogs on the internet run by anti-Microsoft fundamentalists. Oh, and of course IBM was mentioned as the prime company lobbying everyone and providing them with groundless reasons to vote against OOXML. Then came the best tactic of the day. Dismissing my questions as ‘too academic’ and ‘concerned with the needs of other nations, not Ghana’. After I stopped being annoyed at the attempt to shut me down, I was highly amused.

From Africa News is a report “African civil society warns Microsoft“:

(FOSS) Foundation for Africa (FOSSFA), Ms Nnenna Nwakanma, told HANA that Nigeria like any other African country stands to gain by properly investigating the issue on the ground, stressing that Microsoft lobbyists have not been able to convince stakeholders how the OOXML document formats would benefit the public except for those who have Office 2007, which is a proprietary software .

“Only those using Office 2007 can benefit from it. If you use any Office apart from 2007, you first have to upgrade. I cannot understand why norms cannot be used unless certain proprietary changes had to be made,” she said.

On the implication of voting ‘No’ to OOXML being proposed by Microsoft to Africa, especially in relation to e-School initiative, she said, already some African countries are warming up to embrace Open Document Formats (ODF), as an alternative file format.

But back to Sweden. My, my, what a mess. I suspect the same has happened elsewhere, including the US. But no one has been so careless as to leak a memo over here. We feel left out! So, if anyone has a similar “smoking gun” letter sent by Microsoft to line up MS Partners in the US to join INCITS V1 at the last minute, and doesn’t know what to do with it, you might consider letting me know. I’ll trade an original copy of the Utica Saturday Globe of Sept 21st, 1901, the President McKinley memorial issue, with full coverage of his funeral and burial, including a still brilliant page one color portrait (over the fold) of McKinley with Lady Liberty on the side, weeping, draped in flag with shield. Suitable for framing. A true collector’s item for any McKinley fan.

(Trivia: Ever wonder why there are so many McKinley High Schools in the US? Because so many of schools were built after soon after his death.)

So what is wrong with stacking a committee? Isn’t it just an expression of our freedom to associate? An interesting perspective from the Supreme Court, in a case that no one is talking about, but everyone should know: ALLIED TUBE & CONDUIT CORP. v. INDIAN HEAD, INC., 486 U.S. 492 (1988). This appears to be the highest profile case involving stuffing a standards committee:

Petitioner…can, with full antitrust immunity, engage in concerted efforts to influence those governments through direct lobbying, publicity campaigns, and other traditional avenues of political expression. To the extent state and local governments are more difficult to persuade through these other avenues, that no doubt reflects their preference for and confidence in the nonpartisan consensus process that petitioner has undermined. Petitioner remains free to take advantage of the forum provided by the standard-setting process by presenting and vigorously arguing accurate scientific evidence before a nonpartisan private standard-setting body. And petitioner can avoid the strictures of the private standard-setting process by attempting to influence legislatures through other forums.

What petitioner may not do (without exposing itself to possible antitrust liability for direct injuries) is bias the process by, as in this case, stacking the private standard-setting body with decisionmakers sharing their economic interest in restraining competition.

(Over on Slashdot one reader says of the above, “And I don’t think normal people go around reading and quoting 20 year old anti-trust cases for fun.” You don’t know me very well, do you? I read legal analysis for fun. I have my own copy of Tribe’s “American Constitutional Law”, a facsimile edition of Blackstone’s “Commentaries on the Laws of England”, a three volume set of the writings of Edward Coke, and Fergus Kelly’s “A Guide to Early Irish Law”. Never confuse me with normal. But never confuse me for a lawyer either. I don’t generalize well.)

In the “When Your Mom is the Beauty Pageant Judge” department comes news that the most influential “products, applications or technologies of the past 25 years”, according to a super duper scientific poll by CompTIA, is Internet Explorer. Second place is Microsoft Word. Third place is Microsoft Excel. And tied for Fourth Place is Windows 95.

Joe Wilcox over at Microsoft Watch takes a pin to the Microsoft-sponsored puff piece IDC did on OOXML called “Adoption of Document Standards.” And if the data is not rosy enough, Microosft can make it look even better by cutting off the y-axis labels to make a more impressive bar chart. “This one goes to eleven.” You could spend hours exposing the flaws in that paper, but why bother? Life is too short.

Wait… this just in. In a survey of most dumb-ass Microsoft-sponsored surveys of August, first place goes to CompTIA’s “Microsoft, Creator of Civilization, Inventor of Fire & Universal Benefactor of Mankind” and second place goes to IDC’s “4% Looks More Important in a Bar Chart if the Maximum is set to 5%.”

This one brought a smile to my face. Software Engineer job postings at Red Hat in Pune. Resumes must be submitted in ODF format.

Freecode in Norway has link to an an essay [pdf] by Sun’s XML Architect, Jon Bosak entitled “Why OOXML Is Not Ready for Prime Time”. Although I may disagree with Jon on the suitability of this single-vendor format for international standardization (His position is more along the lines of “Not yet” while mine is more like “Hell no”), I must admit he makes some excellent points.

Also, the Linux Foundations Desktop Architects have a statement just out: OOXML – vote “No, with comments“

And speaking of “No, with comments”, now that the Microsoft checks have presumably cleared, self-proclaimed “standards activist” Rick Jelliffe, is recommending that Australia vote “No, with comments.” This after a summer of speaking in favor of OOXML in India, Thailand (twice), Australia, New Zealand and who knows where else. How unfortunate for us all that his sage advice comes only after Standards Australia and most other countries have already finished their deliberations. I can only respond with the words of Lord Byron, from his “Ode to Napoleon”:

And she, proud Austria’s mournful flower,
Thy still imperial bride;
How bears her breast the torturing hour?
Still clings she to thy side ?
Must she too bend, must she too share
Thy late repentance

In the following post Rick reinterprets time, gives sophistry a bad name and takes such liberties with geometry that would make M.C. Escher blush, all in attempt to show that OOXML is really not 6,000 pages long, and it really wasn’t created in less than a year. You can read his attempt to seek a logical basis for redefining reality to fit his preconceptions here, or just consult my one-slide summary below.

(Oh, Rick. One more thing. My last name is shared by Australia’s most famous film director, Peter Weir. I manage to spell your name right. Maybe this mnemonic will help you spell mine right.)

Microsoft’s Stephen McGibbon updates us on the OOXML Ballot Resolution Meeting (BRM), now scheduled for February 25-29 in Geneva. He ends his otherwise informative post with a little jab:

I hear that IBM is still telling national bodies that a BRM isn’t guaranteed. I am unsure how IBM reached that conclusion but this seems to be concrete evidence to the contrary.

Well, let me help refresh Mr. McGibbon’s seemingly repressed memories.

First, scheduling a BRM does not guarantee it will be held. For example, have you heard of DIS 26926 “C++/CLI”? It was another Microsoft/Ecma Fast Track, just last year. The BRM meeting announcement went out on 25 October 2006, saying the BRM would be held 13-15 April 2007 in Oxford, England. Stephen, do you recall that BRM by any chance? Of course not, because it was canceled in February 2007 with the following message from the SC22 Secretariat:

We have been advised that the comments accompanying the Fast Track ballot for DIS 26926 are not resolvable and that holding a Ballot Resolution Meeting (BRM) would not be productive or result in a document that would be acceptable to the JTC 1 National Bodies. Therefore, our proposal is to not hold the BRM and to cancel the project.

So there is one example of a BRM that was scheduled and then canceled.

Want another? Sure, I can do that.

Take the case of DIS 26300 “Open Document Format.” A Ballot Resolution Meeting was scheduled for May 29 to June 1, 2006 in Seoul, Korea, concurrently with the JTC1/SC34 Plenary. But was the BRM actually held? No. It was canceled by the Plenary:

Following the advice of the JTC 1 Secretariat, JTC 1/SC 34 cancels the previously-scheduled ISO/IEC 26300 Ballot Resolution Meeting and the SC34 Secretariat will forward the revised DIS text and accompanying disposition to SC34 national bodies for a 30-day default ballot when ready.

Why? Because ODF received no Disapproval votes. Although 8 of the 23 NB submitted comments with their ballot, these were all “Approval, with Comments” votes rather rather than “Disapproval, with comments. So a BRM was not deemed necessary. Only comments that accompany Disapproval votes must be addressed at a BRM.

So there you go, two examples of BRM’s that were scheduled, but then canceled. The SC Secretariat has some discretion here. JTC1 Directives, Section 13.5 says, “In some cases the establishment of a ballot resolution group is unnecessary and the SC Secretariat can assign the task directly to the Project Editor.” The two examples given show that if a ballot passes by large margins, or fails by large margins, a BRM may not be necessary.

How about another example from the recent past, the Fast Track DIS 29361 “Information technology – Basic profile.” Their ballot closed on June 18th. Its ballot passed with 17 of 20 P-Members voting in favor of it. All Disapproval votes were accompanied by comments, as did one of the approval votes. Since there were Disapproval votes surely there must have been a BRM, right? No, that’s not how it worked. The JTC1 Secretariat decided a BRM was not necessary and the comments could be forwarded directly to the Submitter of the Fast Track for them to “review and respond”. So even having Disapproval votes does not guarantee a BRM will be held.

Does this make more sense now?

Of course, Microsoft already knows all this, and no doubt that is why they are working so hard to urge NB’s to vote “Approval, with comments” with promises that their comments will be addressed at the BRM, a BRM that might not even occur. In fact, if everyone listened to Microsoft and followed their advice then that would almost guarantee that no BRM would be held and no NB’s comments would be adopted.