Microsoft risks a repetitive stress injury from the recent frenzy of patting themselves on the back for responding to some of the ballot comments submitted in the failed OOXML ISO ballot of Sept 2nd.
They claim to be transparent and acting so that NB’s can easily review their progress in addressing their comments.
Well, let’s take a closer look.
First, Microsoft has managed to get JTC1 to clamp down on information. What was a transparent process is now mired in multiple levels of security leading to delay, denial of information to some NB participants and total opaqueness to the public.
Let’s review how things worked with ODF.
- OASIS ODF TC mailing list archives are public for anyone to read
- OASIS ODF TC public comment list archives are public for anyone to read
- OASIS ODC meeting minutes, for every one of our weekly teleconferences going back to 2002, are all public for anyone to read.
- The results of ODF’s ballot in ISO are public, including all of the NB comments
- The comments on ODF from SC34 members are also public
- The ISO Disposition of Comments report for ODF is also public for anyone to read
Short of allowing the public to read my mind, there is not much more we can do in OASIS to make the process more transparent. (And if you read this blog regularly you already have a good idea of what I’m thinking.)
But what about the OOXML process? Every single one of the above items is unavailable to the public, and in many cases cases is not available even to the JTC1 NB’s who are deciding OOXML’s fate.
In fact, OOXML is moving in reverse. Documents that were once public, such as the Sept 2nd. ballot results and NB comments, have been taken down and replaced with password-protected versions (Look for the DIS 29500 documents here. They all used to be available for all to download.) How do you get access to the password? The password is made available to NB points of contact “on request”. But so far few NB’s have requested the password. You can see here which ones have requested the password and which have not. As of today, only 18 of 51 NB’s have requested the password. Only 35% of SC34 NB’s have access to the same information they had back in September. Indeed, we’re moving backwards.
In the particular cases of these “662 responses”, Ecma is hosting them on their web site, on a different password protected page. (Yes, the comments and the resolutions to the comments are on two different web sites with two different passwords.) I’m hearing as well that few NB’s actually have the password, and some who do are not passing it on to their own committee members. I’ve heard from a few NB members who explicitly requested access to these documents but were denied. Others are simply unaware that these comment resolutions are available. What was once an open process is now closing up.
(12/04/2007 Update: Brian Jones claims that these 662 resolutions are protected by JTC1 rules. But JTC1 rules apply to documents submitted into the JTC1 process, hosted by JTC1 , assigned JTC1 “N” numbers, and archived by JTC1, as required by JTC1 process. But these 662 resolutions are not called for by the JTC1 process, are not hosted by JTC1, are not assigned JTC1 “N” numbers and are not archived by JTC1. They are Ecma documents, hosted by Ecma, assigned ID’s by Ecma, and controlled by Ecma passwords. These documents were never submitted to JTC1. Ecma is in total control over whether or not the public has access to them.
Brian highlights some rules that apply to the Disposition of Comments report, but that is not what we have before us. We won’t have the Disposition of Comments report until after the Ballot Resolution Meeting. At that point, it will be an official JTC1 document, assigned an “N” number, hosted by JTC1 and accessible via their password .
Note also that Microsoft continues to dodge how closed the Ecma TC45 process has been and remains. Why not open up the TC45 mailing list archives, Brian? Are the ISO meanies stopping you? I know that Ecma is not forcing you. Their policy is to let each TC decide for themselves. I’m sure if Microsoft took a leadership position in favor of openness that you could convince the other members of TC45 to increase their transparency. What do you say?)
(12/06/2007 Update: The former Ecma Secretary General weighs in on the topic in a blog post, confirming that the responses are not controlled by ISO access rules, though the original NB comments are:
Consequently, Ecma is not constrained in posting its interim responses on a publicly available page as long as they are not tied to specific NB comments. In other words, Ecma would have to do some work to separate the proposed responses from the specific NB comments, but then Ecma may make its work publicly visible. If there is so much interest outside the NB circuit, then Ecma will surely do something here.
.
.
.
Indeed, seen from Ecma there is nothing that forbids Ecma to distribute its proposals. But it should also be clear, in the light of the longstanding relationship, that it is not a MUST for Ecma to do this. Good habits and rules have a value, like in any great game, such as football. And also there the rules and habits don’t change overnight because somebody has another, maybe even brilliant idea
.
But suppose you get through your local NB politics and actually lay your hands on the password to the Ecma web site, what do you get then? You then have the privilege of navigating 50 or so different pages, scrolling through them and click on 662 links to download 662 separate PDF files, all from a painfully slow server. Ughh… It hardly seems worth it. It is almost like someone wants to discourage NB’s from actually reading this stuff.
Aiming to lessen the pain a little, I downloaded all 662 comments, and made a singe PDF file that contains all of the comment responses. I also included the original NB comments, and cross-linked everything, so I can navigate from comment to response, and slice and dice it by similar comments, or by NB. It is full text indexed, so I can search for things like “VML” and see all comments or responses relevant to that topic. Since it is liberated from the Ecma website, I can even use it off-line.
Doesn’t my method sound easier to use than downloading 662 PDF files? If you agree, then I’ll make you an offer. If you are a JTC1 or SC34 NB member, and would like access to this consolidated document, let me know via email. (You can find my email address here.) Note that my compilation is not a formal JTC1 document, and that this is not an offer from the US NB. This a personal offer from me to other individuals who are also JTC1 or SC34 NB members. (Of course, if Ecma wants a copy of this as well to make available for all NB’s to download, then that is even better. They know where to find me.)
So, now that I’ve read through these 662 responses, let me fill you in what we have here. First, I’d like to define some terms, so we’re all on the same page and understand the status of these 662 proposals.
At the BRM, baring any breakdown from lack of consensus, there will be issued an official “Resolution of Comments” document. This is the set of textual changes that JTC1 NB’s authorizes the Project Editor (Microsoft’s contractor in Ecma) to make to the DIS 29500 specification. Only the BRM can authorize these changes.
By January 14th, JTC1 NB’s will receive from the Project Editor a “Proposed Resolution of Comments” document. This will be Ecma’s proposals for how they would like to see the Sept. 2nd ballot comments resolved. The BRM is not limited to considering Ecma’s proposals. Their own NB comments from Sept 2nd may also be in play, since those often came with their own proposed resolutions which differ from the ones that Ecma will propose.
So what do we have now, these recent drop of 662 documents from Ecma? I call these by the verbose name: “Ecma’s Draft Proposed Resolution of Comments”. The are not the final Resolution of Comments, and they are not even the final Proposed Resolution of Comments. They are a draft of proposed resolutions to 662 of the 3,522 comments submitted by JTC1 on Sept 2nd.
So the time line is:
- From now until late January we receive updates from Ecma in the form of Draft Proposed Resolution of Comments. If they continue to be posted in a user-unfriendly form, I will continue to produce updates to my consolidated report.
- By January 14th, Ecma submits their final Proposed Resolution of Comments
- At the adjournment of the BRM we have the approved Resolution of Comments
- The Project Editor then has 30 days to apply the Resolution of Comments to produce the new text of DIS 29500
- It is the above revised text that NB’s will consider whether to approve or not. Note that since the NB only has 30 days to reconsider their Sept. 2nd vote, and the revised text is not due until 30 days after the BRM, it is likely that NB’s will need to use their imagination and decide based on the approved Resolution of Comments document (perhaps 4,000+ pages in length), not having seen the actual revised text of the DIS.
So, what can we sat about the recently released draft proposed resolution of comments document?
This initial set of responses are almost entirely minor, dealing with corrections to examples, spelling errors, punctuation errors, cleanup of broken links, fixing illegible formulas, adding missing units on quantities, etc. There are also many, many duplicates in this area. In particular, the issue regarding spreadsheet functions missing units on some functions (not specifying radians or degrees) was picked up by 12 NB’s. Since there are multiple instances of that defect in the OOXML specification, each one repeated by several NB’s, this single observation results in 48 proposed resolutions. Ecma appears to have concentrated on comments like this, easy to fix and duplicated, in this batch. So although there are 662 resolutions on paper, this maps to perhaps only 80 or so unique issues.
The breakdown of proposed resolutions by NB is in the table below. These numbers are a bit tricky to interpret with the duplicate comments, since one NB’s comments might have been addressed in passing while fixing another NB’s issues. So I doubt Microsoft is spending a lot of time on Columbia, since they voted yes. But there may be a significant duplication between Columbia’s comments and another NB which Microsoft is trying to please. But by looking at unique comments, those submitted by only one NB, we can get a good sense of which NB’s Ecma is trying to please most. And no, I’m not going to tell you which ones they are.
| Member | Comments Submitted | Ecma Responses | % Responded to |
|---|---|---|---|
| UK | 635 | 218 | 34% |
| Ecma | 76 | 23 | 30% |
| Colombia | 237 | 71 | 30% |
| Philippines | 7 | 2 | 29% |
| USA | 288 | 69 | 24% |
| Chile | 217 | 44 | 20% |
| Malta | 5 | 1 | 20% |
| Japan | 82 | 16 | 20% |
| Canada | 79 | 15 | 19% |
| Czech Republic | 75 | 13 | 17% |
| Uruguay | 18 | 3 | 17% |
| Ireland | 12 | 2 | 17% |
| France | 592 | 97 | 16% |
| Australia | 30 | 4 | 13% |
| Germany | 162 | 20 | 12% |
| Portugal | 118 | 14 | 12% |
| Brazil | 64 | 7 | 11% |
| Greece | 113 | 11 | 10% |
| Denmark | 168 | 15 | 9% |
| Kenya | 81 | 7 | 9% |
| Ghana | 12 | 1 | 8% |
| India | 82 | 5 | 6% |
| Israel | 33 | 1 | 3% |
| Venezuela | 73 | 2 | 3% |
| Iran | 58 | 1 | 2% |
| Turkey | 1 | 0 | 0% |
| Jordan | 1 | 0 | 0% |
| Ecuador | 1 | 0 | 0% |
| Thailand | 1 | 0 | 0% |
| Spain | 1 | 0 | 0% |
| Belgium | 1 | 0 | 0% |
| Austria | 1 | 0 | 0% |
| Argentina | 1 | 0 | 0% |
| China | 1 | 0 | 0% |
| Singapore | 2 | 0 | 0% |
| Italy | 2 | 0 | 0% |
| Tunisia | 3 | 0 | 0% |
| Bulgaria | 3 | 0 | 0% |
| Poland | 4 | 0 | 0% |
| Mexico | 7 | 0 | 0% |
| Peru | 10 | 0 | 0% |
| Norway | 12 | 0 | 0% |
| Finland | 15 | 0 | 0% |
| South Africa | 17 | 0 | 0% |
| Switzerland | 19 | 0 | 0% |
| Malaysia | 23 | 0 | 0% |
| Korea, Republic of | 25 | 0 | 0% |
| New Zealand | 54 | 0 | 0% |
To be fair, not every resolution in this batch was editorial. There was some technical detail added. For example, the following points were clarified:
- The SpreadsheetML AND/OR functions do not short circuit, so all parameters must be evaluated.
- The CHAR() function converts an integer into a character. But no character set was defined in the DIS to govern this conversion. Microsoft clarrified tis saying that the function uses the “Macintosh character set”on the Mac and ANSI on all other platforms.
- Spreadsheet functions that do searches or string compares (EXACT, FIND, FINDB, SEARCH, SEARCHB. etc.) do so with lexical character comparisons, not collation-based operations.
- Part names in an OPC package can be IRI’s, not just URI’s. So this allows Unicode characters, with some restrictions in items names
However, the 662 comments carefully tip-toed around the controversial issue. I guess we’ll read proposals for those in a future update. So NB members, take the opportunity now to get access to this portal. Ask your NB head for access if you haven’t already been given the password. And if you want a copy of my consolidated PDF file, let me know.
Your hard work and even-handed objectivity is a great service to the industry. Thank you for your efforts!
“The CHAR() function converts an integer into a character. But no character set was defined in the DIS to govern this conversion. Microsoft clarrified tis saying that the function uses the “Macintosh character set”on the Mac and ANSI on all other platforms.”
Um, WTF? Why have a function work differently on different *platforms*? That’s just insane. Having it work on the character set of the document would make some kind of sense, but on the character set of the platform?
Further, what is the “ANSI” character set? If it’s a “Windows code page” (which are not defined by ANSI[0]) then it should be specified as such instead.
It also raises the question as to what should be done on Linux systems that for the most part use ISO-* character sets, and not the Windows-* deviants of these.
And I’d wonder whether a Linux system should use the system’s default character encoding, or the user’s current character encoding. (Can individual users specify which character encoding to use for their applications on Windows?) In which case you could get the same document giving different results on the same computer when viewed by different people.
One thing to note here is that MS explicitly do not support UTF-8 as an non-UCS2 encoding[1], while most Linux distributions are moving towards putting everything in UTF-8. So it would likely be the case in the near future that Linux and Windows users would not share a common platform character set, even if they spoke the same language. (e.g. Windows English British in Windows-1252, and Linux en_GB.UTF-8) That’s going to make for some fun debugging, I’m sure…
[0] http://en.wikipedia.org/wiki/ANSI
[1] I can’t bring myself to call UTF-8 a “non-Unicode encoding”, which is what Microsoft terminology demands that it is. They call UCS2 “Unicode” and all other character sets “Non-Unicode”. But that’s clearly wrong as there are a number of non-UCS2 character sets that are also Unicode encodings. But if you try to explain this to someone who’s only read MS literature, they just don’t get it. You can’t call non-UCS2 sets MBCSs either, as ISO-* are not multi-byte.
Even their correction is ambiguous. What is the “MacIntosh Character Set”? There is Mac OS Roman, MacCyrillic, MacIcelandic, Mac Central European, and with OS X we have UTF-8 as the default.
In the end, the divergent behavior of this function reflects Excel’s divergent codebases on Windows and the Mac. You see this in other places as well, such as the way dates are treated. Date 0 is January 1st, 1900 on Windows, but January 1st, 1904 on the Mac. OOXML perpetuates this mess, throwing in a legacy bug in leap year calculations for good measure.
IMHO, an ISO standard in 2007 should not be bringing along legacy bugs and hardware constraints from 1990.
Back to CHAR() — the easy way to fix this function is to have an optional 2nd parameter, so the user can pass in a string indicating character encoding they want to use for this conversion.
Hi Rob,
we have all the original comments available and being consolidated in a number of ways, such as duplicates, not relevant, issue of substance, complete show stopper etc.
This is all freely available at http://www.dis29500.org
DIS 29500 should be renamed to
OCXML:
Office Closed XML
Since its beginning all the process of its development has been kept closed.
At this stage, things haven’t changed.
Shame on ECMA, Microsoft and ISO.
We ( the final users ) should be the main part in this process ( we are the ones who “suffer” the result of this standardizations ) and we are kept outside of it.
Shameful
Could you not just post the PDF as an attachment to your blogpost ?
Or place it on an arbitrary webserver.
If not mayby someone else can. That way everybody can just use a link to reference the document and also it could be upgraded when more comments become available?
It is a bit strange that you are complaining about the duplicates in the responses. It seems that a lot of duplicates are a result of IBM representatives submitting a large number of identical comments to the national bodies of more than one country. Mayby you could enlighten us in this area ?
To me, the most important objection to OOXML has always been the fact that ISO (“one standard, one test”) should not bless two gratuitously redundant standards for office documents (unless the first has unfixable problems, and the second is substantially superior and obsoletes the first). Narrow technical criticisms are important, but shouldn’t obscure this major problem.
And yet, I get the impression that Microsoft has almost entirely succeeded in eliminating the question of competing “standards” from the ISO deliberations, reducing debate to relatively minor technical clarifications. (Even our supposed friends from GNOME have been claiming that the question of whether we should have two standards is “political” and “not relevant to ISO’s process”.) Am I mistaken?
I still don’t understand why ISO are even entertaining Microsoft’s product standard. ISO aren’t in the business of endorsing single-vendor standards.
I don’t expect to find an ISO standard for a videotape. I do expect to find an ISO standard for a DVD. If I want to play a videotape, I’ll get a videotape deck (JVC, single vendor); if I want to play a DVD I’ll get something that works according to the ISO standard.
Microsoft Office works just fine for what it’s intended for; productivity for office workers, who happen to be using Lenovo-type Personal Computers.
But the documents it produces are useless for doing anything else with. Indexing. Machine parsing. Filtering on mail gateways. Editing on XBoxes, Playstations, and Wii’s. Editing on Apples.
ISO issuing a standard won’t help any.
I’m not complaining about the duplicate responses. I’m just observing that fact.
What we have is an artifact of the process. The review and comment-generation phase was done in the open marketplace of ideas. We had the wiki on GrokLaw, numerous observations on my and other blog, the BSI wiki, open forums, public input, debates, etc. So there is no surprise that a number of issues show up on more than one NB’s list. I think this redundancy makes the process more open, robust and less suceptible to manipulation.
Imagine the contrary. For example, imagine that only one member of one NB notices a particular serious security flaw in OOXML. The flaw is reported, in secret, to the NB, discussed in secret in the NB, and then squashed by Microsoft who shows up at the last minute with 20 business partners and votes to approve OOXML without comments. The security flaw would then never see the light of day.
A more reasonable process, IMHO, would be to have JTC1 NB’s contribute to a running list of ballot comments, that can be edited and viewed by all JTC1 participants. But when it comes time for the ballot, each NB votes on its own and references the comment numbers of the comments that they are concerned with. That would eliminate, or at least reduce, the duplication issue while still allowing a fuller consideration of the issues.
Of course, ISO has barely moved off of an entirely paper-based process, so it will take some time before they progress to a level where they can manage a real-time collaborative endeavor.
I hope no one misunderstands and thinks that the EMCA doesn’t want anyone to read these comments.
No, had that been the case, there would be 662 OOXML files, not 662 PDFs, on their website :-)
Wow! Ich bin beeindruckt!
Rob,
If ISO (or ECMA, or anybody) accept the odd hundred changes as desirable, and then DIS29500 is changed and a standard is issued, will anybody produce a conformant product ?
Microsoft Office 2007 is what Microsoft Office 2007 is. Maybe it conforms to DIS29500; maybe it doesn’t.
I presume that tying down OOXML will be like tying down Microsoft’s ‘Win32’ interface. If you wrote an application to ‘Win32’ as expressed in Windows95 (and I have a fair number of ‘Edutainment’ CDs which were), most of them don’t run now.
‘Win32’ changes. The old applications aren’t supported any more; I can’t get them revised, no more than I can get a 5-year-old newspaper revised. It’s water under the bridge, history, gone.
Important stuff needs stable standards. POSIX and ISO26300.
I’m curious as to whether ECMA TC45 faces different rules than the ODF folks did, and if so, why? Why did the ODF process manage to be (or at least appear) more open than the current ECMA process? It just seems like they would have had to abide by the same rules as ECMA TC45, and add password protection as well.
D.C., First, ISO has no control over the process under which a standard is developed in OASIS or Ecma. JTC1 only controls what happens within JTC1, after an existing standard is submitted. If the Ecma process was closed when developing OOXML, that was entirely due to decisions made in Ecma.
I’d like to say that IBM made some great effort and fought to make the OASIS ODF TC documents available to the public, but I can’t claim that distinction. The openness is standard procedure for OASIS and part of their rules. All OASIS TC’s have their mailing list archives open to the public, etc.
Thanks Rob. Perhaps Brian can clarify his point about following ISO rules then. I asked him the same question, so we’ll see what he says. I think it’s a fair question, given his response.
Rob –
It does feel a bit like you are playing fast and loose with the facts here. While it is true that “ISO has no control over the process under which a standard is developed in OASIS or Ecma.”, it seems you are almost purposefully confusing the process that came before the vote in September with what comes after. As I understand it, the rules about openess NOW are set by the JTC1, and Microsoft is not keeping things quiet, but just following rules. It may well be that they have made the comments inacessible, but the vast majority of your readers have no way of knowing that except taking your word on it. This is not one of your better posts, as it seems to willingly to fall into the “anything Microsoft does must be wrong” category, which is not only annoyingly partiosan, but actually hurts your case about OOXML by playing into the hands of those who find you overly partisan. Just my opinion, of course.
Ben, let me know if there is any particular part of the above that is confusing. It is a complex subject and I can certainly clarify any part of it.
My assertion is that the 662 documents are a private Ecma consultation with NB’s, not part of the ISO process. You’ll have to take my word for it — and no one has contradicted this — that these 662 PDF’s are hosted on the Ecma website, with an Ecma controlled password, with Ecma assigned ID’s. They have not been submitted to ISO, ISO is not distributing the documents and they have not been assigned ISO “N” numbers. Therefor, ISO has no say in their access control.
There is an important principle at stake here — NB sovereignty. If Microsoft can successfully argue that a private, unofficial communication between two JTC1 member is covered by ISO access controls, even though the document is not submitted to ISO, then every internal document shared from one NB to another is similarly protected.
Consider also JTC1 Directives, 9.9:
“When a document is out for ballot at Stage 3 or higher, NB/Liaison organisations are free to circulate their comments to other NBs provided they do not use the formal SC or JTC 1 documentation distribution system. Formal distribution is prohibited because it could create confusion as to the status of the ballot. Documents out for ballot at Stage 3 or higher are not to be subject to formal discussion at any working level of JTC 1 during the balloting period. Therefore, NB positions on the document under ballot are not to be formally discussed at any working level. Circulation of such comments shall have no formal status within JTC 1 or its SCs, i.e. they shall not bear any document number nor shall they be considered in any ballot resolution meeting unless they were formally submitted to ITTF as comments accompanying the ballot.”
In other words, these comments created by Ecma are not, and cannot be, official JTC1 documents. So one cannot argue that they are governed by ISO access controls, which (according to JTC1 Directives, Annex HD) apply to official JTC1 documents, hosted and controlled by JTC1.