Tuesday, March 25, 2008

Seeking Open Standards Activists

Some thoughts for Document Freedom Day 2008.

Back a few weeks ago in Geneva, OpenForum Europe hosted an evening of mini-talks and a discussion panel with various well-known personalities in our field: Vint Cerf, Bob Sutor, Andy Updegrove and Håkon Lie. I wasn't able to comment on the event at the time, due to my self-imposed blog silence that week, but I'd like to take the opportunity today to carry forward one of the topics discussed then.

I'd like to take as my launching point the theme of Andy Updegrove's talk, which was "Civil ICT Standards". Andy treats this subject more fully on his blog, and also speaks to the topic in his taped interview with Groklaw's Sean Daly.

Thus spake Updegrove:

But as the world becomes more interconnected, more virtual, and more dependent on ICT, public policy relating to ICT will become as important, if not more, than existing policies that relate to freedom of travel (often now being replaced by virtual experiences), freedom of speech (increasingly expressed on line), freedom of access (affordable broadband or otherwise), and freedom to create (open versus closed systems, the ability to create mashups under Creative Commons licenses, and so on.

This is where standards enter the picture, because standards are where policy and technology touch at the most intimate level.

Much as a constitution establishes and balances the basic rights of an individual in civil society, standards codify the points where proprietary technologies touch each other, and where the passage of information is negotiated.

In this way, standards can protect – or not – the rights of the individual to fully participate in the highly technical environment into which the world is now evolving. Among other rights, standards can guarantee:
  • That any citizen can use any product or service, proprietary or open, that she desires when interacting with her government.
  • That any citizen can use any product or service when interacting with any other citizen, and to exercise every civil right.
  • That any entrepreneur can have equal access to marketplace opportunities at the technical level, independent of the market power of existing incumbents.
  • That any person, advantaged or disadvantaged, and anywhere in the world, can have equal access to the Internet and the Web in the most available and inexpensive method possible.
  • That any owner of data can have the freedom to create, store, and move that data anywhere, any time, throughout her lifetime, without risk of capture, abandonment or loss due to dependence upon a single vendor.

Let us call these “Civil ICT Rights,” and pause a moment to ask: what will life be like in the future if Civil ICT Rights are not recognized and protected, as paper and other fixed media disappear, as information becomes available exclusively on line, and as history itself becomes hostage to technology?

This rings true to me. Technology, computer technology in particular, now permeates our lives. We interact with it daily, from the moment the internet-radio alarm clock goes off, until days end, when we check our email "one last time" before going to bed.

Similarly, the standards that define the interfaces between these devices are also of increasing importance. There was once a time when standards dealt only with the "infrastructure", the stuff in the walls and under the panel floor, or in that funny little locked door off the hallway, with all the cables and flashing lights, where strange men with clipboards would occasionally emerge, accompanied by a poof of cold air and the buzzing of machines.

But today, the technology and the standards that mediate the technology are now directly in front of your face. Think MP3 players. Think DVD's. Think DRM. Think cellular phones. Think web pages. Think encryption. Think privacy. Think documents. Think documents-privacy-security-DRM, your data and what you are allowed to do with it, and what others are allowed to do with it, and whether you control any bit of this in this mad world of ours.

Between you and the tasks that want to do today stands technology and the standards that mediate that technology. Standards are damn important.

Now, although the reach of technology and ICT standards has progressed over the years, the organizations and the processes that create these standards have not always kept up. In many cases standardization remains the creature of big industry with little or no consumer input. It is back-room discussions, where companies connive to see how many patents of their own portfolio they can encumber the standard with. A successful standard is one where no major company is left hungry. Consensus means everyone at the table has been fed. That is the traditional world of technology standards. It brings to mind the famous line from Adam Smith:

People of the same trade seldom meet together, even for merriment and diversion, but the conversation ends in a conspiracy against the public, or in some contrivance to raise prices — The Wealth of Nations (I.x.c.27)

Luckily, there is some hope. The proponents of "open standards" seek standards based on principles of open participation, consensus decision making, non-profit stewardship, royalty-free IP, and free access to standards. The web itself, with the underlying network protocol stack, HTML family of formats with DOM and scripting API's is a shining example of what open standards can accomplish. Tim Berners-Lee says it best, in his FAQ's:

Q: Do you have had mixed emotions about "cashing in" on the Web?

A: Not really. It was simply that had the technology been proprietary, and in my total control, it would probably not have taken off. The decision to make the Web an open system was necessary for it to be universal. You can't propose that something be a universal space and at the same time keep control of it.

But it is important to realize that "control" mechanisms in standards go well beyond IP and organization issues. There are other important factors at play, and we need to address these as well. Knut Blind discusses some of these issues a section called "Anti-Competitive Effects of Standards" from his The Economics of Standards (2004).

The negative impact of standards for competition are mostly caused by a biased endowment with resources available for the standardization process itself. Therefor, even when the consensus rule is applied, dominant large companies are able to manipulate the outcomes of the process, the specification of the standard, into a direction which leads to skewed distribution of benefits or costs in favor of their interests.

In other words, participation in standardization activities is time consuming and expensive, and large companies are much more able to make this kind of commitment than small companies, organizations or individuals. So ,large companies rule the world.

This is especially true with standardization at the international level, where decisions are often made at meetings in very expensive international locations. JTC1 is still discussing what technologies would be required to allow participation in meetings without travel. (Hint — its called a "telephone") To put this in perspective, my week in Geneva cost $3687.52. I flew coach, ate most of my meals on the cheap, often just grabbing hors d'oeuvres at receptions, and I received negotiated IBM corporate rates for air and hotel. This is one JTC1 meeting. What if I wanted to be really active? Add in two SC34 Plenary meeting (Norway/Kyoto). Add in JTC1 Plenary meetings. Add in US NB meetings. Add in US NB membership fees, consortium fees, conferences, etc. This starts adding up, around $40,000/year to participate actively in tech standards, and this doesn't include the cost of my time.

How many small companies are going to pay this amount? How many non-profit organizations? How many individuals? Not many.

But in spite of the expense, in spite of the large company bias of the international standardization system, I saw reason for hope at the Geneva BRM. I saw younger participants, with fire in their bellies. I saw FOSS supports from developing countries. I saw Linux on laptops. I saw participants from FOSSFA, SIUG, EFFI, ODF Alliance Brazil, COSS, etc. They joined their NB's, participated in their NB debates and were appointed to represent their countries in the BRM.

Sure, it is only a foot in the door. One in five BRM participants were Microsoft employees. But it was a hopeful sign. We've planted the seed. We must plant more. And we must see that they grow.

Strength in standards participation comes with time, with participation, with networking, with learning the rules (written and unwritten) learning from others, etc. Just as we have FOSS experts in the software engineering, in law, in business, in training/education, we also need experts in standardization. Certainly the bread and butter participation will be from individual engineers, participating for the duration of a particular proposal or group of proposals. But we also need the institutional linchpin participants, those who have taken on leadership positions within standards organizations, and whose influence is broad and deep.

FOSS also needs a standards agenda. In a world of patent encumbered standards controlling the central networks, open source software dies, and dies quickly. We must protect and grow the open standards, for without them we cease to exist.

What standards are important? Which demand FOSS representation? Remember just a few weeks ago, when there was a lot of concern about how the DIS 29500 BRM added explicit mention of the patent-encumbered MP3 standard, but failed to mention Ogg Vorbis at all? Although I sympathize with this concern, the fact is the BRM could not have added Ogg Vorbis at all, because it is not a standard. Are we willing to do more than lament about this? I tell you that if Ogg Vorbis had been an ISO standard it would have been explicitly added to OOXML at the BRM. Are we willing to do something about it?

What are the standards critical to FOSS, and what are we doing about it? What standards, existing or potential, should we be focusing on? I suggest the following for a start:
  1. Ogg Vorbis
  2. Ogg Theora
  3. PNG, ISO/IEC 15948
  4. ODF, ISO/IEC 26300
  5. PDF, ISO 3200
  6. Linux Standard Base (LSB), ISO/IEC 23360
  7. Most of the W3C Recommendations
  8. Most of the IETF RFC's
I'm sure you can suggest many others.

Let's put it all together. Some ICT standards directly impact what we can do with our data and our digital lives. These are the Civil ICT Standards. We need to ensure that these standards remain open standards, so anyone can implement them freely. However, the standardization system, both at the national and international levels is biased in favor of those large corporations best able to afford dedicated staff to work within those organizations and develop personal effectiveness and influence in the process. Showing up once a year is not going to work. If FOSS is going to maintain any level of influence in formal standardization world, especially at the high-stakes international level, it needs to find a way to identify, nurture and support participation of "Open Standards Activists". The GNOME Foundation's joining of Ecma, or KDE's membership in OASIS are examples how this could work. Umbrella organizations like Digistan also are critical and can be a nucleus for standards activists. But what about taking this to the next level, to NB membership? Another example is the Linux Foundation's Travel Fund, designed to sponsor attendance of FOSS developers at technical conferences. Imagine what could be done with a similar fund for attendance at standards meetings?

So that is my challenge to you on this first Document Freedom Day. We're near the end of what promises to be one of many battles. The virtual networks of the future are just as lucrative as the railroad and telephone networks of the last century were. These include the network of compatible audio formats, or the network of IM users using a compatible protocol, or the network of users using a single open document format. If FOSS projects and organizations want to secure the value for their users that comes from being part of these networks, then FOSS projects must encourage the use of open standards, and must also encourage and nurture new talent for the next generation of open standards activists.

I'm looking forward to the day, soon, when I can search Google for "open standards activist" and not find a paid Microsoft shill among the listings on the first page.

Labels:

Digg | Reddit | Del.icio.us

# posted by Rob : 3/25/2008 11:45:00 PM 22 comments (policy) links to this post

Monday, March 24, 2008

OOXML's (Out of) Control Characters

Let's start with the concepts of “lexical” and “value” spaces in XML, as well as the mechanism of “derivation by restriction” in XML Schema. Any engineer can understand the basics here, even if you don't eat and drink XML for breakfast.

The value space for an XML data item comprises the set of all allowed values. So the value space for the “float” data type would be all floating point numbers, such as 12.34 or 43.21. The lexical space comprises all ways of expressing these values in the character stream of an XML document. So lexical representations of the value 12.34 include “12.34”, “12.340” and '1.234E1”. For ease of illustration I will indicate value space items in bold, and lexical space items in quotes. In general there are multiple lexical representations that may represent the same value.

Character data in XML also permits more than one lexical representation of the same value. For example, “A” and “A” both represent the value A. The “numerical character reference” approach allows an XML author to easily encode the occasional Unicode character which is not part of the author's native editing environment, e.g., adding the copyright character or occasional foreign character. The value space allowed by XML includes most of Unicode, including all of the major writing systems of the world, current and historical.

The concern I have with DIS 29500 concerns Ecma's introduction of a ST_XString (Escaped String) datatype. This new type is defined via the following XML Schema definition:

<simpletype name="ST_Xstring">
<restriction base="xsd:string">
</simpletype>

This uses the “derivation by restriction” facility of XML Schema to define a new type, derived from the standard xsd:string schema type. The xsd:string type is defined to allow only character values that are also allowed in the XML standard.

The use of derivation by restriction implies a clear relationship between the ST_Xstring type and the base type xsd:string. This is stated in XML Schema Part 1, clause 2.2.1.1:

A type definition whose declarations or facets are in a one-to-one relation with those of another specified type definition, with each in turn restricting the possibilities of the one it corresponds to, is said to be a restriction.

The specific restrictions might include narrowed ranges or reduced alternatives. Members of a type, A, whose definition is a restriction of the definition of another type, B, are always members of type B as well.

The latest sentence can be taken as a restatement of the Liskov Substitution Principle, a fundamental principle of interface design, that a subtype should be usable (substitutable) wherever a base type is usable. It is this principle that ensures interoperability. A type derived by restriction limits, restricts, constrains, reduces the permitted value space of its base type, but it cannot increase the value space beyond that permitted by its base type.

So, with that background, let's now look at how OOXML defines the semantics of its ST_Xstring type:

ST_Xstring (Escaped String)

String of characters with support for escaped invalid-XML characters.

For all characters which cannot be represented in XML as defined by the XML 1.0 specification, the characters are escaped using the Unicode numerical character representation escape character format _xHHHH_, where H represents a hexadecimal character in the character's value. [Example: The Unicode character 8 is invalid in an XML 1.0 document, so it shall be escaped as _x0008_. end example]

This simple type's contents are a restriction of the XML Schema string datatype.

In other words, although ST_Xstring is declared to be a restriction of xsd:string it is, via a proprietary escape notation, in fact expanding the semantics of xsd:string to create a value space that includes additional characters, including characters that are invalid in XML.

Let's review some of the problems it introduces.

First, the semantics of XML strings that contain invalid XML-characters is undefined by this or any other standard. For example, OOXML uses ST_Xstring in Part 4, Clause 3.3.1.30 to store the error message which should be displayed when a data validation formula fails. But what should an OOXML-supporting application do when given a display string which contains control characters from the C0 control range, characters forbidden in XML 1.0?
How should these characters be displayed?

There is a reason XML excludes these dumb terminal control codes. They are neither desired nor necessary in XML.

Elliotte Rusty Harold explains the rationale for this prohibition in his book Effective XML:

The first 32 Unicode characters with code points 0 to 31 are known as the C0 controls. They were originally defined in ASCII to control teletypes and other monospace dumb terminals. Aside from the tab, carriage return, and line feed they have no obvious meaning in text. Since XML is text, it does not include binary characters such as NULL (#x00), BEL (#x07), DC1 (#x11) through DC4 (#x14), and so forth. These noncharacters are historic relics. XML 1.0 does not allow them.

This is a good thing. Although dumb terminals and binary-hostile gateways are far less common today than they were twenty years ago, they are still used, and passing these characters through equipment that expects to see plain text can have nasty consequences, including disabling the screen.

Further, since these characters are undefined in XML, they are unlikely to work well with existing accessibility interfaces and devices. At best these characters will be ignored and introduce subtle errors. For example, what does “$10,[BS]000” become if one system processes the backspace and another does not? Worst case, the accessibility interface expecting a certain range of characters as defined by the xsd:string type will crash when presented with values beyond the expected range.

Interfaces with existing programming languages are also harmed by ST_Xstring. How does a C or C++ XML parser deal with XML that now can allow a U+0000 (NULL) character in the middle of a string, something which is illegal in that programming language?

What about XML database interfaces that take XML data and store it in relational tables? If they are schema-aware and see that ST_Xstring is merely a restriction of xsd:string, they will assume the normal range of characters can be stored wherever an xsd:string can be stored. But since the value space is expanded, there is no guarantee that this will still be true. These characters may cause validation errors in the database.

By now, the observant reader may be accusing me of pulling a fast one. "But Rob, none of the above is a problem if the application simply leaves the ST_Xstring encoded and does not try to decode or interpret the non-XML character," you might say.

OK. Fair enough. Let's follow that approach and see where it leads us.

Let's look at interoperability with other XML-based standards. Imagine you do a DOM parse of an OOXML document that contains “strings” of type ST_Xstring. Either your parser/application is OOXML-aware, or it isn't. In other words, either it is able to interpret the non-standard _xHHHH_ instructions, or it isn't.

If it doesn't understand them, then any other code that operates on the DOM nodes with ST_Xstring data is at risk of returning the wrong answer. For example, what is the length of the string “ABC”? Three-characters, of course. But what is the length of the string “_x0041_BC” ? These two strings both have the same values according to OOXML. But an XML application might return 9 or return 3, depending on whether it is OOXML-aware or not. Since most (all) XML parsers are unaware of the non-standard escape mechanism proposed by OOXML, they will typically calculate things such as string lengths, string comparisons, string sorting, etc., incorrectly.

But suppose the parser/application is OOXML-aware and correctly decodes these character references into the correct Unicode values, then what? Assuming the host language doesn't crash from the existence of this control characters, we then are presented with problems at the interface with any other code that operates on the DOM. Suppose we try to transform the DOM via XSLT to XHTML. Will the XSLT engine properly handle the existence of these forbidden character values? The XSLT engine may just crash. But suppose it doesn't. How does it write out these control characters into XHTML? It can't. These values are not permitted in XHTML. Dead end. What about DocBook? DITA? OpenDocument Format? Not possible. Since these characters are not permitted in XML 1.0 at all, they will be forbidden in all other markup languages that are based on XML 1.0, or even XML 1.1 for that matter (XML 1.1 allows some but not all of these characters, in particular the NULL character is excluded).

Note further that with XML pipelining and with mashups, the application that writes XML output typically does not have direct knowledge of the application that originally produced the XML values. This decoupling of producers and consumers is an essential aspect of modern systems integration, include Web Services. By corrupting XML string values in the way that it does, DIS 29500 breaks the ability to have loosely coupled systems. Once the value space is polluted by these aberrant control characters, every application, every process that touches this data must be aware of their non-standard idiosyncrasies lest they crash or return incorrect answers. In this way, one standard perverts the entire XML universe, forcing them all to contend with the poor hygiene of a single vendor.

The reader might think that I exaggerate the importance of this, that surely ST_Xstring is only used in OOXML in edge cases, in rare, compatibility modes. We wish that this were true. However, a look at the DIS 29500 shows that ST_Xstring is pervasive, and in fact is the predominant data type in SpreadsheetML, used to express the vast majority of spreadsheet content, including cell contents, headers, footers, displays strings, error strings, tooltip help, range names, etc. Any application that operates on an OOXML spreadsheet will need to deal with this mess.

For example, here are some uses of ST_Xstring in DIS 29500, Part 4:
There are hundreds of additional uses. A search of DIS 29500 Part 4 for “ST_Xstring” returns 467 hits. OOXML also defines two additional types, “lptsr” (7.4.2.8) and “bstr” (7.4.2.4) that have the same flaw as ST_Xstring.

The reader might further argue that, although the type allows characters that are forbidden by XML, the actual occurrence of these values in real legacy documents is likely to be rare. This might be true, but this is cause for even greater concern. If every document contained these control characters, then we would immediately be aware of any interoperability problems when integrating OOXML data with other systems. But if these characters are permitted, but occur rarely and randomly, then the integration errors will also occur rarely and randomly, allowing data corruption and other problems to occur and propagate further before detection.

In summary, we are concerned that the ST_Xstring type in OOXML opens us up to problems such as:

  1. Introducing accessibility problems
  2. Breaking unaware C/C++ XML parsers
  3. Breaking XML databases
  4. Breaking interoperability with other XML languages
  5. Breaking application logic related to string searching, sorting, comparisons, etc.
  6. Introducing errors that will be hard to detect and resolve

Possible remedies include:

  1. Use xsd:string uniformly instead of ST_Xstring, with no use of forbidden XML characters. This would require that applications that read legacy binary documents containing such characters eliminate them at this point, perhaps replacing them with licit characters or with whitespace. No application will be more able to devise the original meaning and intent of these characters than the original vendor. So they should be responsible for cleaning up these strings to make them XML-ready.
  2. Use a non-string type such as the binary xsd:hexBinary or xsd:base64Binary to represent these data items.
  3. Use a mixed content encoding, where the licit characters are represented by xsd:string data, and the forbidden characters are denoted by specially-defined elements. So “A_x0008_BC” would become: <text>A<backspace/>BC </text>. In this case the semantics of the <backspace> element would need to be documented in the DIS 29500 specification, including its effect on searching, sorting, length calculations, etc.

Labels:

Digg | Reddit | Del.icio.us

# posted by Rob : 3/24/2008 04:25:00 PM 14 comments (policy) links to this post

Five (Bad) Reasons to Approve OOXML

  1. If you don't approve OOXML, Microsoft will walk away, and you'll never hear from them again. Forget the fact that OOXML is already an Ecma standard (Ecma-376), and cannot be taken away. Forget the fact that Microsoft has other formats lined up for ISO approval in the near future, like XPS or HD Photo. Microsoft wants you to think that if you don't give them exactly what they want, now, they will walk away from ISO and you will be the worse from it. We need to encourage Microsoft for their abuse of the standardization process, in hopes that their participation will evolve in line with our hopes, and not our fears, that they will improve on the standardization side, while curbing the abuse side. Of course, the encouragement could be misinterpreted to mean the opposite, and we could get more abuse, and even lower quality standards. I guess that's the risk we'll just need to take. By similar abuses of logic small children hold their breath until their faces turn blue, thinking they can scare adults into giving them what they want. It doesn't work there either.
  2. If you approve OOXML, you can have the privilege of spending the next 5 years in the glorious work of fixing thousands of defects in the text. You can get a seat at the table, fixing bugs that should have been fixed in Ecma before OOXML was even submitted to JTC1. Forget the fact that maintenance in JTC1 is a ponderous, time consuming activity, where individual defects are enumerated, changes proposed, discussed, voted on, etc. Forget the fact that the recent BRM showed that you can't really get through more than 60 defects in a week-long meeting. Forget the fact that fixing defects in Ecma, not JTC1, would be far faster and easier due to the lighter-weight process Ecma imposes on their TC's. Forget that Fast Track is intended for mature, adopted standards not for ones that will require a "Perpetual BRM". Forget all that. You want a seat at the bug fixing table? You got it.
  3. Billions and Billions of legacy documents. Well, actually these legacy documents are not in OOXML format; they are in the legacy binary format. And no mapping has been provided from the legacy formats to OOXML. But there are billions and billions of these legacy documents. That must be important. So vote Yes for OOXML because there are billions and billions of documents in some other format that is nebulously related to it.
  4. More standards are better. More standards means more choice, means more decisions, means more consultants, means more money paid to XML experts. You'll sooner find the American Dairy Council recommending less milk consumption than a standards professional calling for fewer standards. So ignore quality, maturity and need. More standards are a good thing. Like Blue-ray and HD DVD.
  5. ODF will be better if OOXML is approved. In OASIS we're too stupid to look up legacy features or Excel spreadsheet formulas in Ecma-376. We would have never thought of that. We believe the only way to make ODF better is to make it more like OOXML. That is why we would like to encourage nice little JTC1 countries like Kazakhstan to vote YES for OOXML. As soon as OOXML is approved, then magically, it becomes useful to us. But the exactly same text, not approved by Kazakhstan and JTC1, is not useful to us at all. It is all or nothing. There is nothing in the middle. Rather than taking a useful, high quality text, and approving it on its merits, we are asked to approve a specification with thousands of defects, and by our approval we transform it into something useful to ODF.

Labels:

Digg | Reddit | Del.icio.us

# posted by Rob : 3/24/2008 03:00:00 PM 7 comments (policy) links to this post

Tuesday, March 18, 2008

How many defects remain in OOXML?

DIS 29500, Office Open XML, was submitted for Fast Track review by Ecma as 6,045 page specification. (After the BRM, it is now longer, maybe 7,500 pages or so. We don't know for sure, since the post-BRM text is not yet available for inspection.) Based on the original 6,045 page length, a 5-month review by JTC1 NB's lead to 48 defect reports by NB's, reporting a total of 3,522 defects. Ecma responded to these defect reports with 1,027 proposals, which the recent BRM, mainly through the actions of one big overnight ballot, approved.

So what was the initial quality of OOXML, coming into JTC1? One measure is the defect density, which we can say is at least one defect for every 6045/1027 = 5.8 pages. I say "at least" because this is the lower bounds. If we believed that the 5-month review represented a complete review of the text of DIS 29500, by those with relevant subject matter expertise, then we would have some confidence that all, or at least most, defects were detected, reported and repaired. But I don't know anyone who really thinks the 5-month review was sufficient for a technical review of 6,045 pages. Further, we know that Microsoft worked actively to suppress the reporting of defects by NB's. So the actual defect density is potentially quite a bit higher than the reported defect density.

But how much higher? This is the important question. It doesn't matter how many defects were fixed. What matters is how many remain.

There are several approaches to answering this question. One approach is to look at defect "find rates", the number of defects found per unit of time spent reviewing, and fit that to a model, typical an S-curve (sigmoid) and use that model to predict the number of defects remaining. However, we have no time/effort data for the DIS 29500 review, so we don't have enough data to create that model. Another approach is to randomly sample the post-BRM text and statistically estimate the defect density by this sample.

Are there any other good approaches?

Here is the plan. I will use the second approach. Since I do not actually have the post-BRM text, I need to make some adjustments. I'll start with the original text, in particular Part 4, the XML reference section, at 5,220 pages, where the meat of the standard is. I'll then create a spreadsheet and generate 200 random page numbers between 1 and 5,220. For each random page I will review the clause associated with that page and note the technical and editorial errors I find. I will then check these errors to see if any of them were addressed by BRM resolutions.

Based on the above, I will be able to estimate two numbers:
So if I find N defects, and 0.9N of those issues were already found during the Fast Track review and were addressed by the BRM, then we can say that the Fast Track procedure was 90% effective in finding and removing errors. Some practitioners would call that the defect removal "yield" of the process. But if we find that only 0.1N of the errors were reported and addressed by the BRM, then we'll have a different opinion on the sufficiency of the Fast Track review.

Clear enough? Microsoft is claiming something like 99% of all issues were resolved at the BRM. So let's see if we get anything close.

I'm not done with this study yet. I'm finding so many defects that recording them is taking more time than finding them. But since this is topical, I will report what I have found so far, based on the first 25 random pages, or 1/8th completion of my target 200. I've found 64 technical flaws. None of the 64 flaws were addressed by the BRM. Among the defects are some rather serious ones such as:
As I said, this study is still underway. I'll list the defects I've found so far, and add to it as I complete the task over the next few days.
  1. Page 692, Section 2.7.3.13 — no errors found
  2. Page 1457, Section 2.15.3.45 — This is a compatibility setting which creates needless complexity for implementers who now must deal with two different ways of handling a page break, one in which a page break ends the current paragraph, and another where it does not. This is not a general need and expresses only a single vendor’s legacy setting.
  3. Page 490, Section 2.4.72 — This defines the ST_TblWidth type, used to express the width of a table column, cell spacing, margins, etc. The allowed values of this type express the measurement units to be used: Auto, Twentieths of a point, Nil (no width), Fiftieths of a percent. I find these choices to be capricious and not based on any sound engineering principle. It also mixes units with width values (Nil) and modes (auto). This should be changed to allow measurements in natural units, such as allowed in XSL-FO or CSS2, such as mm, inches, points, pica. Also, do not mix units, values and modes in the same attribute. Nil is best represented by the value 0 and Auto should be its own Boolean attribute.
  4. Page 328, Section 2.4.17 — The frame attribute description says it “Specifies whether the specified border should be modified to create a frame effect by reversing the border's appearance from the edge nearest the text to the edge furthest from the text.” This is not clear. What does it mean to reverse a border’s appearance? Are we doing color inversions? Flipping along the Y-axis? What exactly? Also a typographical error: “For the right and top borders, this is accomplished by moving the order down and to the right of its original location.” Should be “moving the border down…” Also, it is not stated how far the border should be moved.
  5. Page 1073, Section 2.14.8 — This feature is described as: “This element specifies the connection string used to reconnect to an external data source. The string within this element's val attribute shall contain the connection string that the hosting application shall pass to a external data source access application to enable the WordprocessingML document to be reconnected to the specified external data source.” Since connection to external data typically requires a user ID and a password, the lack of any security mechanism on this feature is alarming. The example given in the text itself hardcodes a plain-text password in it the connection string.
  6. Page 4387, Section 6.1.2.3 — For the “class” attribute it says “Specifies a reference to the definition of a CSS style.” The example implies that some sort of mapping will occur between CSS attributes and DrawingML. But no such mapping is defined in OOXML. The "doubleclicknotify" attribute implies some sort of event model that us undefined in OOXML. How do you send a message for doubleclicknotify? Why do we describe organization chart layouts here when it is not applicable to a bezier curve? What happens if this shape is declared to be a horizontal rule or bullet or ole object? The text allows you label it as one of these, but assigns no meaning or behavior to this. Why do we have an spid as well as an id attribute? The "target" attribute refers to Microsoft-specific I.E. features such as "_media". Although the text says that control points have default values, the schema fragment does not show this.
  7. Page 3164, Section 4.6.88 — This and the following two elements are all called "To" but this seems to be a naming error. 4.6.89 is essentially undefined. What does "The element specifies the certain attribute of a time node after an animation effect" mean? It doesn't seem to really signify anything. Ditto for 4.6.90.
  8. Page 5098, Section 7.1.2.124 — The example does not illustrate what the text claims it does. The example doesn't even use the element defined by this clause.
  9. Page 4492, Section 6.1.2.11 — The "althref" attribute is described as "Defines an alternate reference for an image in Macintosh PICT format". Why is this necessary for only Mac PICT files? Why would "bilevel" necessarily lead to 8 colors? We're well beyond 8-bit color these days. "blacklevel" attribute is defined as "Specifies the image brightness. Default is 0." What is the scale here? This needs to be defined. Is it 0-1.0, 0-255 or what? And what is "image brightness" in terms of the art? Is this luminosity? Opacity? Is this setting the level of the black point? For "cropleft", etc. -- what units are allowed? (implies %) How does "detectmouseclick" work when no event model is defined? "emboss effect" is not defined. "gain" has the same problem as "blacklevel" -- no scale is defined. This element has two different id attributes in two different namespaces, with two different types. "movie" attribute is described as "Specifies a pointer to a movie image. This is a data block that contains a pointer to a pointer to movie data". Excuse me? "A pointer to a pointer to movie data"? This is useless. The "recolortarget" example appears to contradict the description. It shows shows blue recolored to red, not black. The "src" attribute is said to be a URL, yet is typed to xsd:string. This should be xsd:anyURI.
  10. Page 1431, Section 2.15.3.30 — no errors noted
  11. Page 3405, Section 5.1.5.2.7 — The conflict resolution algorithm should be normative, not merely in a note.
  12. Page 875, Section 2.11.21 — Instead of saying that the footnote "pos" element should be ignored if present at the section level, the schema should be defined so as to not allow it at the section level. In other words, this should be expressed as a syntax constraint.
  13. Page 1955, Section 3.3.1.20 — This facility for adding "arbitrary" binary data to spreadsheets is said to be for "legacy third-party document components". No documentation or mapping for such legacy components has been provided, so interoperability with this legacy data cannot be achieved. Why isn't this expressed using the extension mechanisms of Part 5 of the DIS?
  14. Page 4526, Section 6.1.2.13 — The "allowoverlap" attribute is not sufficiently defined. In particular, what determines whether the object shifts to right or left? ST_BWMode is not adequately defined. For example, one option is "Use light shades of gray only". How light? And what is the difference between "hide" and "undrawn"? Also, concept of "wrapping polygon" is not sufficiently defined. For example, what is the wrapping polygon for an oval? The purpose of "dgmlayoutmru" is obscure. Wouldn't the most-recently-used layout option be the one which is actually in use, "dgmlayout"? The "dgmnodekind" attribute is undefined, said to be "application-specific". Is interoperabilty not allowed? The text seems to imply that applications must use application-specific values. The "href" attribute is give a string schema type. Shouldn't this be xsd:anyURI. The "id" attribute is said to be a "unique identifier". Unique in what domain? Among shapes of this type? Among all shapes? All shapes on this page? Among all ID's in the document? The "preferrelative" attribute is not sufficiently defined. Where is the original size stored? After what reformatting? This appears to be a specification for runtime behavior, not a storage artifact. But it is not clear what is required. For the "regroupid", where is the list of these possible id's? The Hyperlink targets _media and _search are Internet Explorer proprietary features.
  15. Page 1193, Section 2.15.1.39 — no errors noted
  16. Page 1459, Section 2.15.3.46 — no errors noted
  17. Page 2671, Section 3.17.7.150 — no errors noted
  18. Page 2347, Section 3.10.1.69 — An "AutoShow" filter is not defined in this standard, though it is called for in several places of this section. "Average" aggregation function is not defined. In fact, none of these aggregation functions are defined. Although some have common mathematical definitions, in a spreadsheet context it is critical to make an explicit statement on treatment of strings, blanks, empty cells, etc. For dataSourceSort, what type of sort is required? Lexical or locale-sensitive? This element seems to mix field-specific settings, like dragToCol with pivotTable-wide settings like hiddenLevel. This will result in large data redundancy as settings like hiddenLevel are stored multiple times, once for each pivotField. "Inclusive Mode" is not defined. "Measure based filter" is not defined. "AutoSort" mode is not defined. The resolution of pivot table versus cell styles is ambiguous. "If the two formats differ, the cell-level formatting takes precedence." Is this negotiation done at the level of the entire text style? Style ID? Or at the attribute level? "Outline form" is not defined. "server-based page field" is not defined. (what is a page field?) "member caption" is undefined.
  19. Page 2885, Section 3.18.51 — The values of the given type (ST_OleUpdate) are explicitly tied to the Microsoft Windows OLE2 technology via the two method calls IOleObject::Update or IOleLink::Update
  20. Page 3951, Section 5.5.3.4 — The base values "margin" and "edge" are ambiguous. Is it specifying positioning from the left or right page edge?
  21. Page 2710, Section 3.17.7.200 — The description of "lookup-vector" is insufficient. It seems to be saying that the range should be sorted. Is this really correct? Spreadsheet functions typically do not have side effects. Also, the sorting procedure is explicitly defined only defined for the Latin alphabet. What about the rest of allowed Unicode characters, including the C0 control characters which are allowed in SpreadsheetML cell contents? Where are they sorted?
  22. Page 934, Section 2.13.5.5 — The "id" attribute is required to be unique, but it is not specified over what domain it must be unique.
  23. Page 607, Section 2.6.2 — What does "reversing the borders's appearance mean"? How much offset is required for a shadow?
  24. Page 201, Section 2.3.2.19 — This feature allows the suppressing of both spell and grammar checking for a text run. These should be two different settings, one for spelling and one for grammar proofing. There are many cases where it is important to check one, but not the other, just as in content comprised of sentence fragments, which are not grammatically complete, but where correct spelling is desired.
  25. Page 1240, Section 2.15.1.74 — This setting specifies that the document should be saved into an undefined invalid XML format. But it is not stated how an XSLT transfor can be applied to an OOXML document, since OOXML is a Zip file containing many XML documents. So what exactly is the specified XSLT applied to?

That's as far as I've gone. But this doesn't look good, does it? Not only am I finding numerous errors, these errors appear to be new ones, ones not detected by the NB 5-month review, and as such were not addressed in Geneva. Since I have not come across any error that actually was fixed at the BRM, the current estimate of the defect removal effectiveness of the Fast Track process is < 1/64 or 1.5%. That is the upper bounds. (Confidence interval? I'll need to check on this, but I'm thinking this would be based on standard error of a proportion, where SE=sqrt((p*(1-p))/N)), making our confidence interval 1.5% ± 3%) Of course, this value will need to be adjusted as my study continues. However, it is starting to look like the Fast Track review was very shallow and that detected only a small percentage of the errors in the DIS.

[20 March Update]

As one commenter noted, the page numbers I'm using above are PDF page numbers, not the page numbers on bottom of each page. If I used the printed pages then I would need to deal with all the Roman numeral front matter pages as an exception. Simpler to just use the one large domain of PDF page numbers.

PDF Page Number = Printed Page Number + 7

I will continue to report new defects, according to the original random number list I generated. I'll update the statistics every 25.

Here's some more for today:

  1. Page 4192, Section 5.8.2.20 — "fPublished" attribute is defined as "Specifies whether the shape shall be published with the worksheet when sent to the spreadsheet server. This is for use when interfacing with a document server." What worksheet? This section is in the DrawingML reference material. Charts could appear in presentations as well. This should not be limited to worksheets. Also what is a "spreadsheet server"? No such technology has been defined in this standard. Also no protocol has been defined for publishing to a spreadsheet server. Is this some proprietary hook for SharePoint? The "macro" attribute allows the storage of application-defined scripts. We are told that the macro "should be ignored if not understood." However there is no mechanism for determining what language the script is in. How do we know if we understand the macro? Content sniffing? Attempt to execute it and see if we get a runtime error? But by that time, once we find out that we do not understand it, it is too late to ignore the macro. We may have already triggered runtime side effects. What we really need here is some way to declare what scripting language is being used, via a namespace or an additional attribute like "lang".
  2. Page 3526, Section 5.1.5.4.21 — The "algn" attribute specifies the text alignment. Allowed values include left, right, center, justified, etc. However, what is lacking is "start" and "end" alignment, which are sensitive to writing direction and are part of internationalization bets practices, for example, XSL-FO. When translating a document between RTL and LTR systems, the approach used by OOXML will harder to deal with and be more expensive to translate, since the translator will need to manually play with styles on not just perform an semi-automated translation.


[End Update]

I'll continue to review the remaining 173 pages of my random sample and update the numbers and the defect list as I go. If you want to play along at home, the upcoming random page numbers will be:

Labels:

Digg | Reddit | Del.icio.us

# posted by Rob : 3/18/2008 06:00:00 PM 53 comments (policy) links to this post

Friday, March 14, 2008

The Disharmony of OOXML

I sometimes hear it said that formats like OOXML, or ODF for that matter, are simply XML serializations of a particular application's native data representation. This is said, seemingly, in an attempt to justify quirky or outright infelicitous representations. "We had no choice. Office 95 represents line widths in units of 1/5th of a barleycorn, so OOXML must as well". This technological determinism indicates poor engineering judgment, laziness, or both.

An easy counter-example is HTML. Does HTML reflect the internals of NCSA Mosaic? Does it represent the internals of Netscape Navigator? Firefox? Opera? Safari? Are any faults in HTML properly justified by what a single browser does internally? Applications should follow standards, not the other way around.

The question we should be asking is not whether a standard is similar to an application's internal representation. That in itself is not necessarily a fault. We need to go a step further, and ask if this encoding represents reasonable engineering decisions, not just for that one application, but for general use? Or in ISO terms, does it represent the "consolidated results of science, technology and experience"? If it is a good, reasonable engineering choice with general applicability, and the original application already found that solution, then this is a good thing. We should be encouraging standards to encode the best practices of industry.

Take colors for example. There are only so many ways one can represent colors in markup. You can have an RGB value encoded as triplets where red= (255,0,0). Or you can have a hexadecimal integer encoded as RRGGBB, where red=FF0000. You can do it W3C style, like CSS or XSL-FO with a hash mark in front, like #FF0000". There are variations on this, adding an alpha channel, using a different color model, etc. These are all reasonable engineering choices, and no one would fault a standard for choosing any one of them, even if the choice happens to match what a particular application does. They are all reasonable choices.

The final arbiter is engineering judgment. Making a capricious choice, merely because a particular application made that same choice, in spite of contrary engineering judgment, this would be a bad thing.

With this in mind, let's take a look at how OOXML and ODF represent a staple of document formats: text color and alignment. I created six documents: word processor, spreadsheet and presentation graphics, in OOXML and ODF formats. In each case I entered one simple string "This is red text". In each case I made the word "red" red, and right aligned the entire string. The following table shows the representation of this formatting instruction in OOXML and ODF, for each of the three application types:

FormatText ColorText Alignment
OOXML Text<w:color w:val="FF0000"/><w:jc w:val="right"/>
OOXML Sheet<color rgb="FFFF0000"/><alignment horizontal="right"/>
OOXML Presentation<a:srgbClr val="FF0000"/><a:pPr algn="r"/>
ODF Text<style:text-properties fo:color="#FF0000"/><style:paragraph-properties fo:text-align="end" />
ODF Sheet<style:text-properties fo:color="#FF0000"/><style:paragraph-properties fo:text-align="end"/>
ODF Presentation<style:text-properties fo:color="#FF0000"/><style:paragraph-properties fo:text-align="end"/>

The results speak for themselves.

What is the engineering justification for this horror? I have no doubt that this accurately reflects the internals of Microsoft Office, and shows how these three applications have been developed by three different, isolated teams. But is this a suitable foundation for an International Standard? Does this represent a reasonable engineering judgment? ODF uses the W3C's XSL-FO vocabulary for text styling, and uses this vocabulary consistently. OOXML's representation, on the other hand, appear incompatible with any deliberate design methodology.

I fear that before we can tackle harmonization of ODF and OOXML, we will first need to harmonize OOXML with itself!

Labels:

Digg | Reddit | Del.icio.us

# posted by Rob : 3/14/2008 03:50:00 PM 31 comments (policy) links to this post

Tuesday, March 11, 2008

Implementation-defined (Not really)

Here begins the lesson on Embrace, Extend and Extinguish (EEE). Classically, this technique is used to perpetuate vendor lock-in by introducing small incompatibilities into a standard interface, in order to prevent effective interoperability, or (shudder) even substitutability of competing products based on that interface. This EEE strategy has worked well so far for Microsoft, with the web browser, with Java, with Kerberos, etc. It is interesting to note that this technique can work equally well with Microsoft's own standards, like OOXML.

An easy way to find these extension points is to search the OOXML specification for "application-defined" or "implementation-defined". You will find dozens of them, such as:

  1. In general, scripting
  2. In general, macros
  3. In general, DRM
  4. Part 1 — "Application-Defined File Properties Part" which is totally undefined, but is referenced 13 times for specific fields in Part 4.
  5. Section 2.16.4.1 — implementation-defined date/time formatting
  6. Section 2.16.5.34 — implementation-defined document filters
  7. Section 3.17.2.6 — implementation-defined string-->number conversions in a spreadsheet
  8. Section 2.8.2.2 — character sets supported by a font
  9. Section 2.9.6 — the interpretation of the mysterious hex "template code" in numbered list overrides — "The method by which this value is interpreted shall be application-defined."
  10. Section 2.14.27 — application-defined storage of exclusion data for a mail merge
  11. Section 2.15.1.28 — application-defined cryptographic hash algorithms
  12. 2.15.1.76 — "Specifies a string identifier which may be used to locate the XSL transform to be applied. The semantics of this attribute are not defined by this Office Open XML Standard - applications may use this information in any application-defined manner to resolve the location of the XSL transform to apply."
  13. Section 5.6.2.12 — application-defined macro string reference for connection shape
  14. Section 5.6.2.15 — application-defined macro string reference for graphic frame
  15. Section 5.6.2.24 — application-defined macro string reference for a picture object
  16. Section 5.6.2.28 — application-defined macro string reference for a shape
  17. Section 5.8.2.9 — application-defined macro string reference for a connection shape
  18. Section 5.8.2.12 — application-defined macro string reference for a graphic frame
  19. Section 6.2.2.14 — "This element specifies the presence of an ink object. An ink object is a VML object which allows applications to store data for ink annotations in an application-defined format."
  20. Section 7.6.2.60 — implementation-defined bibliographic citation formats
  21. And many, many more.
So, one might ask, what exactly does "implementation-defined"mean? Here is how OOXML defines it and related terms:

behavior, implementation-defined — Unspecified behavior where each implementation documents that behavior, thereby promoting predictability and reproducibility within any given implementation. (This term is sometimes called “application-specific behavior”.)

behavior, locale-specific — Behavior that depends on local conventions of nationality, culture, and language.

behavior, unspecified —Behavior where this Standard imposes no requirements. [Note: To add an extension, an implementer must use the extensibility mechanisms described by this Standard rather than trying to do so by giving meaning to otherwise unspecified behavior. end note]

Note that this is not an entirely novel definition. Anyone who has spent time reading over the C and C++ Programming Language standards, in ANSI or in ISO, will recall a similar set of definitions. For example, these from ISO/IEC 9899:1999 C-Programming Language:

implementation-defined behavior
unspecified behavior where each implementation documents how the choice is made

locale-specific behavior
behavior that depends on local conventions of nationality, culture, and language that each implementation documents

unspecified behavior
behavior where this International Standard provides two or more possibilities and
imposes no further requirements on which is chosen in any instance


So, you can see that OOXML pretty much copies these definitions. However, ISO standards like ISO/IEC 9899:1999 go one step further and make an additional statement in their conformance clause, something that is distinctly missing from OOXML:

"An implementation shall be accompanied by a document that defines all implementation-defined and locale-specific characteristics and all extensions."

If you poke around you will see that all conformant C compilers indeed do come with a document that defines how their implementation-defined features were implemented. For example, GNU's gcc compiler comes with this document.

So, by failing to include this in their conformance clause, OOXML's use of the term "implementation-defined" is toothless. It just means "We don't want to tell you this information" or "We don't want to interoperate". Conformant applications are not required to actually document how they extend the standard. You can look at Microsoft Office 2007 as a prime example. Where is this documentation that explains how Office 2007 implements these "implementation-defined" features? How is interoperability promoted without this?

(This item not discussed at the BRM for lack of time.)

Labels:

Digg | Reddit | Del.icio.us

# posted by Rob : 3/11/2008 11:45:00 AM 16 comments (policy) links to this post

Contra Durusau, Part 1

I have a lot of respect for Patrick Durusau. He has taught me much about how ISO standards work in practice, and I have benefited from his thoughts on that subject. I hope I can repay my debt to Patrick even in part, by teaching him something about how Microsoft works, in practice, a subject where I have expertise he lacks.

From the start Patrick has remained publicly silent on the topic of OOXML. No blog posts, no press, nothing. If you asked, he would say that this was his policy. Privately, you would get an earful (all negative), but as befits the unbiased chair of the committee which is responsible for the technical recommendation for the US NB, he kept his personal opinions out of the public arena.

This public orientation changed recently. As best I can figure it, on returning from a conference in Seattle in late January, Patrick was a changed man. Patrick is now an enthusiastic OOXML supporter and is eager to inform the world of his delight in OOXML at every opportunity. He posts his "open letters" on his web site, which are linked to, often within minutes, by the various Microsoft bloggers, and then sent around by Microsoft employees to the press and the various JTC1 NB's.

Patrick is entitled to his own opinions. Free speech (and free enterprise for that matter) are things which all red-blooded Americans believe in, among other things. So long as Patrick makes it clear that he is speaking for himself, I have no problem with this.

Of course, Microsoft will not be so careful to distinguish Patrick's personal opinions from his professional affiliations. So a post from Patrick's personal web site is retold on a Microsoft blog as "The ODF Editor says....", and then the next day is sent in an email to NB's with a larger set of "endorsements":

Chair, V1 - US TAG to JTC 1/SC 34
Convener, JTC 1/SC 34/WG 3 (Topic Maps)
Editor, OpenDocument Format TC (OASIS), Project Editor ISO/IEC 26300
Co-Editor, ISO/IEC 13250-1, 13250-5 (Topic Maps)

By the time it is actually discussed at the NB committee level, I wouldn't be surprised if it morphs into an assertion that JTC1/SC34, INCITS, the ODF TC and the City Council of Covington, Georgia have all approved OOXML. It is dangerous to wear many hats when dealing with Microsoft. They are not ones for fine distinctions.

But now on to the substance of Patrick's letters.

In his first note, the "OpenXML Poster Child", Patrick says:

OpenXML has progressed from being developed in a closed environment to being handed over to approximately 70% of the world's population for future development so I am missing the "not open" aspect of OpenXML. If anything, the improvements made to OpenXML during that process make it a poster child for the open standards development process.
.
.
.
I understand that SC 34 will be taking on the maintenance and future development of OpenXML (with the participation of Ecma). That will mean that approximately 70% of the world's population will have a say (through their respective national bodies) on how OpenXML continues to develop. I can't speak for anyone other than myself but that sounds pretty open to me. (That presumes approval of OpenXML as an ISO standard, which must be decided by every national body for itself.)

We've covered this before. Let's go down the list again. Where are the minutes from Ecma TC45 teleconferences? Where are the public archives of their mailing list? Where is the list of individuals participating in the TC? Where is the list of voting members? Where are the public comments they have received on OOXML? You call this open?

For ODF, all of this information is easily available to the public, here, here and here.

And don't give me the canard about how moving to SC34 results in greater representation.

In the US who represents our population? The 7 members of V1 before the DIS 29500 process began? Or the 26 members after Microsoft stuffed V1 (the committee that you chair) with business partners last summer? Or V1 after several of them were kicked out for not paying their dues? Or the V1 after the DIS 29500 procedure completes and the warm bodies fade away? In your opinion, which one do you believe truly represents our US population?

Similarly, SC34 was stuffed with new P-members and swelled from 9 P-members in 2006 to 40 today, most of which voted in favor of OOXML and then failed to participate in any other SC activities. Are you seriously suggesting that SC34 was increasing the world's influence over Microsoft's decisions? That sounds quite naive. To me this looks much more like Microsoft is increasing their influence over the world, and JTC1 NB's in particular.

The long list of shenanigans recorded, from Sweden to Portugal, from Poland to Switzerland is further evidence that the second interpretation is the accurate one. Is offering Microsoft partners rewards for joining a committee a way of increasing openness? Is joining JTC1 three days before the Sept. 2nd vote, then voting Yes without comments the way in which the world is able to gain a seat at the table?

Moving on.

Patrick's next post is "Co-Evolution". This, plus Microsoft's recent interoperability announcements (yes, yet more announcements) give the impression that they believe it is better to talk about interoperability than to do something about it. Interoperability is something we only talk about now, but accomplish sometime in the nebulous future, like weight loss or reducing the national debt. Create studies, write reports, open labs, make test cases, write more reports. But when given the opportunity to do something now which would actually improve interoperability, like adding missing features to OOXML to accommodate the richer text model in ODF, then just say "No". You can always do a study on this later, and write another report, and make a test case.

But if announcements alone could improve interoperability, then Microsoft would have solved this problem long ago and many times over.

The perspective that is missing in Patrick's analysis is that of the vast part of the world's population that does not benefit, and in fact is distinctly disadvantaged by having multiple incompatible document standards. We've been here before, in the 1980's and 1990's. It was not fun. We should not be seeking ways to repeat that failure.

Much of the world is also disadvantaged by the monopolist's rent paid on Microsoft products and the associated lack of choice in today's software monoculture. I'd rather help the world free itself of this oppression than appease the oppressor in hopes that he'll wield a more lenient whip.

Last September, the NB's of Great Britain, Brazil, Chile, Colombia, New Zealand, and the United States all requested that specific features be added to OOXML in order to improve interoperability with ISO ODF, in total 40 features such as the ability to have background images in tables or to have font weights beyond “normal” and “bold”. These were the exact features that Microsoft's translator project on SourceForge identified as needed to improve interoperability with ODF. Ecma rejected all of these requests. They did not reject them because the features were unreasonable. They were rejected purely because they were ODF features.

So given the chance to do more than just write reports and have panel discussions, Ecma refused to move interoperability forward even one inch. If this is them on their best behavior (they desperately need NB approval votes), then why would we expect greater consideration from them if OOXML were approved?

In his next letter,"Confusion", Patrick responds to Andy Updegrove, but not having followed that debate, I'm the one who is now confused. Patrick seems to be arguing that it doesn't matter whether OOXML is "good" or not (in fact he seems to argue that there is no "good" or "bad" when it comes to XML) but that it will be better if OOXML was someplace where we could talk at it more.

I don't know whether I'd choose to use moral terms when describing engineering artifacts either, but I would note that if the basic protocols and formats of the web were as poorly designed as OOXML, the web would never have thrived to become the glory it is today.

In "On the Importance of Being Heard" Patrick generously gives us his opinion of the DIS 29500 BRM he did not attend. The argument formally comes down to this:

  1. Based on published and unpublished reports from the BRM, it appears that "everyone at the table was heard" and "Microsoft was listening to everyone" in a "public and international" forum.
  2. If we now reject OOXML, we "all lose a seat at the table where the next version of the Office standard is being written".
  3. If we approve OOXML, even though "rough" then this "gives all of us a seat at the table for the next Office standard".
  4. Therefore, Patrick recommends approval of DIS 29500.

This argument has several critical flaws.

First, it is inaccurate to call the BRM proceedings "public". Neither the public nor the press was allowed to attend. Security guards were posted at the door to enforce this mandate. JTC1 is a private, Swiss-headquartered NGO, answerable to no one, with no statutory responsibility to the public. Patrick talks about "ordinary users, governments, smaller interests" having a seat at the table. This is a fantasy. I did not see any such representation at the table in Geneva. One in five BRM attendees were Microsoft employees. Over 25% of the 114 people in attendance were either Microsoft or Ecma TC45 members. I fear that Patrick underestimates the extent to which NB's have been stacked over the past two years and that he preserves some illusion of SC34 NB's comprised of "ordinary users, governments, smaller interests". Maybe that was true a few years ago, but the neighborhood has changed.

Was everyone at the table heard? Formally, it is true that every delegation had the opportunity to raise a single issue during the week. Some (those earlier in the alphabet) had the opportunity to raise two issues. But I think it is disingenuous to cast that as "everyone at the table was heard". For many delegations it was true that for every issue they were able to raise, they had 10 or 20 more that they wanted to raise, based on their analysis of Ecma's proposed dispositions, but were unable to because of insufficient time.

Was Microsoft listening? Yes. Everyone in the room was listening. Formally only the BRM itself could authorize changes to the standard at this point, regardless of Microsoft's or Ecma's opinion. So it is moot as to whether Microsoft was attentive. Whether they listened or not has zero impact on the ability of the BRM to make changes.

Patrick also appears to be impressed that this discussion all takes place "at a table where a standard for a future product was being debated by non-Microsoft groups?" What future product? The future product is Office 14 (Office 2009). Microsoft has not informed JTC1 nor Ecma on what the changes to OOXML will be for Office 14, due out later this year in beta form.

And then we come to main point of Patrick's argument. Vote "Yes" so we all have a seat at the table. Before we buy into that logic, I suggest we examine other Microsoft/Ecma standards and see how their approval has or has not lead to increased participation.

Microsoft has two primary ways to negate broader participation in a standard's maintenance. The first is standards abandonment. Take for example Ecma-234 "Application Programming Interface for Windows". A contemporary observer might have been just as enthusiastic as Patrick is now. Wow! Isn't this great? They are finally opening up and listening to the world! We finally have a seat at the table! I have a feeling that things are going to be better from now on!

Unfortunately, this standard was approved in December 1995 and covers the Windows 3.1 API only. Since Windows 95 shipped in August 1995, this Ecma standard was obsolete on the day it was approved. No revision of the standard was ever issued. Microsoft abandoned it.

Now certainly, there was nothing in principle that prevented the non-Microsoft Ecma members from continuing to maintain Ecma-234, creating errata documents, polishing up the language of the clauses, etc. But they had no effective way of actually evolving the standard when Microsoft withdrew from the process. That is the danger when you approve a single-vendor standard on the false assumption that this leads to openness.

The other way to negate broader participation in standards development is to create technical revisions at a rapid pace, and to create them within Microsoft with little outside participation. Note that this is how OOXML was created in the first place. And this is how Microsoft/Ecma maintains standards like the C# Programming Language. Ask your friends in JTC1/SC22 whether "70% of the world's population" has a "seat at the table" in evolving that standard. Let me know what you hear. I believe you'll hear that there has been negligible WG activity around C# maintenance, and that new revisions are promulgated by Microsoft, rubber stamped by Ecma, and sent on to SC22, canceling the previous standard and replacing it with the new one.

This trick can be very effective whenever the underlying Microsoft product has an update every 2-3 years. If your product revisions are more frequent than the required JTC1 maintenance checkpoints, then you can effectively ignore JTC1. That's how Microsoft/Ecma has played the game in the past.

Note that Office 2007 has been out since late 2006. Office 14 (Office 2009) is due out in beta form this year, with expected release next year. Any bets on whether the file format will require a technical revision to accommodate Office 2009? There is absolutely nothing that prevents Microsoft from submitting a revised file format specification for Ecma, getting a rubber stamp approval and then Fast Tracking it back into JTC1. Since that is how they have treated other Microsoft/Ecma standards, the burden is on those who argue the contrary to support their optimism.

So consider the facts:

  1. Microsoft has not supported the JTC1 maintenance process with their other Ecma Fast Tracks. There is no broader "seat at the table", no power sharing, no ownership by "70% of the world's population". It is 100% Microsoft.
  2. Microsoft's current charter in Ecma TC45 explicitly calls for Ecma to own maintenance of OOXML if approved, not SC34.
  3. Ecma in fact has submitted a proposal [PDF] to SC34 asking for control of OOXML to be handed back to them.
  4. With their "rejuvenation" of SC34 (from 9 to 40 P-members in 2 years) Microsoft clearly has the votes it would need to force any maintenance regime they desire.
  5. No one at Microsoft has made an official statement in writing confirming Patrick's vision of future maintenance. In fact their only official statement, the Ecma proposal to SC34 cited above, contradicts what Patrick is suggesting. So why are only 3rd parties speaking so glowingly about the future control of OOXML? Plausible deniability, anyone?
Until the following occur I'd advise a bit more skepticism, considering that we're dealing with a company with a clear record of abusing, subverting, abandoning, embracing and extending etc., standards:

  1. Ecma changes their TC45 charter to explicitly call for all maintenance activities (corrigenda as well as technical revisions) to be performed in an SC34 WG.
  2. Ecma explicitly withdraws their submission on DIS 29500 maintenance from the agenda of the Oslo SC34 Plenary and instead submits a proposal asking for future OOXML work to be done in a new WG in SC34, with a non-Microsoft chair.
  3. Microsoft publicly states that they will hand operational control of OOXML to SC34, not only for maintenance of OOXML 1.0, but also for technical revisions, and that they will support this being done under JTC1 IPR rules, and using the JTC1 process, and that they will implement whatever revisions SC34 develops within 1 year of approval.
Until you have that, you have nothing. Get that, and then you can start talking about having a "seat at the table".

In his most recent post, "Russian Peasant" Patrick suggests that the only reason one would vote against OOXML is spite, and that any problems could be fixed in maintenance.

Let's try another analogy. You are shopping for a new TV and you go to your local consumer electronics store and look at the array of television sets lined up. Most come with a warranty. Any defects detected within the maintenance period will be fixed at the manufacturer's expense. This is generally a good thing, having a maintenance period to fix problems that were not evident at purchase time.

So you find the model TV you want, the salesperson rolls out the box and just before you hand over your credit card, you notice a big gash on the side of the box, where a forklift had pierced it. You say, "I can't accept this TV, it has been smashed!". The salesperson says, "Don't worry. No TV is perfect. We can fix this in maintenance. You're fully covered."

Do you hand over your credit card? Of course not. Maintenance periods, with TV's as with standards, are for defects detected after the fact. It is not a replacement for proper inspection, review and approval processes. You expect a TV to work properly at the start.

No standard is perfect. We all know that. But at the time of approval, NB's should be confident that their technical review was sufficient to find all of the important issues, and that these issues have all been fixed in the standard. OOXML should not be approved unless it is suitable now. The maintainers of OOXML will be busy enough fixing other problems that will be found later. We should not willingly approve a defective standard and set up a future maintenance group for failure by front-loading their agenda with defects that we already know about.

Consider: If we do that, then on what grounds can we reject another Fast Track proposal ever again? This slippery argument — we can fix that in maintenance — can be used for every single proposal that ever comes along. Why even have JTC1 at this point? Easier for everyone involved just hand the "International Standard" stamp over to Ecma and allow them to rubber stamp their own International Standards. This will save the time and expense of engaging hundreds of representatives from 87 JTC1 NB's for a year for a sham review.

My advice is this. Let's turn this train wreck around. Vote No on DIS 29500 and send a clear message that 6,000 page immature standards are not appropriate for JTC1 Fast Track. It showed poor judgment and great disrespect toward JTC1 NB's for Microsoft to send this mess via Fast Track in the first place.

Microsoft has every right to feel that they are late to the game, and risk being left behind for their lack of an open document standard. But they should not expect that they can simply throw money around and remedy their long neglect overnight. And certainly they should not expect JTC1 NB's to do the work for them. Microsoft should work on their specification at the consortium level and get it right first. Once when they have something mature, then they should send it along, preferable in smaller parts submitted sequentially. If they are unwilling or incapable of fixing the specification in Ecma then they could propose it as a new work item in SC34, where they may find some assistance. But if they persist on the standard remaining a single vendor standard, unilaterally controlled to benefit that single vendor, then I wouldn't expect a warm reception in SC34 either.

Labels:

Digg | Reddit | Del.icio.us

# posted by Rob : 3/11/2008 08:15:00 AM 21 comments (policy) links to this post

Thursday, March 06, 2008

JTC1 Improv Comedy Theater

JTC1 has been improvising its Fast Track processing from the start of the DIS 29500 procedure.

The latest "let's invent a new rule" came at the BRM in Geneva, where a novel approach to tallying meeting votes was surreptitiously foisted on delegations, one which is clearly against the plain text of JTC1 Directives.

The question is how votes should be counted at a Fast Track BRM, where consensus cannot be reached, in this case for lack of time. Specifically, in that final batch-vote on 1027 comments, how should votes be counted. I believe the rules call for positions to be established by the majority of P-members. The leadership of the meeting instead counted both P-members and O-members. In the balance lies the fate of over 100 Ecma proposals which may or may not be included in the final text of the DIS, depending on how this question is resolved.

Let's review the rules, from the current JTC1 Directives (5th Edition, Version 3.0)

First let's start with the overriding rule from section 1.2 "General Provisions":

These Directives shall be complied with in all respects and no deviations can be made without the consent of the Secretaries-General.

Or in plain English — "These are the rules, you can't just make stuff up".

So what is a P-member and an O-member? This is covered in chatper 3 "Membership Categories and Obligations". P-members are defined as:

P-members within JTC 1 shall be NBs that are Member Bodies of ISO or National Committees of IEC, or both. Only one NB per country is eligible for membership in JTC 1. P-members have power of vote and defined duties.

and O-members are defined as:
Any NB that is a Member Body of ISO or National Committee of IEC, or both, may elect to be an O-member within JTC 1. Correspondent members of ISO are also eligible to be O-members of JTC 1. O-members have no power of vote, but have options to attend meetings, make contributions and receive documents.

So clear enough? O-members can attend meetings and contribute, but cannot vote. P-members can vote at meetings.

Section 9 deals with the voting rules, and 9.1.4 speaks about meeting votes in particular:

In a meeting, except as otherwise specified in these directives, questions are decided by a majority of the votes cast at the meeting by P-members expressing either approval or disapproval.

So, in a meeting, only P-members vote and they vote by majority. "Except as otherwise specified in these directives" means that this rule can be overridden in specific cases. But the override must be "specified", i.e., actually written down that it is an override of the normal meeting voting rules.

So drilling down a level deeper we come to the Fast Track rules themselves in chapter 13, where in 13.8 is covered meeting votes at a Fast Track BRM:

At the ballot resolution group meeting, decisions should be reached preferably by consensus. If a vote is unavoidable the vote of the NBs will be taken according to normal JTC 1 procedures.

So on the surface this seems to be a vague statement. What are "normal JTC 1 procedures"? However, a moment's reflection on 9.1.4 above shows that the Directives have already declared this as the normal procedure for meeting votes by saying that this is the rule that holds unless specified otherwise.

One can easily seek confirmation of this by looking at the parallel rules for PAS process BRM votes, given in 14.4.3.9. Here it is more explicit:

At the ballot resolution group meeting, decisions should be reached preferably by consensus. If a vote is unavoidable, the approval criteria in the subclause 9.1.4 is applied.

So despite the clear and plain text of the Directives, the JTC1 leadership decided to improvise a new rule, or more precisely the application of a different rule in the wrong context. The argument appears to be that section 9.5 applies to BRM votes. Section 9.5 "Combined Voting Procedure" is introduced as:

The voting procedure which uses simultaneous voting (one vote per country) by the P members fo [sic] JTC 1 and by all ISO member bodies and IEC national committees on a letter ballot is called the combined voting procedure. This procedure shall be used on FDISs, DISs, FDAMs, DAMs and FDISPs.

This is absurd. JTC1 Directives are not a menu. You can't just pick what voting procedure you want to use from the list. The Directives tell you what procedure to use. First, the combined voting procedure is for letter ballots given to an NB, not for a BRM meeting vote by a delegation. Second, the BRM was not voting on an FDIS, DIS, FDAM, DAM or FDISP. We were voting on whether to include changes into a set of meeting resolutions. We were told repeatedly that the BRM could not take a position on the DIS. Finally, if combined voting procedures are read as applying to Fast Track, then they would also, by that same logic, need to apply equally to PAS, since both PAS and Fast Track are DIS's. But as shown earlier, the PAS process explicitly calls for P-member majority voting according to 9.1.4.

One does not arrive at the voting rules of 9.5 by any straightforward or natural reading of the Directives.

So again, repeating from JTC1 Directions 1.2:

These Directives shall be complied with in all respects and no deviations can be made without the consent of the Secretaries-General.

I wasn't in favor of having any batch ballot, because it violates the spirit of the consensus process, as defined in JTC1 Directives 1.2:

These Directives are inspired by the principle that the objective in the development of International Standards should be the achievement of consensus between those concerned rather than a decision based on counting votes.

[Note: Consensus is defined as general agreement, characterised by the absence of sustained opposition to substantial issues by any important part of the concerned interests and by a process that involves seeking to take into account the views of all parties concerned and to reconcile any conflicting arguments. Consensus need not imply unanimity. (ISO/IEC Guide 2:1996)]

To resort to "counting votes" on the vast majority of the technical issues of DIS 29500, without discussion or opportunity for objection, this is a failure of the JTC1 process. But if we are to have a vote at all, then let it be done in accordance with the rules.

So, let's stop the nonsense. Let's quit the tortuous post facto reinterpretation of the rules. Let's recount and republish the results of the BRM counted according to the Directives and move on with the process. If JTC1 cannot consistently adhere to its own rules, then it should consider another line of business.

Labels: ,

Digg | Reddit | Del.icio.us

# posted by Rob : 3/06/2008 02:31:00 PM 32 comments (policy) links to this post

Tuesday, March 04, 2008

OOXML, Macros and Security

As we all know, rich desktop editors, such as those provided in Microsoft Office, offer a range of end-user programming options, such as Visual Basic macros. These can be used to automate repetitive clerical tasks, such as a mail merge, or to add a custom user interface over a data entry form. These capabilities have existing in personal productivity applications since the late 1980's — so 20 years now. This is a not cutting-edge feature.

Such scripting capabilities are essential for the creation of high-value scripted documents. These features are essential in modern applications. Almost every word process or spreadsheet today has automation capabilities. Even open source applications like OpenOffice have macro features. So, considering the popularity and value of scripting in a productivity application, it is much lamented that DIS 29500 does not define how scripts or macros are to work. This lack will cause serious interoperability concerns, as each vendor, lacking standards guidance, will implement these features in incompatible ways.

Specifically, in order to have any interoperability among scripted documents, it is necessary to define:
OOXML defines none of these. So how can it meet its goal to "represent faithfully the existing corpus of word-processing documents, spreadsheets and presentations that have been produced by Microsoft Office applications (from Microsoft Office 97 to Microsoft Office 2008 inclusive)"? How can it do that and ignore the macros that have been around for decades?

Note that there is ample precedent for a markup standard answering these questions in a flexible and interoperable manner. For example the common web paradigm would be:
OOXML provides none of this, so interoperability of these high value documents is not possible. Note again that scripting is widespread and has been around for 20 years. So it is especially unfortunate that a newly proposed standard lacks this capability.

Note however that scripting is not without its problems. We all remember the Word Macro Viruses of several years ago, such as Melissa. Portable code has well-known risks, and these risks have well-known counter-measures. For example, it is common for anti-virus software to scan Word documents for viruses. It is also common for mail servers to scan incoming emails for attachments with viruses, and even remove the macros or block documents with macros, according to admin policy. So there is a need toenable 3rd party applications that can locate, retrieve, scan and delete scripting elements from documents. However, since OOXML does not define even where the scripts are stored, or how they can be located, such 3rd party applications cannot be written in general for a document described by this specification. The standard provides an insufficient foundation for implementing a reasonable security policy around OOXML documents.

For example, take Ecma Response 101, approved in Geneva in a 9-4 vote as part of a large batch 0f 1027 changes, without discussion or opportunity for dissent. Four NB's, in their ballot comments from last September, pointed out that Section 2.16.5.41 of DIS 29500's Part 4 defines a "MACROBUTTON" field that allows the definition of a button in the document that will trigger a macro. But nothing is said about how the macro is stored, bound, what API's are available, what the security model is, etc.

The request from one NB was to "Describe this feature to a level where cross-platform, cross-application interoperability is possible." However, what Ecma provided in their draft Disposition of Comments report, approved in batch by the BRM without discussion or opportunity for objection, was something quite different. They merely added the the following text:

The mechanism by which the command specified by text in field-argument-1 is located and/or executed by an application is implementation-defined

So not only is it impossible to have cross-platform interoperability of this feature, it is not even possible to implement a reasonable security policy to detect, scan or block macros. Even the location of the macro is outside the scope of the standard. It could be just another file in the Zip. It could be a binary blob with an obscure content type that varies from application to application. It could be base64Encoded in the XML. Or it could be steganographically encoded in low-order bits of an image file. The OOXML standard is singularly unhelpful in telling us how to deal with this risks of this macro function.

Finally, note that this lack of information on how to locate macros within a document makes it impossible for anyone to programmatically combine or divide OOXML documents which may contain macros. For example, imagine a 2-page spreadsheet, with a macro on sheet one only. How can it be split into two one-page documents, if there is no defined way to locate the script associated with page one? This is the type of automated composition and document manipulation that OOXML should be enabling. Similarly, how can one combine two single documents containing macros into one document, if there are no defined rules for locating and naming macros? Many basic types of applications,such as merging slide shows, etc., will break in the presence of macros.

The above topic was of interest to several NB's in Geneva, but could not be discussed for lack of time at the BRM.

Labels:

Digg | Reddit | Del.icio.us

# posted by Rob : 3/04/2008 05:30:00 PM 9 comments (policy) links to this post

The Carolino Effect

"There is it some game in this wood?"

Pedro Carolino wanted to write and publish an Portuguese/English phrase book.

"Another time there was plenty some black beasts and thin game, but the poachers have killed almost all."

But one small problem — Carolino did not know English.

"Look a hare who run! let do him to pursue for the hounds! it go one's self in the plonghed land."

Undeterred, Carolino hatched a clever plan.

"Here that it rouse. let aim it! let make fire him!"

He had a copy of an Portuguese/French phrasebook, O Novo guia da conversação em francês e português by José da Fonseca. And he had a French/English dictionary.

"I have put down killed."

With these two resources, writing his phrase book would be easy. Or so he thought.

"Me, i have failed it; my gun have miss fixe."

Starting from the French half of the text in da Fonseca's book, Carolino dutifully used his dictionary to translate, word-for-word, the French into English.

The result, O Novo Guia da Conversação, em Português e Inglês, e