An Antic Disposition

Introduction

There are several words, more widely used than understood, that recur frequently when discussing standards. Specification and standardization requires us precisely to describe technology in such a way that practitioners in that field can achieve the goals set out in the standard. But this precision is only perfectly intelligible to those who share the same code words. What follows is a handful of the more important ones, what they mean, and how they are unintentionally confused or intentionally misused. You are at a distinct disadvantage when reading (or writing) a news article, a blog post, or evaluating an argument if you do not know the correct meaning of the following words.

Standard

Take the definition from ISO/IEC Guide 2:2004, definition 3.2:

[A] document, established by consensus and approved by a recognized body, that provides, for common and repeated use, rules, guidelines or characteristics for activities or their results, aimed at the achievement of the optimum degree of order in a given context.

NOTE Standards should be based on the consolidated results of science, technology and experience, and aimed at the promotion of optimum community benefits.

So, it is a document, a written description, not an embodiment in the form of a product, that is standardized. Its aims are the "achievement of optimum degree of order" and "promotion of optimum community benefits", and it is achieved through consensus and consolidation.

international standard

According to ISO/IEC Guide 2:2004, definition 3.2.1.1:

[A] standard that is adopted by an international standardizing/standards organization and made available to the public.

International Standard

[An] international standard where the international standards organization is ISO or IEC

Note the distinction. With capital letters only ISO or IEC standards apply. With lowercase, other standards are included. This is a bit self-serving. ISO and IEC Standards are the only International Standards, because ISO says so. Sorry ITU, sorry CEN, sorry W3C.

So think of "International Standards" as a controlled mark of ISO, like "parmigiano reggiano" is a controlled mark of the Northern Italian Cheese Consorzio.

Normative

The normative parts of a standard are those which set out the scope and provisions of the standard. See ISO Directives, Part 2, section 3.8.

Provisions

The provisions of a standard consist of:

1. Requirements that must be met for conformance
   
2. Recommendations
3. Statements of permissible, possible or capable actions

See ISO Directives, Part 2, section 3.12.

Note that standards have specific words which denote and distinguish requirements, recommendations and capabilities. Different standards organizations have different vocabulary for this, so a W3C Recommendation, an IETF RFC and an ISO Standard may have different ways of stating the same provision. For ISO Standards, the conventions are:

• "shall" and "shall not" are the normal terms for expressing requirements.
• "should" and "should not" are the normal terms for expressing recommendations.
• "may" and "need not" are the normal terms for expressing permission.
• "can" and "cannot" are the normal terms for expressing possibility and capability.
  

This is necessary because of the extreme ambiguity of the English language in the area of modality. Consider the following sentences, using the word "must":

• (On hearing the doorbell ring), "Oh, that must be the mailman!" [expressing likelihood]
  
• (To a misbehaving child) "You must obey your mother" [expressing obligation]

Or the following exchange with a teenage daughter:

• Teen: "I shall return by 11pm" [simple future]
• Parent: "No, you shall return by 10pm" [expressing a command]

We can be loose and still be understood, in context, in normal conversation, but in standards work we try to be precise and uniform in the use of our control vocabulary.

Conformance

This simply is a question of whether something meets the requirements of the standard. However, for many standards, there are multiple levels, perhaps even multiple classes of conformance. So you need to be very specific about what you are saying.

For example, you should not ask "Does Excel 2007 conform to OOXML?" You should ask "Is Excel 2007 a conforming transitional class SpreadsheetML Producer?" If you count it all up, OOXML probably has at least 18 distinct conformance classes, by various combinations of applications, documents, readers/writers and transitional/strict conformance classes.

Not in particular that conformance does not mean that an application implements the entire standard.

[My definition above is not very satisfactory. Anyone have something better? Is there an ISO definition of conformance?]

Compliance

This is not a typical standards term. The more typical term is "conformance". Best to avoid it unless you are talking in regulatory or legal context. See ISO Directives, Part 2, section 6.6.1.1:

A document does not in itself impose any obligation upon anyone to follow it. However, such an obligation may be imposed, for example, by legislation or by a contract. In order to be able to claim compliance with a document, the user needs to be able to identify the requirements he/she is obliged to satisfy. The user also needs to be able to distinguish these requirements from other provisions where there is a certain freedom of choice.

Validity

This is an XML term, referring to the relationship between an XML document instance (an XML file) and a schema (the definition of the syntax of the markup language). Generally, an XML document instance is valid if it adheres to the constraints defined in the schema. The precise definition of validity will depend on the schema definition language used.



I'd welcome any suggestions for other words or definitions that should be included here.

Labels: Standards

Another neo-colonialist press release from Microsoft's CompTIA lobbying arm, this time inveighing against South Africa's adoption of ODF as a national standard. One way to point out the absurdity of their logic is to replace the reference to ODF with references to any other useful standard that a government might adopt, like electrical standards.

When we do this, we end up with the following.

---

South Africa Electrical Current Adoption Outdated

South Africa’s recent adoption of the 230V/50Hz residential electrical standard represents a tact that will blunt innovation, much needed for their developing economy. The policy choice – which actually reduces electrical current choice – runs contrary to worldwide policy trends, where multiple electrical standards rule, thus threatening to separate South Africa from the wealth creating abilities of the global electrical industry.

For MonPrevAss, the Monopoly Preservation Association, the overall concern for the global electrical industry is to ensure that lawmakers adopt flexible policies and set policy targets rather than deciding on fixed rules, technologies and different national standards to achieve these targets. Such rigid approaches pull the global electrical market apart rather than getting markets to work together and boost innovation for consumers and taxpayers. “The adoption sends a negative signal to a highly innovative sector” says I.M. Atool, MonPrevAss's Group Director, Public Policy EMEA.

The “South African Bureau of Standards” (SABS) approved the 230V/50Hz residential electrical standard on Friday 18 April as an official national standard. This adoption, if implemented, will reduce choice, decrease the benefits of open competition and thwart innovation. The irony here is that South Africa is moving in a direction which stands in stark relief to the reality of the highly dynamic market, with some 40 different electrical current conventions available today.

“Multiple co-existing electrical standards as opposed to only one standard should be favoured in the interest of users. The markets are the most efficient in creating electrical standards and it should stay within the exclusive hands of the market”, I.M. Atool explains.

In light of the recent ISO/IEC adoption of the Microsoft 240V/55Hz electrical standard, the South African decision will not lead to improvements in the electrical sector. MonPrevAss urges Governments to allow consumers and users to decide which electrical standards are best. We fear that the choice of just one electrical standard runs the risk of being outdated before it is even implemented, as well as being prohibitively costly to public budgets and taxpayers.

Governments should not restrict themselves to working with one electrical standard, and should urge legislators to refrain from any kind of mandatory regulation and discriminatory interventions in the market. The global electrical industry recommends Governments to embrace the reality and logic of choice and to devote their energies to ensuring interoperability through this choice.

---

Of course, this is just a rehash of an old logical fallacy, related to the old "Broken Windows" fallacy. It is like saying heart disease is a good thing because you have such a wide choice of therapies to treat it. We would all agree that it is far preferable to be healthy and have a wide choice of activities that you want to do, rather than a wide choice of solutions to a problem that you never asked for and don't want.

Consumers don't want a bag of adapters to convert between different formats and protocols. That is giving consumers a choice in a solution to a interoperability problem they didn't ask for and they don't want. Consumers want a choice of goods and services.

Observe the recent standards war with Blu-ray and HD DVD. Ask yourself:

1. Did consumers want a choice in formats, or did they want a wider choice in players and high definition movies?
2. Did movie studios want a choice in formats and either the uncertainty over choosing the winner, or the expense of supporting both formats? Or did they really just want a single format that would allow them to reach all consumers?
3. Did the uncertainty around the existence of two competing high definition formats help or hurt the adoption of high definition technologies in general?
   
4. Did consumers who make the early choice to go with HD DVD, say Microsoft XBox owners, benefit from having this choice?

If every private individual, and every private business has the right to adopt technology standards according to their needs, why should governments be denied that same right? Why should they be forced to take the only certain losing side of every standards war -- implementing all standards indiscriminately -- a choice that no rational business owner would make?

How many spreadsheet formats does Microsoft use internally for running their business on? Why should governments be denied choice in the same field that Microsoft itself exerts its right to chose?

Labels: Microsoft, Standards

A curious FAQ put up by an unnamed ISO staffer on MS-OOXML. Question #1 expresses concerns about Fast Tracking a 6,000 page specification, a concern which a large number of NB's also expressed during the DIS process. Rather than deal honestly with this question, the ISO FAQ says:

The number of pages of a document is not a criterion cited in the JTC 1 Directives for refusal. It should be noted that it is not unusual for IT standards to run to several hundred, or even several thousand pages.

Now certainly there are standards that are several pages long. For example, Microsoft likes to bring up the example of ISO 14496, MPEG 4, at over 4,000 pages in length. But that wasn't a Fast Track. And as Arnaud Lehors reminded us earlier, MPEG 4 was standardized in 17 parts over 6 years.

So any answer in the FAQ which attempts to consider what is usual and what is unusual must take account of past practice JTC1 Fast Track submissions. That, after all, was the question the FAQ purports to address.

Ecma claims (PowerPoint presentation here) that there have been around 300 Fast Tracked standards since 1987 and Ecma has done around 80% of them. So looking at Ecma Fast Tracks is a reasonable sample. Luckily Ecma has posted all of their standards, from 1991 at least, in a nice table that allows us to examine this question more closely. Since we're only concerned with JTC1 Fast Tracks, not ISO Fast Tracks or standards that received no approval beyond Ecma, we should look at only those which have ISO/IEC designations. "ISO/IEC" indicates that the standard was approved by JTC1.

So where did things stand on the eve of Microsoft's submission of OOXML to Ecma?

At that point there had been 187 JTC1 Fast Tracks from Ecma since 1991, with basic descriptive statistics as follows:

• mean = 103 pages
• median = 82 pages
• min = 12 pages
  
• max = 767 pages
• standard deviation = 102 pages
  

A histogram of the page lengths looks like this:





So the ISO statement that "it is not unusual for IT standards to run to several hundred, or even several thousand pages" does not seem to ring true in the case of JTC1 Fast Tracks. A good question to ask anyone who says otherwise is, "In the time since JTC1 was founded, how many JTC1 Fast Tracks have been submitted greater than 1,000 pages in length". Let me know if you get a straight answer.

Let's look at one more chart. This shows the length of Ecma Fast Tracks over time, from the 28-page Ecma-6 in 1991 to the 6,045 page Ecma-376 in 2006.




Let's consider the question of usual and unusual again, the question that ISO is trying to inform the public on. Do you see anything unusual in the above chart? Take a few minutes. It is a little tricky to spot at first, but with some study you will see that one of the standards plotted in the above chart is atypical. Keep looking for it. Focus on the center of the chart, let your eyes relax, clear your mind of extraneous thoughts.

If you don't see it after 10 minutes or so, don't feel bad. Some people and even whole companies are not capable of seeing this anomaly. As best as I can tell it is a novel cognitive disorder caused by taking money from Microsoft. I call it "Sinclair's Syndrome" after Upton Sinclair who gave an early description of the condition, writing in 1935: "It is difficult to get a man to understand something when his salary depends upon his not understanding it."

To put it in more approachable terms, observe that Ecma-376, OOXML, at 6,045 pages in length, was 58 standard deviations above the mean for Ecma Fast Tracks. Consider also that the average adult American male is 5′ 9″ (175 cm) tall, with a standard deviation of 3″ (8 cm). For a man to be as tall, relative to the average height, as OOXML is to the average Fast Track, he would need to be 20′ 3″ (6.2 m) tall !

For ISO, in a public relations pitch, to blithely suggest that several thousand page Fast Tracks are "not unusual" shows an audacious disregard for the truth and a lack of respect for a public that is looking for ISO to correct its errors, not blow smoke at them in a revisionist attempt to portray the DIS 29500 approval process as normal, acceptable or even legitimate. We should expect better from ISO and we should express disappointment in them when they let us down in our reasonable expectations of honesty. We don't expect this from Ecma. We don't expect this from Microsoft. But we should expect this from ISO.

Labels: ISO, OOXML, Standards

The world should be pleased to note, that with the approval of ISO/IEC 29500, Microsoft's Vector Markup Language (VML), after failing to be approved by the W3C in 1998 and after being neglected for the better part of a decade, is now also ISO-approved. Thus VML becomes the first and only standard that Microsoft Internet Explorer fully supports.

Congratulations are due to the Internet Explorer team for reaching this milestone!

Now that it has been demonstrated that pushing proprietary interfaces, protocols and formats through ISO is cheaper and faster than writing code to implement existing open standards, one assumes that the future is bright for more such boutique standards from Redmond. Open HTML, anyone?

Labels: Standards

Some thoughts for Document Freedom Day 2008.

Back a few weeks ago in Geneva, OpenForum Europe hosted an evening of mini-talks and a discussion panel with various well-known personalities in our field: Vint Cerf, Bob Sutor, Andy Updegrove and Håkon Lie. I wasn't able to comment on the event at the time, due to my self-imposed blog silence that week, but I'd like to take the opportunity today to carry forward one of the topics discussed then.

I'd like to take as my launching point the theme of Andy Updegrove's talk, which was "Civil ICT Standards". Andy treats this subject more fully on his blog, and also speaks to the topic in his taped interview with Groklaw's Sean Daly.

Thus spake Updegrove:

But as the world becomes more interconnected, more virtual, and more dependent on ICT, public policy relating to ICT will become as important, if not more, than existing policies that relate to freedom of travel (often now being replaced by virtual experiences), freedom of speech (increasingly expressed on line), freedom of access (affordable broadband or otherwise), and freedom to create (open versus closed systems, the ability to create mashups under Creative Commons licenses, and so on.

This is where standards enter the picture, because standards are where policy and technology touch at the most intimate level.

Much as a constitution establishes and balances the basic rights of an individual in civil society, standards codify the points where proprietary technologies touch each other, and where the passage of information is negotiated.

In this way, standards can protect – or not – the rights of the individual to fully participate in the highly technical environment into which the world is now evolving. Among other rights, standards can guarantee:

• That any citizen can use any product or service, proprietary or open, that she desires when interacting with her government.
• That any citizen can use any product or service when interacting with any other citizen, and to exercise every civil right.
• That any entrepreneur can have equal access to marketplace opportunities at the technical level, independent of the market power of existing incumbents.
• That any person, advantaged or disadvantaged, and anywhere in the world, can have equal access to the Internet and the Web in the most available and inexpensive method possible.
• That any owner of data can have the freedom to create, store, and move that data anywhere, any time, throughout her lifetime, without risk of capture, abandonment or loss due to dependence upon a single vendor.

Let us call these “Civil ICT Rights,” and pause a moment to ask: what will life be like in the future if Civil ICT Rights are not recognized and protected, as paper and other fixed media disappear, as information becomes available exclusively on line, and as history itself becomes hostage to technology?

This rings true to me. Technology, computer technology in particular, now permeates our lives. We interact with it daily, from the moment the internet-radio alarm clock goes off, until days end, when we check our email "one last time" before going to bed.

Similarly, the standards that define the interfaces between these devices are also of increasing importance. There was once a time when standards dealt only with the "infrastructure", the stuff in the walls and under the panel floor, or in that funny little locked door off the hallway, with all the cables and flashing lights, where strange men with clipboards would occasionally emerge, accompanied by a poof of cold air and the buzzing of machines.

But today, the technology and the standards that mediate the technology are now directly in front of your face. Think MP3 players. Think DVD's. Think DRM. Think cellular phones. Think web pages. Think encryption. Think privacy. Think documents. Think documents-privacy-security-DRM, your data and what you are allowed to do with it, and what others are allowed to do with it, and whether you control any bit of this in this mad world of ours.

Between you and the tasks that want to do today stands technology and the standards that mediate that technology. Standards are damn important.

Now, although the reach of technology and ICT standards has progressed over the years, the organizations and the processes that create these standards have not always kept up. In many cases standardization remains the creature of big industry with little or no consumer input. It is back-room discussions, where companies connive to see how many patents of their own portfolio they can encumber the standard with. A successful standard is one where no major company is left hungry. Consensus means everyone at the table has been fed. That is the traditional world of technology standards. It brings to mind the famous line from Adam Smith:

People of the same trade seldom meet together, even for merriment and diversion, but the conversation ends in a conspiracy against the public, or in some contrivance to raise prices — The Wealth of Nations (I.x.c.27)

Luckily, there is some hope. The proponents of "open standards" seek standards based on principles of open participation, consensus decision making, non-profit stewardship, royalty-free IP, and free access to standards. The web itself, with the underlying network protocol stack, HTML family of formats with DOM and scripting API's is a shining example of what open standards can accomplish. Tim Berners-Lee says it best, in his FAQ's:

Q: Do you have had mixed emotions about "cashing in" on the Web?

A: Not really. It was simply that had the technology been proprietary, and in my total control, it would probably not have taken off. The decision to make the Web an open system was necessary for it to be universal. You can't propose that something be a universal space and at the same time keep control of it.

But it is important to realize that "control" mechanisms in standards go well beyond IP and organization issues. There are other important factors at play, and we need to address these as well. Knut Blind discusses some of these issues a section called "Anti-Competitive Effects of Standards" from his The Economics of Standards (2004).

The negative impact of standards for competition are mostly caused by a biased endowment with resources available for the standardization process itself. Therefor, even when the consensus rule is applied, dominant large companies are able to manipulate the outcomes of the process, the specification of the standard, into a direction which leads to skewed distribution of benefits or costs in favor of their interests.

In other words, participation in standardization activities is time consuming and expensive, and large companies are much more able to make this kind of commitment than small companies, organizations or individuals. So ,large companies rule the world.

This is especially true with standardization at the international level, where decisions are often made at meetings in very expensive international locations. JTC1 is still discussing what technologies would be required to allow participation in meetings without travel. (Hint — its called a "telephone") To put this in perspective, my week in Geneva cost $3687.52. I flew coach, ate most of my meals on the cheap, often just grabbing hors d'oeuvres at receptions, and I received negotiated IBM corporate rates for air and hotel. This is one JTC1 meeting. What if I wanted to be really active? Add in two SC34 Plenary meeting (Norway/Kyoto). Add in JTC1 Plenary meetings. Add in US NB meetings. Add in US NB membership fees, consortium fees, conferences, etc. This starts adding up, around $40,000/year to participate actively in tech standards, and this doesn't include the cost of my time.

How many small companies are going to pay this amount? How many non-profit organizations? How many individuals? Not many.

But in spite of the expense, in spite of the large company bias of the international standardization system, I saw reason for hope at the Geneva BRM. I saw younger participants, with fire in their bellies. I saw FOSS supports from developing countries. I saw Linux on laptops. I saw participants from FOSSFA, SIUG, EFFI, ODF Alliance Brazil, COSS, etc. They joined their NB's, participated in their NB debates and were appointed to represent their countries in the BRM.

Sure, it is only a foot in the door. One in five BRM participants were Microsoft employees. But it was a hopeful sign. We've planted the seed. We must plant more. And we must see that they grow.

Strength in standards participation comes with time, with participation, with networking, with learning the rules (written and unwritten) learning from others, etc. Just as we have FOSS experts in the software engineering, in law, in business, in training/education, we also need experts in standardization. Certainly the bread and butter participation will be from individual engineers, participating for the duration of a particular proposal or group of proposals. But we also need the institutional linchpin participants, those who have taken on leadership positions within standards organizations, and whose influence is broad and deep.

FOSS also needs a standards agenda. In a world of patent encumbered standards controlling the central networks, open source software dies, and dies quickly. We must protect and grow the open standards, for without them we cease to exist.

What standards are important? Which demand FOSS representation? Remember just a few weeks ago, when there was a lot of concern about how the DIS 29500 BRM added explicit mention of the patent-encumbered MP3 standard, but failed to mention Ogg Vorbis at all? Although I sympathize with this concern, the fact is the BRM could not have added Ogg Vorbis at all, because it is not a standard. Are we willing to do more than lament about this? I tell you that if Ogg Vorbis had been an ISO standard it would have been explicitly added to OOXML at the BRM. Are we willing to do something about it?

What are the standards critical to FOSS, and what are we doing about it? What standards, existing or potential, should we be focusing on? I suggest the following for a start:

1. Ogg Vorbis
2. Ogg Theora
3. PNG, ISO/IEC 15948
   
4. ODF, ISO/IEC 26300
   
5. PDF, ISO 3200
   
6. Linux Standard Base (LSB), ISO/IEC 23360
   
7. Most of the W3C Recommendations
   
8. Most of the IETF RFC's
   

I'm sure you can suggest many others.

Let's put it all together. Some ICT standards directly impact what we can do with our data and our digital lives. These are the Civil ICT Standards. We need to ensure that these standards remain open standards, so anyone can implement them freely. However, the standardization system, both at the national and international levels is biased in favor of those large corporations best able to afford dedicated staff to work within those organizations and develop personal effectiveness and influence in the process. Showing up once a year is not going to work. If FOSS is going to maintain any level of influence in formal standardization world, especially at the high-stakes international level, it needs to find a way to identify, nurture and support participation of "Open Standards Activists". The GNOME Foundation's joining of Ecma, or KDE's membership in OASIS are examples how this could work. Umbrella organizations like Digistan also are critical and can be a nucleus for standards activists. But what about taking this to the next level, to NB membership? Another example is the Linux Foundation's Travel Fund, designed to sponsor attendance of FOSS developers at technical conferences. Imagine what could be done with a similar fund for attendance at standards meetings?

So that is my challenge to you on this first Document Freedom Day. We're near the end of what promises to be one of many battles. The virtual networks of the future are just as lucrative as the railroad and telephone networks of the last century were. These include the network of compatible audio formats, or the network of IM users using a compatible protocol, or the network of users using a single open document format. If FOSS projects and organizations want to secure the value for their users that comes from being part of these networks, then FOSS projects must encourage the use of open standards, and must also encourage and nurture new talent for the next generation of open standards activists.

I'm looking forward to the day, soon, when I can search Google for "open standards activist" and not find a paid Microsoft shill among the listings on the first page.

Labels: Standards

Let's work through a few hypothetical "what if" scenarios to illustrate some common engineering themes related to quality control and the inherent stresses between those who build, those who test, and those who sell. Every engineer is deeply familiar with these patterns, but I believe even the general reader will understand the dynamics better by reading these scenarios.

Let us start by imagining that a new bridge is being built in your area. The company that is building the bridge is very eager to have it open by a particular date. In fact, their contract calls for monetary penalties for every day the opening is delayed beyond that date. However, before it can be opened to traffic, it must be inspected to ensure that the welds conform to the applicable standard. For sake of argument let's say the standard is the AASHTO/AWS D1.5M/D1.5:2002 Bridge Welding Code.

The inspectors may inspect all of the welds and find that they are all acceptable. What do you you think of this, as someone who will soon ride over that bridge? Is this good news? Yes, if you trust the expertise and independence of the inspectors, and their testing process and equipment. If the inspectors do their job properly, and they find no defects, then this indeed is cause for celebration.

But what if the inspectors found a handful of defects, perhaps some welds that failed fatigue testing? If indeed the defects are few, and are localized, then they can be fixed and retested and we can still open the bridge on time. But it is critical that the changes are localized, that there are no far reaching changes. A bridge is not just a collection of independent pieces of metal. They all work together, and as a whole have static and dynamic mechanical properties and relate to load capacity, stresses, thermal characteristics, resonance, etc. Although some fixes may be only localized in their impact, meaning only the area changed needs to be retested, other fixes may have a larger impact and require that everything be retested.

In any complex system, some defects are expected. A sign of good of engineering process is that larger, structural defects are detected or prevented at the earliest possible moment, when they are easiest and least expensive to fix. Where this is not accomplished, large design defects may be first detected at final inspection time, and costly and pervasive rework and retesting may be required, or in the extreme, the bridge may need to be torn down.

The engineering maxim is "fail early". Now this may seem like an odd thing to say. Shouldn't we always try to prevent failure or at least delay it as long as possible? Certainly, if you can prevent failure, then do so. But it is rarely the case where all defects can be prevented. But as engineers, we can design systems, and testing procedures so that flaws become evident as early in the process as possible, when they can be fixed in architecture and design documents rather than in built structures, or at least be found as early in the construction process as possible. This is a frequent source of stress between those who build and those who sell. The important thing for all to understand is that failing early is actually a form of risk reduction. The sooner you fail, the sooner you can fix the defect and start again.

Back to the analogy.

Let's build another bridge. Along comes MegaCorp, who wants to build a bigger bridge, a much bigger bridge than any attempted previously, a MegaBridge. There is nothing wrong with that per se. The history of engineering is the history of making bigger pyramids, wider vaulted ceilings, taller skyscrapers and longer bridges.

Of course, the fact that MegaBridge is right down the street from the new bridge that just opened last week is a bit odd. But MegaCorp tells us that is OK. We're not required to use their bridge if we don't want to.

Further suppose MegaCorp also wants to construct this MegaBridge in record time, faster than others have constructed bridges even a fraction of their size. This is certainly ambitious, but there is no law against ambition. Progress is made by those who are ambitious. We learn from their successes as well as their failures. The important thing is that an ambitious MegaBridge, like any other bridge, is held to the same standards as any other bridge, that proper inspections are carried out and that quality criteria are satisfied.

Months later and the construction of MegaBridge is complete. Time for inspection. But one problem -- the MegaBridge is so large that it is impossible to carry out an inspection in the scheduled time. There are simply not enough inspectors available to carry out the task and complete it by the targeted opening time.

What should we do?

It is useful at this time to consider another engineering maxim, "fail safe". If a system is overloaded, or detects an error condition, it should fail to a safe state, a state least likely to cause damage. We see this applied in many of the systems we use every day. Traffic lights fail safe to flashing red, GFCI circuits fail safe by switching off current if a ground fault is detected, and train air brakes fail safe by applying the breaks if air pressure is lost.

The concept of a "fail safe" applies to processes as well as mechanical systems. A committee, by having a quorum requirement, ensures that it fails to a harmless, inactive state if a snowstorm prevents a representative portion of the committee from attending a meeting. A criminal trial, by presuming innocence and requiring a unanimous verdict to convict, ensures that in case of deadlock, the defendant is let free. Similarly, a bridge quality inspection protocol should include a fail safe provision, that if the inspection cannot be completed, the bridge should not be certified as fit for use. The inspection process should fail safe to non-certification. Ordinarily, engineering practice would be to take whatever time is necessary to inspect the bridge fully, or fail the inspection.

(Here our tale diverges from standard engineering practice and starts to relay, by analogy, the increasingly bizarre tale of OOXML's exploits in and of ISO.)

But MegaCorp wants the MegaBridge to open on time. They force the inspection to continue, even though the inspectors claim there is not enough time. In order to "help" the inspection and despite the obvious conflict of interest, MegaCorp instructs a large number of its own employees, qualified or unqualified, to volunteer as bridge inspectors. They further recruit employees from subsidiaries and suppliers to become inspectors as well. In at least one case, MegaCorp tells a supplier, newly-minted as an inspector, "Don't worry if you know nothing about bridges. We'll tell you what to say. All you need to do is say that the bridge is safe. You'll be rewarded later for helping us here."

So the bridge inspectors go out, old and new, qualified and unqualified and come back with their individual preliminary reports. The older, more experienced inspectors are critical in their evaluation:

The bridge is full of defects. Although, as we mentioned earlier, the mandated schedule did not permit us to test all of the critical welds, of the ones we did test, we found numerous defects. In fact, the number of defects we report is artificially low, since it was limited by our available inspection time. If we had been able to complete a full inspection, we would have detected and reported many more problems.

We further found pervasive structural problems. This bridge is unsound. We can not certify it. We further question why it is necessary to open up a new toll bridge at all, when we just opened up a new free bridge down the street.

The newly-minted inspectors, who for the most part are economically dependent on MegaCorp, were more supportive:

Although some minor problems were indicated, we believe these can all be fixed during routine maintenance. We are not concerned about the time permitted for inspection. We did what the process required. And when you count all the new inspectors that MegaCorp has brought to the process, no bridge has been more inspected. Considering the number of defects reported, this is the most-inspected bridge in history. We recommend that MegaBridge be certified and opened as scheduled.

Of course, from an quality control perspective, this is seriously flawed. The checks and balances between those who build, those who test and those who sell have been eliminated. Although it would not be unusual for some MegaCorp inspectors to be involved in the inspection process, the late arrival of so many unqualified, newly-minted inspectors, and the shift of balance to MegaCorp's hand-picked inspectors, calls into question the independence and technical sufficiency of the entire inspection process.

The inspectors are polled to see whether the bridge can be certified. The vote is close, but the answer is no, the MegaBridge cannot be certified in its current condition. The inspectors, mainly the older, more experienced ones, record a report of 3,522 specific defects in the MegaBridge, far more defects than have ever been found in any other bridge.

MegaCorp is irate. They blast the experienced inspectors in the press, while simultaneously reassuring their stockholders that this setback is just the next step forward to success. They give their engineers the inspection report and demand a quick response. "We must open the bridge on time!" they yell. The MegaCorp engineers work day and night, over weekends, over the holidays even, in order to develop written proposals to address each of the reported flaws in the bridge.

The inspectors are given the proposals and asked whether they believe the proposals are sufficient to allow the MegaBridge to be certified. Although the newly-minted inspectors are quick to affirm the adequacy of the proposal, the old-timers just shake their heads in disbelief, with one stating to the press:

You could fix every last defect in that report and the MegaBridge would still not be sound. Since we never inspected all of the critical welds in the first place, fixing only the defects we reported is insufficient. It is not enough for us to merely retest the ones we reported as defective. We need to test all of them.

Also, the fact that you are making pervasive changes to the road surface, the suspension materials and the pillar diameters, far-reaching design changes which were clearly rushed and have not gone through normal review procedures, I'm afraid that all of our previous tests are now invalidated as well.

Additionally, many of your proposals either avoid addressing the flaws, paper around the flaws, or even introduce new flaws. We need to re-certify the new design before we can even think about retesting the bridge.

However considering the huge number of defects reported, the even larger number of defects undetected because of lack of inspection time, the questionable competency of the newly-minted inspectors, and overt corruption of the process by MegaCorp, my recommendation would be to tear this thing down before it falls over and hurts someone.

Thus ends the tale of what every engineer knows.

Labels: Engineering, OOXML, Standards

An interesting historical anecdote to relate, from our nation's industrial adolescence, a tale with relevance today when we discuss standards.

The year was 1853 and the place was Erie, Pennsylvania, a town at the junction of two incompatible rail gauges. This gauge incompatibility was inefficient and frustrating, but the citizens of Erie loved it, and resisted every attempt to join the emerging common standard gauge in what would be called the Erie Gauge Wars. Why? Let's find out more in the words of industrialist and historian James Ford Rhodes from a passage quoted in extenso from his 1895 History of the United States from the Compromise of 1850 to the McKinley-Bryan Campaign of 1896 (Vol III Pages 18-23). I will follow this by some additional thoughts.

For the prosperity of 1846-57 there were several contributing causes, either special to that period, or then for the first time effective. The greatest of these (whose influence, continuing with ever-increasing momentum to our own day, is still transcendent, and will pervade the future to a degree to which no philosopher can now set bounds) were railroad transportation, beginning its first great era, and the coming into general use of the electric telegraph. We may mark the year 1849 as the commencement of railroad extension. Having less than 6,000 miles January 1, 1849, the country had at the end of 1860 30,635 miles. In 1850 it was impossible to go by direct railway from New York to either Albany or Boston; in 1860 New York had continuous lines reaching beyond the Mississippi. In 1850 Chicago had one short road; in 1860 that city was a great railroad center, her main lines "reaching hundreds of miles — east, west, north, south. In 1850, Ohio, Indiana, and Illinois were open fields; in 1860 they were crossed and recrossed many times." "I arrived here last night," wrote Emerson from Pittsburgh in 1851, "after a very tedious and disagreeable journey from Philadelphia, by railway and canal, with little food and less sleep; two nights being spent in the rail-cars, and the third on the floor of a canal-boat." Not until the end of 1860 did the railway system between the East and the West approach unification and give promise of that consolidation of separate railroads and branches into systems which in our own day has characterized this development.

The primitive ideas in regard to railway travel prevailing in the decade of 1850-60, are well illustrated by what was then known as the Erie Railroad War. The traveler who goes from New York to Chicago in our day is not obliged to set foot from his train, for he is provided with the comforts and conveniences of a hotel. Far different was it in 1853. The traveler could, indeed, then go from New York to Albany in four hours; but there he must change to another road and another train which carried him from Albany to Buffalo, and he esteemed himself fortunate to be able to cover so great a distance in the same car. If he made the western connection at Buffalo it was considered good-luck. The tales of those days are full of complaints of trains behind time, of connections missed, of tedious delays. From Buffalo the traveler had a short run to the station on the line between New York and Pennsylvania, called State Line, where, on account of a difference in gauge, a transfer was necessary. On the broader gauge he could travel twenty miles to Erie, Pa., when he must change again to a road of the Ohio gauge. The train on this railroad carried him to Cleveland; but on the way, if at all late, he was subject again to the anxiety of missing connections. At Cleveland he must hurry to the river, where a scow, carrying at most a dozen passengers and sculled by a weather-beaten mariner, was used as a ferry to take passengers to the Toledo railroad station. In this open boat travelers suffered from exposure to rain and snow; at times the waves ran high and the crossing was attended with danger. If the eastern train was crowded or a few minutes late, haste was necessary to secure passage in the first trip of the scow, for it was well known that the Toledo train started on schedule time and waited for neither train nor boat. At Toledo the traveler made the last change, and — if not more than five minutes behind time — found the Michigan Southern train awaiting him; otherwise he had a tedious delay, which, if his arrival at Toledo happened on a Saturday, might extend to thirty-six hours. The traveler from New York who missed no connections and arrived at Chicago on time had a marvelous story to tell.

The railroad managers of the lines between Buffalo and Erie, eager to improve their route, decided to alter the six-foot gauge of the railway between State Line and Erie to four feet ten inches — the gauge of the roads east of State Line and west of Erie — so that passengers could go from Buffalo to Cleveland without change. The railroad ran a distance through the streets of Erie. The Erie municipal authorities refused to give a permit for making the alteration unless the railway company would agree to stipulations to which its directors, considering them unreasonable, declined to accede. In the contest which followed, a color of law and reason was given to the position taken by Erie; but no one was deceived as to the real ground of the trouble. Erie objected to the change of gauge because the transfer of passengers and freight was deemed important to the borough's prosperity. The wait involved brought custom to her eating-houses; the transfer of freight and live-stock gave work to her people. The populace ignored the legal points and the pretended grounds of demur, but they keenly appreciated the vital objection.

On December 7, 1853, the railroad company began at State Line the work of changing the gauge. The news came quickly to Erie. A cannon was fired to call out the citizens. A large mob assembled, tore up the track, and cut down the railroad bridge in the borough. The infection spread to Harbor Creek, a Pennsylvania town seven miles east of Erie, and that evening its citizens held an orderly meeting and resolved to remove the track of the railroad running on the public highway. The resolution was the next day carried into effect. Two days later (December 10) the track of the new gauge was completed to the borough limits of Erie. That night rioters at Harbor Creek tore up the track, destroyed the bridge, and ploughed up part of the grade of the road. War had begun in earnest. The mayor and the sheriff at times directed the mob, while the local militia, arrayed for service, swelled its number. Even the governor of the commonwealth seemed to sympathize with the Erie people. Certainly they had the sentiment of the whole of Pennsylvania on their side. The United States Circuit Court then granted an injunction restraining all persons from interfering with the railroad company. An Erie justice of the peace pronounced the injunction null and void, and the populace, believing the later decision to be the better law, refused to respect the order of the court. Two days after Christmas, the Harbor Creek bridge was torn down for the fourth time.

The contest attracted the attention of the country. In Buffalo the excitement was intense. Cincinnati held an indignation meeting presided over by Thomas Corwin, to protest against the conduct of the Erie citizens. The New York Tribune said: "Let Erie be avoided by all travelers until grass shall grow in her streets, and till her piemen in despair shall move away to some other city." The press of Philadelphia espoused the cause of Erie. The City of Brotherly Love held a large public meeting to express sympathy with the borough at the other end of the commonwealth. It was declared that "the only protection Erie has to prevent her own ruin is to require the break to be made within her boundaries."

About this time Horace Greeley had occasion to go West. He wrote to his newspaper that he was obliged to ride the seven miles from Harbor Creek to Erie in an open sleigh "through a cutting storm of wind, snow, and sleet.... Let Erie have her way," he continued," and all passengers and freight must change cars before her pie-shops.... The whole world is to be taxed, as in the days of Caesar Augustus, in order that Erie may clutch a sixpence for every dollar of expense she imposes on others. Is it strange that so mean and selfish an exaction should be enforced by mobs, arson, devastation, and ostentatious defiance of judicial mandates?"

With the new year the excitement grew. The Erie people became vindictive. They warned the president and director of the railroad company, living at Erie, to leave the borough. Women joined the rioters and assisted in the work of destruction of the bridges. The New York Tribune called upon President Pierce to interfere, and suggested that he issue a proclamation and call out troops in order that the laws might be executed. "Had a runaway negro," this journal said, "been somehow mixed up with the matter, we should have had half of the United States army in Erie a month ago." The trouble brought into view the rivalry between New York and Philadelphia, between New York State and Pennsylvania. The Commonwealth of Pennsylvania was declared an accomplice with the Erie rioters and bridge-burners, for the purpose of diverting business from the West to the seaboard through her territory and to her port by a projected line from Erie to Philadelphia, and an appeal was made to the West to frustrate her purpose. It is possible that the sentiment of the West had some influence in bringing about a settlement; but in the early part of January, 1854, the minds of Northern men became engrossed with the proposed repeal of the Missouri Compromise, and the Erie war ceased to attract attention. An act of the Pennsylvania legislature, by a fair compromise, brought the trouble to an end. The railroad company, having consented to confer certain desired advantages upon Erie and Pennsylvania, was allowed to complete the change of gauge and run its trains through Erie without molestation.

So, what do we have here? Two, incompatible standards meet in Erie. Maintaining the two standards is wasteful, inefficient and expensive to the railroad owners, and causes great inconvenience to the travelers who pass through Erie. Everyone knows this. However, the shopkeepers and laborers of Erie benefit from directly from these circumstances. This situation is common. We don't all benefit from the same things. Some people benefit from other people's misfortune. War is good for arms dealers. Hurricanes are good for the glazier. Tuberculosis is good for the undertakers. And two incompatible rail gauges are good for those whose livelihood depends on jobs manually loading and unloading goods at the junction, or providing refreshments to those who must wait for connecting trains. In any change, there are winners and losers, those who benefit from maintaining the status quo, no matter how inefficient it is, and those who will benefit from change.

So what do you do? Choose a commercial interest? Line up behind either the rail barons or the teamsters? Not necessarily. In the end the competing commercial interests don't amount to a hill of beans compared to the interests of the travelers, the consumers and the public at large. They are the ones who were absorbing the cost of maintaining two rail gauges, in shipping costs and frustration, and they are the ones who benefited from the convergence on a single standard. Sure there were two commercial interests vying for supremacy, but there was also a choice that was clearly better for the public.

So whenever anyone tells you that two office document format standards is a good idea, that plugins and convertors can deal with shuffling data back and forth between two formats, ask yourself some basic questions:

1. Who benefits from having two standards? Who are the "pieman of Erie" that will make money off of the inefficiency that results?
2. Who has the most to lose if Erie becomes just another railroad town, one of many?
3. How far will the piemen of Erie go to protect their monopoly? What technological advances and innovations will they prevent or actively destroy in order to preserve their exclusive access to their customers?
   
4. And most important, what about the interests of the public at large? What is good policy?
   

If Microsoft were fighting the Erie Gauge War, they would portray it as a fight between the good townspeople of Erie and the rail barons. The big bad mean old rail barons are forcing their single gauge standard on people who don't want it. Why not give them the choice of gauges? Only the old broad gauge is 100% backwards compatible with the cars that were designed for it. In fact legislation should be passed for force all railroad towns to support both gauges so everyone can have the choice. Piemen of the word, unite !

That is the distortion you get if you look at a standards war through the narrow blinders of commercial interest. But if you look at the full market impact, the simple economics of it, it becomes a lot clearer. What brings greater efficiency, greater fidelity, greater innovation and lower costs? Having two incompatible document format standards? Or having a single harmonized document format standard? Fighting against economics is like fighting against gravity or the 2nd Law of Thermodynamics. You are going to lose in the end. The piemen of Erie, and their modern counterparts, are on the wrong side of economics, and history,

Labels: Standards

It must have passed beneath my radar it when it first was filed in 2004, but it caught my eye recently when Andy Updegrove mentioned it in Chapter 3 of his book-in-progress, The War of the Words. I'm talking about Novell's November 2004 antitrust complaint against Microsoft, filed shortly after settling an different, OS-related, complaint with Microsoft for $536 million. You can view the second complaint, which I'll call the "WordPerfect" complaint, here [PDF] on GrokLaw.

What is interesting to me, and why this "old news" is worth talking about, is the analysis Novell made in their complaint of Microsoft's treatment of document format standards. The concerns of 2004 (or 1995 even) are very similar to the concerns of 2007. Let's go through Novell's argument and see where it leads us.

91. As Microsoft knew, a truly standard file format that was open to all ISVs would have enhanced competition in the market for word processing applications, because such a standard allows the exchange of text files between different word processing applications used by different customers. A user wishing to exchange a text file with a second user running a different word processing application could simply convert his file to the standard format, and the second user could convert the file from the standard format into his own word processor's format. This, a law firm, for instance, could continue to use WordPerfect (which was the favorite word processor of the legal profession), so long as it could convert and edit client documents created in Microsoft Word, if that is what clients happened to use...

This is a good statement of the benefits of an open document standard. Note that Novell is not arguing that the benefit of a standard is to get information in or out of a single vendor's product, like Microsoft Office. The benefit is that a standard provides for interchange between any pair of word processors.

...Microsoft knew that if it controlled the convertibility of documents through its control of the RTF standard, then Microsoft would be able to exclude competing word processing applications from the market and force customers to adopt Microsoft Word, as it soon did.

Note also that Novell is not complaining here about Microsoft's control of the binary DOC format (and its many variations). Instead, what Novell complains about is Microsoft's control over the document exchange format RTF, or Rich Text Format, used in those days to exchange data between word processors. He who controls RTF, controls document exchange, controls vendor lock-in and has the sole means of improving the fidelity of document exchanges.

In fact, Microsoft claimed that RTF addressed this very concern -- document exchange in a cross-platform, cross-application fashion, as stated in the introduction to version 1.0 of their self-styled "standard":

The RTF standard provides a format for text and graphics interchange that can be used with different output devices, operating environments, and operating systems. RTF uses the ANSI, PC-8, Macintosh, or IBM PC character set to control the representation and formatting of a document, both on the screen and in print. With the RTF standard, you can transfer documents created under different operating systems and with different software applications among those operating systems and applications

It should have been obvious at the time that vesting exclusive control of an interoperability interface in a single company was a bad idea. But I guess the world didn't realize what dealing with Microsoft meant. But we know better now. So why are we making the same mistakes in 2007?

Those who control the exchange format, can control interoperability and turn it on or off like a water faucet to meet their business objectives. I don't know how many people noticed the language in Microsoft's press release announcing their sponsored interoperability track at XML 2007 a few weeks ago:

In its approach, Microsoft strives to bring technologies to market in a way that balances competitive innovation with the real interoperability needs of customers and partners.

Let that sink in for a minute. Microsoft is saying that they need to balance interoperability and profit. (Their profit, not yours) They can't maximize for both simultaneously. They need to trade one off for the other.

Continuing with Novell's 2004 complaint:

92. The specifications for RTF were readily available to Microsoft's applications developers, because RTF was the format they themselves developed for Microsoft's office productivity applications. Microsoft withheld the RTF specifications from Novell, however, forcing Novell to engage in a perpetual, costly effort to comply with a critical "industry standard" that was, in reality, nothing more than the preference of its chief competitor, Word. Indeed, whenever Word changed its own file format, Microsoft unilaterally and identically changed the RTF standard for Windows, forcing Novell and other ISVs constantly to redevelop their applications. In this manner, Microsoft gave Word a permanent, insurmountable lead in time-to-market and made document conversions difficult for users otherwise interested in running non-Microsoft applications. Many WordPerfect users were thus forced to switch to Microsoft Word, which predictably monopolized the word processing market....

So, the RTF standard was just a dump of Word's features, done when and how Microsoft felt like doing it. As one wag quipped, "RTF is defined as whatever Word saves when you ask it to save as RTF."

This should sound familiar. OOXML is nothing more than the preferences of Microsoft Office. Whenever Word changes, OOXML will change. And if you are a user or competitor of Word, you will be the last one to hear about these changes. ISO does not own OOXML. Ecma does not own OOXML. OOXML, in practice, is controlled and determined solely by the Office product teams at Microsoft. No one else matters.

Consider that Microsoft has recently proposed over 1,700 changes to the OOXML specification, including fixes that presumably will be made into a future Office 2007 fixpack. Microsoft knows what these fixes will be. The Office developer teams know what these fixes will be. But if you are a competitor of Microsoft's in this space, do you know what these changes are? No. Microsoft has decided to keep them a secret, claiming that the ISO process allows them to withhold interoperability information from competitors in what they maintain is an "open standard".

Further, the coding of Office 14 a.k.a. Office 2009 is well underway. Beta releases are expected in early 2008. But are file format changes needed to accommodate the new features being discussed in Ecma? No. Are they being discussed in ISO? No. Are they being discussed anywhere publicly? No.

Is this how an open standard is developed?

My prediction is that the first time anyone hears about what is in the next version of OOXML will be when Office 14 Beta 1 is announced. Other vendors will not hear a word about the format changes until after the Beta 1 is already released. Not even Ecma will hear about the changes until after then.

DIS 29500 is already obsolete, has already been embraced and extended. You just don't know about it yet. You weren't meant to know. In fact, pretend you don't know. Give Microsoft a big head start. They need it.

Further from the Novell complaint:

93. ...As in the case of of RTF, Microsoft forced Novell to delay its time-to-market while redeveloping its applications to an inferior standard. Because these standards were lifted directly from Microsoft's own applications, those applications were always "compatible" with the standards.

And that is the key, isn't it? By owning the "standard" and developing it in secret, without participation from other vendors, in an Ecma rubber-stamp process, Microsoft rigs the system so they can author an ISO standard with which they are effortlessly compatible, while at the same time ensuring that their products maintain an insurmountable head start in implementing these same standards. There is no balance of interests in OOXML. It is entirely dictated by Microsoft, and voted on, in many cases, by their handpicked committees in Ecma and ISO.

So much for Novell's complaint from 2004. I'm told that this is still case is suspended as of November, 2007, as the two parties pursue mediation. A status report on that mediation is due to Judge Motz by January 11th, 2008. Maybe we're hear more then.

Looking at this long history of standards abuse by Microsoft, in the file format arena and elsewhere, I'm drawn to take a broader view of this controversy. It is not really a battle between ODF and OOXML. It isn't even really a battle between OOXML and ISO. It is, in the end, a battle between having document standards and not having them. Microsoft is trying to dumb down the concept of standards and interoperability to a point where these concepts are meaningless and ineffective. This is not because they want to support standards more easily in their products. No, it is because they do not want standards at all.

Remember, standards bring interoperability, the ability to try out new tools and techniques, the ability to migrate, the ability to chose among alternatives, the ability even to run non-Microsoft products. If standards are meaningless and ineffective, then the incumbent' vendor lock-in will win every time. At that point, isn't it convenient for them to have a monopoly in operating systems and productivity applications? This, in my opinion, is the essence of Novell's 2004 complaint, Opera's present complaint, and the ongoing file format debate. Microsoft's monopoly power and the resulting network effects have lead to a relationship with standards where they win by winning, by drawing, or even by cheating so much that they discredit the system.

Labels: OOXML, Standards

So what do we have here?

Sixteen men, lined up. They seem to be having a good time. Some are older, some younger. A historian of fashion might be able to tell us their relative social status, and perhaps their trade, by looking at their clothing. In the background, three men are observing and comparing notes. To the right is a church, and to the left is the village.

So what are they doing?

Is it an early early depiction of the hokey-pokey ("You put your left foot in...") ?

No.

Although the scene obviously has some social aspects, the primary activity depicted here is standards development, particularly the historically mandated procedure for determining the linear measurement known as the "rood", related to the English "rod", the German "rute" and the Danish "rode".

This print, from a 16th century surveyor's manual by Jacob Koebel, called Geometrei. Von künstlichem Feldmessen und absehen, explains the procedure:

Stand at the door of a church on a Sunday and bid 16 men to stop, tall ones and small ones, as they happen to pass out when the service is finished; then make them put their left feet one behind the other, and the length thus obtained shall be a right and lawful rood to measure and survey the land with, and the 16th part of it shall be the right and lawful foot.

From a technical point of view, you might wonder why they didn't have a standard rule, a metal bar etched with two lines, something tangible which could be carried about and used to calibrate? But who would maintain the standard? And would you trust them? Physical objects may be counterfeited, replaced, shaved, distorted, even stolen. Those who are buying land would like a longer rood, and those selling land would like a shorter rood, so the motivation for fraud is clear.

But the average length of the feet of 16 random men -- that is probably not going to change much in a given town, or even across a country. Compared to the logistics required to create, duplicate and distribute a standard rule, the described statistical approach is easier to administer and was accurate enough for the time.

But there is more to it than that. Why didn't the surveyor just measure his own feet? Or those of his friends? And why require that it be done at church? Why not wherever the surveyor wants to do it?

There must have been something about the process itself, the lining up and being measured, publicly, neighbor beside neighbor, next to the church, that lent it legitimacy. These men are literally voting with their feet.

The transparency of the process is also notable. The rood was determined in public, at the time and place most likely to offer everyone in the town the opportunity to observe. It is hard to cheat with the public watching. Anyone there trying to wear clown shoes or going barefoot would be immediately detected.

Also, it is notable that participation was on an equal basis. No one was able to say, "I am a rich merchant, so I should be allowed to bring 5 pairs of my shoes and line them up in front of me". And certainly no one could say, "I am the King, the standard is determined by my foot and my foot alone". This is good, because the variation from King to King would tend to be much greater than the variation from different random samplings of 16 men.

When we talk about ISO meaning "equal" in Greek, and the ideal of one-country, one-vote, and the corruption this system has fallen into, I think of this picture and sigh.

Labels: Standards

We typically use words to communicate, to be understood. That is the common case, but not the only case. In some situations, words are used like metes and bounds to carefully circumscribe a concept by the use of language, in anticipation of another party attempting a breach. This is familiar in legislative and other legal contexts. Your concept is, "I want to lease my summer home and not get screwed," and your attorney translates that into 20 pages of detailed conditions. You can be loose with your language, so long as your lawyer is not.

But even among professionals, the attack/defense of language continues. One party writes the tax code, and another party tries to find the loopholes. Iteration of this process leads to more complex tax codes and more complex tax shelters. The extreme verbosity (to a layperson) of legislation, patent claims or insurance policies results from centuries of cumulative knowledge which has taught the drafters of these instruments the importance of writing defensively. The language of your insurance policy is not there for your understanding. Its purpose is to be unassailable.

This "war of the words" has been going on for thousands of years. Plato, teaching in the Akademia grove, defined Man as "a biped, without feathers." This was answered by the original smart-ass, Diogenes of Sinope, aka Diogenes the Cynic, who showed up shortly after with a plucked chicken, saying, "Here is Plato's Man." Plato's definition was soon updated to include an additional restriction, "with broad, flat nails." That is how the game is played.

In a similar way Microsoft has handed us all a plucked chicken in the form of OOXML, saying, "Here is your open standard." We can, like Plato, all have a good laugh at what they gave us, but we should also make sure that we iterate on the definition of "open standard" to preserve the concept and the benefits that we intend. A plucked chicken does not magically become a man simply because it passes a loose definition. We do not need to accept it as such. It is still a plucked chicken.

(This reminds me of the story told of Abraham Lincoln, when asked, "How many legs does a dog have if you call the tail a leg?" Lincoln responded, "Four. Calling a tail a leg does not make it a leg.")

With the recent announcement here in Massachusetts that the ETRM 4.0 reference architecture will include OOXML as an "open standard" we have another opportunity to look at the loopholes that current definitions allow, and ask ourselves whether these make sense.

The process for recommending a standard in ETRM 4.0 is defined by the following flowchart:



So, let's go through the first three questions that presumably have already been asked and answered affirmatively in Massachusetts, to see if they conform to the facts as we know them.

1. Is the standard fully documented and publicly available? Can we really say that the standard is "fully documented" when the ISO review in the US and in other countries is turning up hundreds of problems that are pointing out that the standard is incomplete, inconsistent and even incorrect? We should not confuse length with information content. Just as a child can be overweight and malrnourished at the same time, a standard can be 6,000 pages long and still not be "fully documented." Of course, we could just say, "A standard fully documents the provisions that it documents" and leave it at that. But such a tautological interpretation benefits no one in Massachusetts. We should consider the concept of enablement as we do when prosecuting patent applications. If a standard does not define a feature such that a "person having ordinary skill in the art" (PHOSITA) can "make and use" the technology described by the standard without "undue experimentation" then we cannot say that it is "fully documented." By this definition, OOXML has huge gaps.
   
2. Is the standard developed and maintained in a process that is open, transparent and collaborative? We're talking about Ecma here. How can their process be called transparent when they do not publicly list the names of their members or attendance at their meetings, do not have public archives of their meeting minutes, their discussion list or document archive, do not make publicly available their own spreadsheet of known flaws in the OOXML specification nor of the public comments they received during their public review period? How is this, by any definition, considered "transparent"? We can also question whether the process was open. When the charter constrains the committee from making changes that would be adverse to a single vendor's interests, it really doesn't matter what the composition of the committee is. The committee's hands are already tied and should not be considered "open." If I were writing a definition of an open, transparent process, I'd be sure to patch those two loopholes.
3. Is the standard developed, approved and maintained by a Standards Body? Without further qualifying "Standards Body" this is a toothless statement. As should be apparent right now, not all SDO's are created equal. Some of the standards equivalent of diploma mills. Accreditation is the way we usually solve this kind of problem. Ecma's Class A Liaison status with JTC1 is not an accreditation since their liaison status has no formal requirements other than expressing interesting in the technical agenda of JTC1. In comparison, OASIS needed to satisfy a detailed list of organizational, process, IPR and quality criteria before their acceptence as a PAS Submitter to JTC1/SC34. Why bother having a requirement for a Standards Body unless you have language that ensures that it is not a puppet without quality control?
   
4. Is there existing or growing industry support around the use of the standard? Again, very vague. A look at Google hits for OOXML documents shows that there are very few actually in use. My numbers show that only 1 in 10,000 new office documents are in OOXML format. But I guess that is more than 0 in 10,000 that existing last year. But is this really evidence for "growing industry support"? I'd change the language to require that there be several independent, substantially full implementations.
   

There are two additional questions which I won't presume to answer since they rely more on integration with internal ITD processes.

We learn lessons and move on to the next battle. Just as GPLv2 required GPLv3 to patch perceived vulnerabilities, we'll all have much work to do cleaning up after OOXML. Certainly JTC1 Directives around Fast Tracks will need to be gutted and rewritten. Also, the vague and contradictory ballot rules in JTC1, and the non-existent Ballot Resolution Meeting procedures will need to be addressed. I suggest that ITD take another look at their flowchart as well, and try to figure out how they can avoid getting another plucked chicken in the future.

Labels: Ecma, OOXML, Standards

This was my response to the call for public comments on the Information Technology Division's (ITD) Enterprise Technical Reference Model (ETRM) 4.0 draft.

---

I’d like to write to you as a long-time Massachusetts resident and taxpayer. My employer (IBM) will likely submit their own comments, but I’d like to offer you my own personal views on the ETRM 4.0 draft.

I am proud of the Commonwealth’s tradition of openness in government, enshrined in our Public Records Law and Open Meeting Law. As James Madison wrote, “A popular government, without popular information, or the means of acquiring it, is but a prologue to a farce or a tragedy. A people who mean to be their own governors must arm themselves with the power which knowledge gives them.” So access to government documents, now and for posterity, is critical for public oversight and participation in government, as well as for preserving our heritage. Now that we’ve moved into the digital age, access to government documents requires that these documents be made available in a format that all Commonwealth residents can read. So the move toward open documents formats, as called for in the ETRM, is laudable. A citizen must never be dependent on any single vendor for the software needed to read their government’s documents.

However, I am concerned at the proposed addition of Ecma Office Open XML (OOXML) to the list of acceptable document formats. As you may have heard, OOXML is currently undergoing review by ISO/IEC JTC1 for possible approval as an ISO standard. As part of this review, technical committees in standards bodies around the world are reviewing OOXML and appraising it’s suitability as an International Standard. As a participant in the US committee reviewing OOXML, INCITS V1, I had the opportunity to review the text of the OOXML specification and to discuss it with others. I am sorry to report that I found the OOXML specification to be full of errors and omissions. Of course, no technical document is perfect. But this one, in particular, is of far greater length (more than 6,000 pages) and of far lower quality than any I have seen before. If it has advanced this far in the ISO process it is because of vendor pressure, not because of technical merit.

What is the problem with a buggy standard? Interoperability suffers. That is the problem. There is no doubt that if everyone in the Commonwealth used Microsoft Office 2007 on Windows Vista, that their interoperability will be good. But as soon as we admit choice in applications and operating systems, then interoperability will only occur when all sides follow a common standard. So the technical quality of a standard (accuracy, comprehensiveness, level of detail, consistency, etc.) is directly proportional to the level of interoperability achievable and the cost to achieve it.

The ISO ballot on OOXML will not end until September 2nd, after which a resolution process to fix defects in the text of the standard will take at least an additional 6-18 months. That is, of course, if OOXML gains ISO approval, something which is not certain at this point. So I would recommend a cautious approach, and wait for the ISO process to conclude, or conduct your own independent technical evaluation of the OOXML specification to confirm its technical quality before adding OOXML to your list. Ask other vendors: Is this something you can implement? Ask yourself: Will this truly give the Commonwealth the interoperability and choice that you desire? These are important questions to ask.

Finally, I’d note that the ETRM also calls out OpenDocument Format (ODF) as an acceptable format. ODF was approved by ISO last year. So why do we need OOXML? I personally think that the complexity of document exchange and translation in a multi-format world would take us back to the confusion and frustration of the early 1990’s when we all juggled WordStar, WordPerfect, Word and WordPro files, and could collaborate only poorly. Better to push for a single unified/harmonized standard document format for personal productivity applications, much as we have a single standard (HTML) for web pages.

I’ll leave you with a quote from Tim Berners-Lee, the inventor of the web, from an interview he gave with David Berlind from ZDNet when Berners-Lee was recently in Boston receiving a Lifetime Achievement Award from the Massachusetts Innovation & Technology Exchange.

Berners-Lee said:

It was the standardization around HTML that allowed the web to take off. It was not only the fact that it is standard, but the fact that it’s open and the fact that it is royalty-free.

So what we saw on top of the web was a huge diversity and different business which are built on top of the web given that it is an open platform.

If HTML had not been free, if it had been proprietary technology, then there would have been the business of actually selling HTML and the competing JTML, LTML, MTML products. Because we wouldn’t have had the open platform, we would have had competition for these various different browser platforms, but we wouldn't have had the web. We wouldn't have had everything growing on top of it.

So I think it very important that as we move on to new spaces ... we must keep the same openness we that had before. We must keep an open internet platform, keep the standards for the presentation languages common and royalty free. So that means, yes, we need standards, because the money, the excitement is not competing over the technology at that level. The excitement is in the businesses and the applications that you built on top of the web platform.

I believe we want to ensure the same qualities in document formats. We want competition and choice among vendors, applications and services, but not among standards. If we compete on standards, then no one wins.

Labels: ODF, OOXML, Standards

Here in Westford, Massachusetts, some of our public schools have boilers that can be powered by natural gas or heating oil. This way the schools can have their choice of fuel, which they can alter year to year, or even month to month according to the comparative prices of these two commodities. Such a choice has a value, a very tangible value at any given point. For example, suppose that today the price of natural gas was $1.15/therm (100,000 BTU's) and the price of heating oil was $1.72/therm. The value of choice is ($1.72-$1.15)*# of therms purchased. Those clever with finance could probably estimate the long-term value by pricing the analogous commodities futures.

Of course, choice here has a cost as well, namely the increased cost to purchase and maintain the more complex boiler that offers the choice of gas or oil. If the value of having the choice is worth more than the cost of maintaining a world that offers that choice, then you have a net gain by preserving the choice. Otherwise, you are losing by having choice. It is odd to hear that, isn't it? You can lose by having choice, if the cost of maintaining that choice is greater than the benefit from having a choice.

For example, take shoe sizes. In the U.S. we buy shoes in 1/2 size increments. In theory a store could offer shoes in 1/10 size increments. This would give you, the consumer, an increase in choice, and this choice would have a distinct value to you. Whereas the previous shoe sizes would be an average of 1/4 of a size away from perfect fit, the new shoes would be on average only 1/20th of a size away. So a tangible benefit to you the consumer. But this comes at a cost, since the larger inventory and slower turnover for the retailer would increase their costs. Since we are unlikely to buy more shoes than we do today, this cost increase would be passed on to the consumer. So in this case, the benefit of better fitting shoes is not seen to be worth the increased costs to maintain those choices, so the industry remains with 1/2 size shoe increments.

As an aside, I'll give you another example, as a brainteaser. You are walking down a street evenly lined with many stores, all of which sell some commodity, let's say orange juice. The prices at the various stores are random. You want to buy orange juice at the best price, but you can only make one purchase, and you can only make one pass down the street. So you can look at many prices, but at some point you need to make a decision and purchase the orange juice, and you can't turn back or make a second choice once you've made a purchase. The street is 1 kilometer long. Where do you buy your orange juice? Even with an abundance of choice, it isn't always clear how you make an optimal decision. Note that many life decisions are like this, since time acts as a one-way street, where often we must make an important choice, based on the info we have so far, but with uncertain knowledge of the future, and often we can only choose once.

So what does this mean for document formats? It is popular these days to use the word "choice" as a "god term", a phrase introduced by Richard Weaver in The Ethics of Rhetoric, referring to words like "progress", "culture" and "for the Fatherland" that are used to appeal more by seduction than by rational argument. But we should avoid the seduction and ask ourselves what this choice really means. What is it really worth to you and your business? Sitting down today, writing a document, or creating a spreadsheet, what is the value to you, knowing that you could save a document to ODF, OOXML, UOF, SmartSuite, WordPerfect Suite format, etc.? And what is the tangible value of having that choice, that option?

What I want in a document format is:

1. It is supported by my word processor.
2. When I save the document and later retrieve it, the document looks and behaves the same.
3. When I give it someone else, who may be using the same or a different word processor, on the same or a different operating system, it looks and behaves the same.
4. It is easily processable by other software tools. I care about this directly because I am a programmer. But even if I were not, I would want this characteristic, since this is what ensures that an ecosystem of other tools will emerge to support the format, offering me more choice.
5. I want the format to be open for the same reason, so it encourages the creation of other tools that I may later choose to use.
6. I want the format to be controlled by a group of vendors and other interests, not dominated by a single player. Further, I'd want them to be to be working openly and transparently, so the public can all see what they are doing. We should all remember the line by Adam Smith, "People of the same trade seldom meet together, even for merriment and diversion, but the conversation ends in a conspiracy against the public." The remedy is given by Justice Louis Louis Brandeis in his line, "Sunlight is the best disinfectant."
   
7. I want the format to be well-designed according to industry best practices, since I know that will make it easier to work with for tools vendors and will help ensure its longevity as a format.
   

Given a single format that can accomplish these goals, I see zero value in having a second standard. In fact, having multiple formats brings increased complexity and expense to the software vendor who maintains and supports all the translator code and this expense gets passed on to the consumer. And then there is the opportunity cost of the features that may have been coded if my vendor hadn't been distracted by writing translator code. Also, there is the cost, in performance and fidelity loss when translating between formats, and the resulting business losses that may be caused by errors introduced in this processing. This is all very real. But where is the benefit?

To solve this puzzle, we need to look at it from Microsoft's perspective. A standard in this space is a very scary proposition for them. A comparison can be made to the early years of the automobile industry:

Between 1904 and 1908, more than 240 companies entered the fledgling automotive business. In 1910 there was a mini-recession, and many of these entrants went out of business. Parts suppliers realized that it would be much less risky to produce parts that they could sell to more than one manufacturer. Simultaneously, the smaller automobile manufacturers realized that they could enjoy some of the cost savings from economies of scale and competition if they also used standardized parts that were provided by a number of suppliers.

Guess which two players were not interested in parts standardization? The two largest companies in the industry: Ford Motor Company and General Motors. Why? Because they were well able to achieve strong economies of scale in their own operations, and had no interest in "interconnecting'' with anyone else: standardization would (partially) level the playing field regarding economies of scale at the component level. As usual, then and now, standardization benefits entrants, complementors, and consumers, but may hold little interest for dominant incumbents. — Carl Shapiro and Hal R. Varian, Intro for Managing in a Modular Age

We're in a very similar situation now. Microsoft, the sole dominant player in this market, is perfectly happy with having total control over their proprietary formats. It has worked very well for them for many years. But just as Ford and GM eventually gave in to the obvious necessity of true interoperability, Microsoft will as well. The companies that win in this world are the ones that adapt, not the ones that sell adapters.

We need to start talking about what we can do to ensure that we have a single open document format that can be used by everyone. Making a second ISO standard for document formats is a bad idea. What we need to do is continue to evolve ODF, continue the work to harmonize UOF and ODF, and also take on the task of harmonizing OOXML and ODF. The value of having a single standard in this space is clear. We just need to remain vigilant in the face of those commercial interests that would stand to lose the most if customers had true document portability and could choose platforms and applications based on features and price and support, and not solely on fears, uncertainty and doubt about whether they could still access their legacy documents.

Labels: Economics, OOXML, Standards

Some "short notes" to share with you:

From a GrokLaw news pick we hear that ZDNet's David Berlind recently interviewed Tim Berners-Lee in Boston, where Sir Tim received the Massachusetts Innovation and Technology Exchange's Lifetime Achievement Award. Watch the whole interview if you have 12 minutes, though I will transcribe one passage which highlights the importance of agreeing on a single open standard for a problem domain and fostering competition among the applications built upon that standard:

It was the standardization around HTML that allowed the web to take off. It was not only the fact that it is standard, but the fact that its open and the fact that it is royalty-free.

So what we saw on top of the web was a huge diversity and different business which are built on top of the web given that it is an open platform.

If HTML had not been free, if it had been proprietary technology, then there would have been the business of actually selling HTML and the competing JTML, LTML, MTML products. Because we would''t have had the open platform, we would have had competition for these various different browser platforms, but we wouldn't have had the web. We wouldn't have had everything growing on top of it.

So I think it very important that as we move on to new spaces ... we must keep the same openness we that had before. We must keep an open internet platform, keep the standards for the presentation languages common and royalty free. So that means, yes, we need standards, because the money, the excitement is not competing over the technology at that level. The excitement is in the businesses and the applications that you built on top of the web platform.

Well said. I tried to make a similar point, but with pictures, back in February.

I recently ordered some podcasting equipment. It should arrive tomorrow. I will be looking for people to interview soon. So hide while you can, don't answer the phone, and if it looks like I'm carryin