Sunday, February 25, 2007
Essential and Accidental in Standards
The earliest standards were created to support the administration of the government, which in antiquity primarily consisted of religion, justice, taxation and warfare. Crucial standards included the calendar, units of length, area, volume and weight, and uniform coinage.Uniform coinage in particular was a significant advance. Previously, financial transactions occurred only by barter or by exchanging lumps of metals of irregular purity, size and shape, called “aes rude”. With the mass production of coins of uniform purity and weight imprinted with the emperor's portrait, money could now be exchanged by simply counting, a vast improvement over having to figure out the purity and weight of an arbitrary lump of metal. Standards reduced the friction of commercial transactions.
Cosmas Indicopleustes, a widely-traveled merchant, later a monk, writing in the 6th Century, said:
The second sign of the sovereignty which God has granted to the Romans is that all nations trade in their currency, and in every place from one end of the world to the other it is acceptable and envied by every other man and every kingdom
“You can see a lot just by observing,” as Yogi Berra once said. A coin can be read much like a book. So, what can you see by reading a coin, and what does this tell us about standards?
To the left are examples from my collection of a single type of coin. The first picture shows the the obverse of one instance, followed by the reverse of eight copies of the same type.
The legend on the observe is “FLIVLCONSTANTIVSNOBC”. The text is highly abbreviated and there are no breaks between the words as is typical in classical inscriptions whether on coins or monuments. Brass or marble was expensive so space was not wasted. We can expand this inscription to “Flavius Julius Constantius Nobilissimus Caesar” which translates to “Flavius Julius Constantius, Most Noble Caesar”.
So this is a coin of Constantius II (317-361 A.D.), the middle son of Constantine the Great. The fact that he is styled “Caesar” rather than “Augustus” indicates that this coin dates from his days as heir-designate (324-337), prior to his father's death. We know from other evidence that this type of coin was current around 330-337 A.D.
There is not much else interesting on the obverse. Imperial portraits had become stylized so much by this period that you cannot really tell one from the other purely by the portrait.
The reverse is a bit more interesting. When you consider that the such coins were produced by the millions and circulated to the far corners of the empire, it is clear the coins could have propaganda as well as monetary value. In this case, the message is clear. The legend reads “Gloria Exercitus” or “The Glory of the Army”. Since the army's favor was usually the deciding factor in determining succession, a young Caesar could never praise the army too much. Not coincidentally, Constantius's brothers, also named as Caesars, also produced coins praising the army before their father's death.
At bottom of the reverse, in what is called the “exergue”, is where we find the mint marks, telling where the coin was minted, and even which group or “officina” within the mint produced the coin. From the mint marks, we see that these particular coins were minted in Siscia (now Sisak, Croatia), Antioch (Antakya, Turkey), Cyzicus (Kapu-Dagh, Turkey), Thessalonica (Thessaloníki, Greece) and Constantinople (Istanbul, Turkey).
The image on the reverse shows two soldiers, each holding a spear and shield, with two standards, or “signa militaria”, between them. The standard was of vital importance on the battle field, providing a common point by which the troops could orient themselves in their maneuvers. These standards appear to be of the type used by the Centuries (units of 100 men) rather than the legionary standard, which would have the imperial eagle on top. You can see a modern recreation of a standard here.
If you look closely, you'll notice that soldiers on these coins are not holding the standards (they already have a spear in one hand and a shield in the other), and they lack the animal skin headpiece traditional to a standard bearer or “signifer”. So this tells us that these soldiers are merely rank and file soldiers encamped, with the standards stuck into the ground.
If you compare the coins carefully you will note some differences, for example:
- Where the breaks in the legend occur. Some break the inscription into “GLOR-IAEXERC-ITVS”, while others have “GLORI-AEXER-CITVS”. Note that neither match word boundaries.
- The uniforms of the soldiers differ, in particular the helmets. Also the 2nd coin has the soldiers wearing a sash that the other coins lack. This may reflect legionary or regional differences.
- The standards themselves have differences, in the number of disks and in the shape of the topmost ornament.
- The stance of the soldiers varies. Compare the orientation of their spear arm, the forward foot and their vertical alignment.
Despite the above differences, to an ancient Roman, all of these coins were equivalent. They all would have been interchangeable, substitutable, all of the same value. Although they differed slightly in design, they matched the requirements of the type, which we can surmise to have been:
- obverse portrait of the emperor with the prescribed legend
- reverse two soldiers with standards, spears and shields with the prescribed legend
- mint mark to indicate which mint and officina made the coin
- specified purity and weight
I'd like to borrow two terms from metaphysics: “essential” and “accidental”. The essential properties are those which an object must necessarily have in order to belong to a particular category. Other properties, those which are not necessary to belong to that category, are termed “accidental” properties. These coins are all interchangeable because they all share the same essential properties, even though they differ in many accidental properties.
As another example, take corrective eye-glasses. A definition in terms of the essential properties might be, “Corrective eyeglasses consist of two transparent lenses, held in a frame, to be worn on the face to correct the wearer's vision.” Take away any essential property and they are no longer eyeglasses. Accidental properties might include the material used to make the lenses, the shape and color of the frame, whether single or bifocals, the exact curvature of the lens etc.
The distinction between the essential and accidental is common wherever precision in words is required, in legislation, in regulations, in contracts, in patent applications, in specifications and in standards. There are risks in not specifying an essential property, as well as in specifying an accidental property. Underspecification leads to lower interoperability, but over-specification leads to increased implementation cost, with no additional benefit.
Technical standards have dealt with this issue in several ways. One is through the use of tolerances. I talked about light bulbs in a previous post and the Medium Edison Screw. The one I showed, type A21/E26 has an allowed length range of 125.4-134.9 mm. An eccentricity of up to 3-degrees is allowed along the base axis. The actual wattage may be as much as 4% plus 0.5 watts greater than specified. Is this just an example of a sloppy standard? Why would anyone use it if it allows 4% errors? Why not have a standard that tells exactly how large the bulb should be?
The point is that bulb sockets are already designed to accept this level of tolerance. Making the standard more precise would do nothing but increase manufacturing costs, while providing zero increase in interoperability. The reason why we have cheap and abundant lamps and light bulbs is that their interconnections are standardized to the degree necessary, but no more so.
There is often a sweet spot in standardization that gives optimal interoperability at minimal cost. Specify less than this and interoperability suffers. Specify more than that and implementation costs increase, though with diminished interoperability returns.
An allowance for implementation-dependent behaviors is another technique a standard has available to find that sweet spot. A standard can define some constraints, but explicitly state that others are implementation-dependent, or even implementation-defined. (Implementation-defined goes beyond implementation-dependent in that not only can an implementation choose their own behavior in this area, but that they should also document what behavior they made implemented.) For example, in the C or C++ programming languages the size of an integer is not specified. It is declared to be implementation-defined. Because of this, C/C++ programs are not as interoperable as, say Java or Python programs, but they are better able to adapt to a particular machine architecture, and in that way achieve better performance. And even Java specifies that some threading behavior is implementation-dependent, knowing that runtime performance would be significantly enhanced if implementations could directly use native OS threads. Even with these implementation-dependent behaviors, C, C++ and Java have been extremely successful.
Let's apply this line of reasoning to document file formats. Whether you are talking about ODF or OOXML there are pieces left undefined, important things. For example, neither format specifies the exact pixel-perfect positioning of text. There is a common acceptance that issues of text kerning and font rasterization do not belong in the file format, but instead are decisions best deferred to the application and operating environment, so they can make a decision based on factors such as availability of fonts and the desired output device. Similarly a color can be specified, in RGB or another color space, but these are not exact spectral values. Red may appear one way on a CRT under florescent light, and another way on a LCD monitor in darkness and another way on color laser output read under tungsten lighting. An office document does not specify this level of detail.
In the end, a standard is defined as much by what it does not specify as by what it does specify. A standard that specifies everything can easily end up being merely the DNA sequence of a single application.
A standard provides interoperability within certain tolerances and with allowances for implementation-dependent behaviors. A standard can be evaluated based on how well it handles such concerns. Microsoft's Brian Jones has criticized ODF for having a flexible framework for storing application-specific settings. He lists a range of settings that the OpenOffice application stores in their documents, and compares that to OOXML, where such settings are part of the standardized schema. But this makes me wonder, where then does one store application-dependent settings in OOXML? For example, when Novell completes support of OOXML in OpenOffice, where would OpenOffice store its application-dependent settings? The Microsoft-sponsored ODF Add-in for Word project has made a nice list of ODF features that cannot be expressed in OOXML. These will all need to be stored someplace or else information will be lost when down-converting from ODF to OOXML. So how should OpenOffice store these when saving to OOXML format?
There are other places where OOXML seems to have regarded the needs of Microsoft Office, but not other implementors. For example, section 2.15.2.32 of the WordprocessingML Reference defines an “optimizeForBrowser” element which allows the notation of optimization for Internet Explorer, but no provision is made for Firefox, Opera or Safari.
Section 2.15.1.28 of the same reference specifies a “documentProtection” element:
This element specifies the set of document protection restrictions which have been applied to the contents of a WordprocessingML document. These restrictions shall be enforced by applications editing this document when the enforcement attribute is turned on, and should be ignored (but persisted) otherwise.
This “protection” relies on a storing a hashed password in the XML, and comparing that to the hash of the password the user enters, a familiar technique. But rather than using a secure hash algorithm, SHA256 for example, or any other FIPS compliant algorithm, OOXML specifies a legacy algorithm of unknown strength. Now, I appreciate the need for Microsoft to have legacy compatibility. They fully acknowledge that the protection scheme they provide here is not secure and is only there for compatibility purposes. But why isn't the standard flexible enough to allow an implementation to utilize a different algorithm, one that is secure? Where is the allowance for innovation and flexibility?
What makes this worse is that Microsoft's DRM-based approach to document protection, from Office 2003 and Office 2007, is entirely undocumented in the OOXML specification. So we are left with a standard with a broken protection feature that we cannot replace, while the protection that really works is in Microsoft's proprietary extensions to OOXML that we are not standardizing. How is this a good thing for anyone other than Microsoft?
Section 2.15.3.54 defines an element called “uiCompat97To2003” which is specified simply as, “Disable UI functionality that is not compatible with Word97-2003”. But what use is this if I am using OOXML in OpenOffice or WordPerfect Office? What if I want to disable UI functionality that is not compatible with OpenOffice 1.5? Or WordPerfect 8? Or any other application? Where is the ability for other implementations to specify their preferences?
It seems to me is that OOXML in fact does have application-dependent behaviors, but only for Microsoft Office, and that Microsoft has hard-coded these application-dependent behaviors into the XML schema, without tolerance or allowance for any other implementations settings.
Something does not cease to be application-dependent just because you write it down. It ceases to be application-dependent only when you generalize it and accommodate the needs of more than one application.
Certainly, any application that stores document content or styles or layout as application-dependent settings rather than in the defined XML standard should be faulted for doing so. But I don't think anyone has really demonstrated that OpenOffice does this. It would be easy enough to demonstrate if it were true. Delete the settings.xml from an ODF document (and the reference to it from the manifest) and show that the document renders differently without it. If it does, then submit a bug report against OpenOffice or (since this is open source) submit a patch to fix it. A misuse of application-settings is that easy to fix.
But a standard that confuses the accidental application-dependent properties of a single vendor's application for an essential property that everyone should implement, and to do this without tolerance or allowance for other implementations, this is certainly a step back for the choice and freedom of applications to innovate in the marketplace. Better to use ODF, the file format that has multiple implementations and acknowledges the propriety of sensible application-dependent behaviors, and provides a framework for them to record such settings.
3/16/2007 — added Cosmas Indicopleustes quote
Labels: Numismatics, OOXML, Standards
Saturday, February 24, 2007
Standards and Enablement
A Harvard Divinity School student reviews a proposed dissertation topic with his advisor. The professor looks over the abstract for a minute and gives his initial appraisal.
“You are proposing an interesting theory here, but it isn't new. It was first expressed by a 4th Century Syrian monk. But he made the argument better than you. And he was wrong.”
So it is with some trepidation that I make an observation which may not be novel, well-stated, or even correct, but here it goes:
There is (or should be) an important relationship between patents and standards, or more precisely, between patent quality and standards quality.
As we all know, a patent is an exclusive property right, granted by the state for a limited period of time to an inventor in return for publicly disclosing the workings of his invention. In fact the meaning of “to patent” was originally, “to make open”. We have a lingering sense of this in phrases like, “that is patently absurd”. So, some public good ensues for the patent disclosure, and the inventor gets a short-term monopoly in the use of that invention in return. It is a win-win situation.
To ensure that the public gets their half of the bargain, a patent may be held invalid if there is not sufficient disclosure, if a “person having ordinary skill in the art” cannot “make and use” the invention without “undue experimentation”. The legal term for this is “enablement". If a patent application has insufficient enablement then it can be rejected.
For example, take the patent application US 20060168937, “Magnetic Monopole Spacecraft” where it is claimed that a spacecraft of a specified shape can be powered by AC current and thereby induce a field of wormholes and magnetic monopoles. Once you've done that, the spacecraft practically flies itself.
The author describes that in one experiment he personally was teleported through hyperspace over 100 meters, and in another he blew smoke into a wormhole where it disappeared and came out another wormhole. However, although the inventor takes us carefully through the details of how the hull of his spacecraft was machined, the most critical aspect, the propulsion mechanism, is alluded to, but not really detailed.
(Granted, I may not be counted as a person skilled in this particular art. I studied astrophysics at Harvard, not M.I.T. Our program did not cover the practical applications of hyperspace wormhole travel.)
But one thing is certain — the existence of the magnetic monopole is still hypothetical. No one has shown conclusively that they exist. The first person who detects one will no doubt win the Nobel Prize in Physics. This is clearly a case of requiring “undue experimentation” to make and use this invention, and I would not be surprised if it is rejected for lack of enablement.
I'd suggest that a similar criterion be used for evaluating a standard. When a company proposes that one of its proprietary technologies be standardized, they are making a similar deal with the public. In return for specifying the details of their technology and enabling interoperability, they are getting a significant head start in implementing that standard, and will initially have the best and fullest implementation of that standard. The benefits to the company are clear. But to ensure that the public gets their half of the bargain, we should ask the question, is there sufficient disclosure to enable a “person having ordinary skill in the art” to “make and use” an interoperable implementation of the standard without “undue experimentation”. If a standard does not enable others to do this, then it should be rejected. The public and the standards organizations that represent them should demand this.
Simple enough? Let's look at the new Ecma Office Open XML (OOXML) standard from this perspective. Microsoft claims that this standard is 100% compatible with billions of legacy Office documents. But is anyone actually able to use this specification to achieve this claimed benefit without undue experimentation? I don't think so. For example, macros and scripts are not specified at all in OOXML. The standard is silent on these features. So how can anyone practice the claimed 100% backwards compatibility?
Similarly, there are a number of backwards-compatibility “features” which are specified in the following style:
2.15.3.26 footnoteLayoutLikeWW8 (Emulate Word 6.x/95/97 Footnote Placement)
This element specifies that applications shall emulate the behavior of a previously existing word processing application (Microsoft Word 6.x/95/97) when determining the placement of the contents of footnotes relative to the page on which the footnote reference occurs. This emulation typically involves some and/or all of the footnote being inappropriately placed on the page following the footnote reference.
[Guidance: To faithfully replicate this behavior, applications must imitate the behavior of that application, which involves many possible behaviors and cannot be faithfully placed into narrative for this Office Open XML Standard. If applications wish to match this behavior, they must utilize and duplicate the output of those applications. It is recommended that applications not intentionally replicate this behavior as it was deprecated due to issues with its output, and is maintained only for compatibility with existing documents from that application. end guidance]
This sounds oddly like Fermat's, “I have a truly marvelous proof of this proposition which this margin is too narrow to contain”, but we don't give Fermat credit for proving his Last Theorem and we shouldn't give Microsoft credit for enabling backwards compatibility. How is this description any different than the patent application claim magnetic monopoles to drive hyperspace travel? The OOXML standard simply does not enable the functionality that Microsoft claims it contains.
Similarly, Digital Rights Management (DRM) has been an increasingly prominent part of Microsoft's strategy since Office 2003. As one analyst put it:
The new rights management tools splinter to some extent the long-standing interoperability of Office formats. Until now, PC users have been able to count on opening and manipulating any document saved in Microsoft Word's “.doc” format or Excel's “.xls” in any compatible program, including older versions of Office and competing packages such as Sun Microsystems' StarOffice and the open-source OpenOffice. But rights-protected documents created in Office 2003 can be manipulated only in Office 2003.
This has the potential to make any other file format disclosure by Microsoft irrelevant. If they hold the keys to the DRM, then they own your data. The OOXML specification is silent on DRM. So how can Microsoft say that OOXML is 100% compatible with Office 2007, let alone legacy DRM'ed documents from Office 2003? The OOXML standard simply does not enable anyone else to practice interoperable DRM.
It should also be noted that the legacy Office binary formats are not publicly available. They have been licensed by Microsoft under various restrictive schemes over the years, for example, only for use on Windows, only for use if you are not competing against Office, etc., but they have never been simply made available for download. And they've certainly never been released under the Open Specification Promise. So lacking a non-discriminatory, royalty-free license for the binary file format specification, how can anyone actually practice the claimed 100% compatibility? Isn't it rather unorthodox to have a “standard” whose main benefit is claimed to be 100% compatibility with another specification that is treated as a trade secret? Doesn't compatibility require that you disclose both formats?
Now what is probably true is that Microsoft Office 2007, the application, is compatible with legacy documents. But that is something else entirely. That fact would be true even if OOXML is not approved by ISO standard, or even if it were not an Ecma standard. In fact, Microsoft could have stuck with proprietary binary formats in Office 2007 and this would still be true. But by the criterion of whether a person having ordinary skill in the art can practice the claimed compatibility with legacy documents, this claim falls flat on its face. By accepting this standard, without sufficient enablement in the specification, the public risks giving away its standards imprimatur to Microsoft without getting a fair disclosure or the expectation of interoperability in return.
Tuesday, February 20, 2007
The Anatomy of Interoperability
Rather than attempting the reckless frontal assault of defining interoperability, I'll take an oblique approach and discuss the forces that tend to increase or decrease interoperability. By understanding these forces, and how they interact, you can get a good sense of how interoperability works, and have a better idea how the decisions that are being made today will determine the interoperability consumers will experience in the years to come.
I'll start with the specification itself, the text of the standard. What characteristics might a specification have that hinder interoperability?
- Ambiguities — The specification may describe a feature in a way that is open to more than one interpretation. This may be caused by imprecise language, or by incomplete description of the feature. For example, if a specification defines a sine and cosine function, but fails to say whether their inputs are in degrees or radians, then this function is ambiguous.
- Out of scope features — The specification totally lacks description of a feature, making it out of scope for the standard. For example, neither ODF nor OOXML specifes the storage model, the syntax or the semantics of embedded scripts. If a feature is out of scope, then there is no expectation of interoperability with that feature.
- Undefined behaviors — These may be intentional or accidental. A specification may explicitly call out some behaviors as "undefined", "implementation-dependent" or "implementation-defined". This is often done in order to allow an implementation to implement the feature in the best performing way. For example, the size of integers are implementation-defined in the C/C++ programming languages, so they are free to take advantage of the capabilities of different machine architectures. Even a language like Java, which goes much further than many to ensure interoperability, has undefined behaviors in the area of multi-threading, for performance reasons. There is a trade-off here. A specification that specifies everything and leaves nothing to the discretion of the implementation will be unable to take advantage of the features of a particular platform. But a specification that leaves too much to the whim of the implementation will hinder interoperability.
- Errors — These may range from typographical errors, to incorrect use of control language like "shall" or "shall not", to missing pages or sections in the specification, to inconsistency in provisions. If one part of the specification says X is required, and another says it is not, then implementations may vary in how feature X is treated.
- Non-thematic provisions — If a standard is claimed to be platform-neutral or application-neutral, then anything that is tied to a particular platform or application will hinder that type of interoperability.
- Feature Creep — A standard can collapse under its own weight. There is often a trade-off between expressiveness of a standard (what features it can describe) and the ease of implementation. The ideal is to be very expressive as well as easy to implement. If a standard attempts to do everything that everyone could possibly want, and does so indiscriminately, then the unwieldy complexity of the standard will make it more difficult for implementations to implement, and this will hinder interoperability.
- Public participation — The public gets involved early, reading drafts of the standard before it is approved, and also meeting minutes and discussion list traffic. The public provides comments to the technical committee and reads the comments submitted by others. Note: the mere theoretical ability of the public to participate results in absolutely no benefit to the standard. This is like a beta program that exists in theory only will uncover zero bugs. What counts is the actual participation. Openness is a means to participation, but is insufficient on its own.
- Expert review — The feature set for a modern word processor or spreadsheet is large and requires interdisciplinary expertise in such diverse areas as vector graphics, text layout, mathematical layout, accessibility, internationalization, schema design, statistical and financial calculations, etc. So the design of a file format will similarly require a broad range of expertise. Tapping into outside experts (for example, from member companies or other technical committees) when designing the standard, or as reviewers, is a way to reduce the types of problems that hinder interoperability.
- Multi-vendor participation — It is good to have multiple active participants in the standards development process, representing implementors both proprietary and open source, both traditional implementations as well as web-based. You might consider other constituencies as well, such as end-users, accessibility advocates, academic and government representatives, consumer advocates, etc. There is a saying familiar to programmers, "Every class of users finds a new class of bugs". This is as true with standards as it is will code. More eyeballs, especially with different perspectives, is key. It is better to have 5 active participants with different perspectives than to have 100 participants from a single vendor. Again, I'm talking reality here, not just perception. It is easy to have a technical committee that is large on paper, but still have all of the ideas come from a single company. This does nothing for interoperability. You want to have really multi-vendor participation. This is why having public read-only access to the mailing list of the technical committee is so valuable. That is great way to see what the real participation level is.
- Multiple implementations — If a standard is intended to be platform-neutral and vendor-neutral, then the participation of multiple implementors working on multiple platforms is essential. They are the ones who truly drive interoperability. Especially if they are implementing while the specification is under development or review, they will find, report and resolve many interoperability issues.
- Reuse of existing standards — When writing a program, the availability of reusable code or design patterns can improve quality and shorten schedules. In manufacturing, the use of off-the-shelf components can be faster to tool and be less expensive than custom components. Similarly, in standards development, when you reuse another standard you reuse the domain expertise, the standards development effort and the review effort that went into the development of that standard. If you are lucky and you reuse a widely-implemented standard, implementors will likely also be able to reuse knowledge and even code when implementing it.
- Existence of a reference implementation and test suite — These work, together with the standard itself, as a triad of verifications that reinforce each other. The test suite is written to test the provisions of the specification. The reference implementation is written to implement the provisions of the specification. The test suite is executed against the reference implementation. Any errors indicated are thus either errors in the reference implementation, the specification, or the test suite. Fix the bug and repeat until the reference implementation passes the test suite. Having a reference implementation that passes a test suite that has high or complete coverage of the provisions of the specification is ideal for interoperability.
- Good engineering — In the end, a technical standard is technical, and it will be influenced by the same kinds of engineering criteria that influence how we architect and design software applications. If you can make a standard that is modular, exhibits internal consistency in its parts, is cohesive, has reasonable dependency chains among its parts, does what it set out to do with minimal complexity, but allows of uses beyond that, then you've made a good start. But if you find yourself shaking your head, slapping your forehead or even holding your nose when reading a standard, then you are heading for interoperability hell. The standards that have worked well were as simple as possible, but no simpler. Think XML 1.0 or XSLT 1.0. Engineers know how to build complexity on top of simplicity. But you don't build anything on top of crap.
The factors that hinder interoperability include:
- Implementation bugs — Conformance to a standard, like any other product feature, gets weighed against a long list of priorities for any given product release. There is always more work to do than time to do it. Whether a high-quality implementation of a standard becomes a priority will depend on factors such as user-demand, competition, and for open source projects, the level of interest of developers contributing to the community.
- Functional subsets — Even in heavily funded commercial ventures standards support can be partial. Look at Microsoft's Internet Explorer, for example. How many years did it take to get reasonable CSS2 support? When an application supports only a subset of a standard, interoperability with applications that allow the full feature set of the standard, or a different subset of the standard, will suffer.
- Functional supersets — Similarly, an application can extend the standard, often using mechanisms allowed and defined by the standard, to create functional supersets that, if poorly designed, can cause interoperability issues.
- Varying conceptual models — For example, a traditional WYSIWYG word processor has a page layout that is determined by the metrics of the printer the document will eventually print to. But a web-based editor is free from those constraints. In fact, if the eventual target of the document is a web page, these constraints are irrelevant. So we have here a conceptual difference, where one implementation sees the printed page as a constraint on layout, and another application is in an environment where page width is more flexible. Document exchange between two editors with different conceptual models of page size will require extra effort to ensure interoperability.
- Different code bases — The more distinct implementations, the more paths of exchange. If you have only a single implementation of your standard, then there will be no interoperability issues, at least not with others using the same version of your application on the same platform. But if there are two implementations, then you have 2 paths of exchange to worry about. If there are 3 implementations then you have 6 paths, and so on with N*(N-1) possible paths for exchanging files with N applications. Consider the ODF format for word processing documents, where we will soon have implementations in Office (at least two different Plugins), OpenOffice, Lotus Notes, KOffice, Google Docs and Abi Word. This is 42 exchange paths (or more if you consider platform variations) though obviously not all of these paths will be common.
And the forces that counteract these and lead to improved interoperability are:
- Test suites and reference implementations — As described earlier, these are key. During the development of the standard they improve the standard, and after the standard is approved they continue paying dividends by helping improve the quality additional implementations.
- Conformance certification — This can take several forms, from self-testing and self-certifying, to more formal tests by 3rd party labs. The result is a compliance report, listing which features of the standard were correctly implemented, and which were not. Having this information available to consumers can help interoperability to the extent it allows them to narrow their consideration to the applications that most fully and correctly implement the standard.
- Profiles — This is a formal mechanism for defining an interoperable subset of a standard. For example, XHTML Basic is a W3C-defined subset of XHTML for use on mobile devices. By defining such a subset, interoperability is enhanced in several ways. First, a single web service can produce XHTML Basic that is understood by mobile devices from several vendors. Second, XHTML Basic is a proper subset of XHTML, so it is also interoperable with all the tools that accept XHTML.
- Adaptive extension mechanisms — It is good to have a formal mechanism for extending the specification while at the same time allowing a graceful recovery in the case where the document is loaded in an application that doesn't understand the extension.
- Inter-vendor cooperation — Make no mistake. Once an interoperability problem is found, where application A and application B are not rendering the same document in an acceptably similar fashion, then we have a bug. Such a bug may be reported by the customers of application A or the customers of application B. Ideally, A and B will cooperate and share information on interoperability bugs found. The customer that reports A's bug to company B, may tomorrow be matched by another that reports B's bug to customer A. So some sort of interoperability bug clearinghouse or other protocols can help here.
3/29/07 — Corrected error sent in by an anonymous reader. The number of interoperability concerns scale as N(N-1), not as N!.
Labels: Standards
Wednesday, February 14, 2007
Washing Machines are not Lamps
David correctly points out that some appliances, like washing machines or electric dryers, with higher power requirements, have a different plug design. "Clearly, a one size standard does not fit all", as he says. However, this is an intentional design decision made for safety reasons. If things cannot safely be interchanged, then good industrial design is to make them impossible to be interchanged. These plugs are incompatible and non-interoperable on purpose.
No one would intentionally do that with a file format, would they?
David then suggests that a single standard is insufficient because it would stifle competition and innovation:
Electricity is a largely mature and stable technology and there is not much room for innovation in the socket and receptacle space. Document formats, on the other hand, are constantly evolving to keep pace with changing technology. Competition is vital to ensure that those formats continue to meet those ever changing needs. Imagine if a single document format was adopted 15 years ago. How would that format deal with things that we take for granted today like including links to web pages, adding digital photos, or even embedding video in our documents? Unlike electricity, document formats are evolving at a rapid pace and competition will help drive that innovation.
I see it differently. Has a single HTML standard held back competition and innovation on the web? Has the move from vendor-specific network protocols to TCP/IP deprived consumers of innovation? Has the standardization of SQL held back the database industry? Have standardized programming languages like C and C++ prevented others from innovating? I see no evidence of this. On the contrary, standardized HTML, TCP/IP, SQL and C/C++ have been fundamental to the modern IT economy and have been responsible for many billions of dollars of value.
I'd also challenge the assertion that standardization equates with lack of innovation. If this were true, how does Microsoft reconcile their work standardizing OOXML, .NET, C++/CLI, C#, etc., with their needs for continuing innovation? Are these areas, "largely mature and stable"?
Or is this really just a belief that standardization is good when Microsoft originates and controls the standard, but it is bad otherwise?
Back to the examples of HTML, TCP/IP, SQL, C/C++. These standards continued to evolve, and innovations were brought to consumers, but they were done in a multi-vendor standards process where they reconciled their multiple perspectives and needs. Is that such a bad model to follow?
In the end, where does innovation come from? Does it require absolute control? Or does it come from having bright people? I'd suggest the latter, and point out that Microsoft employs several, but not all of the bright people in the area of file formats. Microsoft and Microsoft's customers would benefit greatly if Microsoft would join with their competitors who are already innovating and competing in the evolution of the ISO ODF standard.
Remember the "X" in XML stands for Extensible. Making a single file format that meets Microsoft's needs, as well as IBM's, Sun's, Corel's, Novell's, Google's, etc., is not only technically possible, it is the best approach for the consumer. This does not mean that competition ends, or that all office applications will have identical features, or that we can only have lowest-common-denominator functionality. It just means that we should agree on a common representation for the features that we already have in common, and then have a framework for layering on vendor-specific innovations to express the areas where we differ.
Tuesday, February 13, 2007
The Word Ends on May 1st, 2010
According to ZDNet's Dan Farber, quoting an IBM whitepaper, by 2010, "the world’s information base will be doubling in size every 11 hours."
Every 11 hours? That's quite a statement. Let's see what this means. The largest storage system in the universe is the universe. (Let that sink in for a moment). When I grew up, I was taught that there were approximately 10^79 electrons in the universe. Let's use them all! 10^79 bits of storage, stored using the spin state of the electrons, in a giant quantum computer.
I have no idea how much data we will have on January 1st, 2010, so let's assume, for sake of argument, that a virus wipes out all the data in the world on New Year's Eve, and we start the year with only 1 bit of data, and it doubles every 11 hours. So after 22 hours, we have 2 bits of data, after 33 hours 4 bits, and then after almost two days we get our first byte (8 bits). This isn't too bad, is it?
The equation is: 2^x=10^79. Solve for x, a simple exercise in logarithms, giving the answer 262.43. We can only double that many times before hitting the universal limit and we exhaust all of the storage in the entire universe on May 1st at 6:45AM. Of course, maybe we'll just Zip it all up and last until dinner time?
I think I'll call in sick that day.
But seriously, I wonder if this "every 11 days" figure is a typo? Doubling "every 11 months" would be easier to imagine and would give us to 2250.
Labels: Math
Saturday, February 10, 2007
How Standards Bring Consumers Choice
Let's take the humble floor lamp as an example, and look at some of the standards that govern its design, and the choices this enables for the consumer. In the United States, many of the parameters for electrical fixtures are governed by standards promulgated by the National Electrical Manufacturers Association (NEMA). Some of these standards are approved by ANSI or the IEC as well.

Let's start at the wall. We see here a NEMA 5-15 Type B 3-Pin duplex socket. As you may notice, the left slot is slightly taller than the right one. This is the neutral. The live line is on the right, and the ground is on the bottom.
This socket is rated for 5 Amps at 125 Volts. The definition and calibration of the Amp and Volt and other key values in the Metric or SI system of measurement are standardized by the General Conference on Weights and Measures (CGPM).

From the lamp we have the male end, an AC power plug. On my lamp the connector is a polarized, ungrounding NEMA 1-15P plug.

This is an electrical cord for the lamp. The codes indicate that this is a SPT-2 , #18 gauge flexible cord. SPT (Service Parallel Thermoplastic) is a standard defined by the American Society for Testing and Materials (ASTM) and approved by Underwriters Laboratories (UL). The gauge is measured in American Wire Gauge (AWG) units, also known as the Brown & Sharpe gauge, a standard dating back to 1855.

A critical interface is the one between the bulb and the lamp. The most-common connection in the United States is called the MES or Single Contact Medium Edison Screw (E26). This is an ANSI Standard, C78.20-2003.

The bulb itself is an NEMA A21-style bulb, with an E26d style base. It will be 134.9 mm long, 28.2mm wide at the base. The height of the conductor screw will be 24.4mm. As indicated, this is a three-way bulb, rated at 50W, 100W and 150W.
As you can see, the inputs and outputs of the lamp are heavily-constrained by a number of standards. Is this a bad thing? Is the consumer deprived for not having to worry about different plug and outlet types, or what gauge cord to use for their lamp, or what thread connection to use? On the contrary, it is a blessing for the consumer that such pieces are interchangeable commodities.
If one bulb burns out, you can replace it with whatever brand is cheapest. Or you can get a lower wattage one to save electricity. Or even get a florescent one that fits in the same Medium Edison Screw socket. You can get clear glass, soft white, red glass or black light. You have these choices because bulb standards make bulbs interchangeable.
If the dog chews on the cord and you need to replace it, there is not need to return the lamp to the manufacturer at great expensive. SPT-2 cords are standard and available at any hardware store. You can simply replace it yourself.
If you move to another house, will your lamp stop working and need to be thrown out? No, of course not. The NEMA 15-5 power outlets are in every home in North America. You can use your lamp wherever you go.
What if you want to buy a new lamp next year, will you need to change the wiring or the outlets in your house to work with it? Certainly not. The power outlets are a standard will work with any lamp, the ones you have today, the old ones you buy at an flea market, or the ones you buy 10 years from now. The standards ensure interoperability, interchangeability and substitutability.
In fact, far from constraining choice, standards enable greater choice. Because the basic plugs, receptors and connectors are governed by standards, these core components have become commodities and are produced off-shore at low cost to you, the consumer. This causes lighting designers and manufacturers to compete on the basis of style, elegance, utility and features. So standards result in lower cost, greater competition and greater choice for the consumer.
These are the choices enabled by standards, and the choices that consumers want:

It is the same thing with document formats. Consumers don't want to worry about document formats. They don't want to even think about document formats. They just want them to work, invisibly, without problems. Of course consumers want choice, but it is the choice of applications, choice of features, choice of vendors, choice of support options, choice of open source versus proprietary source, choice of heavy weight versus web-based, a choice of buying a single application versus buying a suite, etc. A single universal file format is what makes these other choices possible, just like a choice of the Medium Edison Screw bulb leads to an affordable choice in lamp designs.
To perpetuate vendor-specific file formats is like having a lamp that requires a special plug and a special light bulb and even special electricity. It is to go through life with a bag full of adapters and transformers that you will need to apply whenever you, or someone else, needs to use your document in another application. The cost of multiple document standards is that the industry will invest in a wide variety of convertors between the formats, and this cost will be passed on to the consumer. These convertors will work sometimes, but will inevitably be slower, buggier, have less fidelity, and their availability will lag the office applications by months or years.
The choice is yours.
Updates:
2/14/2007: A discussion of one critic's response to this essay can be read in this follow-up.
Friday, February 09, 2007
Once More unto the Breach
Since his blog's title, like mine, is from Shakespeare, and he occasionally writes about ODF, I am doubly obliged to give a mention.
Looking at the posts tagged ODF, there is some good stuff. In Vendor-Speak: Microsoft and OOXML, he takes a close reading of some of the recent statements from Microsoft on standards and choice, a long-time confusion of terms that I had previously called a language game. Walli points out:
...Standards happen when a technology space matures to the point that customers are over-served and want choice to encourage competition. Customers complaining about price is the market signal. Competitors know they can collectively chase the incumbent vendor with a standard at this point, if they pick the right level of collective abstraction to standardize. This is how standards work in the marketplace. (I would hope the GM for standards at Microsoft knew this.)
A sympathy play isn't going to work here. Customers WANT the standard that encourages multiple implementations. True Microsoft support for ODF in their Office product suite would have been listening to customers. Complaining that the marketplace is competitive while shoving your own product specification through a standards forum is naive at best and arrogant in the extreme at worst.
Another good post, is How Microsoft Should Have Played the ODF Standards Game:
The interesting thing is to look back on the number of times a vendor with a single implementation tried to win playing an overlapping standards game. Looking at the UNIX wars I remember three occasions off the top of my head where this was tried over a long period.
- "tar wars" over archiving formats.
- The GUI Wars where OSF/Motif and Sun's GUI toolkit battled it out.
- The sockets versus streams debate.
In each case, we ended up with two standards being forced upon us. In each case, the dominant technology that won in the marketplace was the one with the most implementations, with the other withering on the vine. Even when both specifications became required for an implementation to claim conformance to the single standard that included them, customers in the market used the specification which was most widely implemented every time.
Microsoft chose the wrong strategy here on multiple levels, betting against customers and the market in general. It may buy a bit of time, but will ultimately cost them more in the long run.
This is a theme that Walli repeats in several of his blog posts — the standard with the most implementations wins.
So a hearty recommendation for Once More unto the Breach, a blog that deserves a slot in your feed aggregator.
Thursday, February 08, 2007
Here today, gone tomorrow
(Why don't I cite using section numbers? Good question. This is because the version of OOXML submitted to JTC1 reused section numbers, so a reference to "section 3.4.2" could be ambiguous.)
A more significant change is that the annexes are now zipped up with the PDF file to which it pertains. So Part 4 is now a zip file with 4 electronic annexes enclosed. This is different than what JTC1 received. I don't believe that JTC1 NB's received the electronic annexes at all.
Maybe when Ecma finishes deciding how they want to paginate the thing I'll go back and update page references in previous posts. But for now, I'll leave them as-is, which will match the version that JTC1 NB's have received, but not the version that the public has.
I'll close by saying that this is a bit odd for open standard, that the version that was submitted to JTC1 is not the same as what is available to the public. You would almost think that someone out there did not want public input on OOXML to be easily consumable by JTC1 NB's.
Labels: OOXML
Merely a flesh wound?
Since that post came out there has been some interesting spin placed on these results, spin that I'm seeing popping up in several places. I'll give you a few examples, and than explain why these ballot results are more significant than some might have you believe.
An anonymous comment on Groklaw:
This is a normal step in the fasttrack proces. (bypassed by ODF however) This might add one or two months to the proces [sic]. The ballot in ISO requires a 2/3 majority. Only 20% have reacted with contradictions or a question for information and Ecma can probalby [sic] satisfy a lot of those 19 by answering questions, expanding on their information and mayby [sic] amending the proposal.
A blog post from Jerry Fishenden, Microsoft's National Technology Officer for the UK:
I'm not sure where the media are getting their information, but apparently out of the hundred or more ISO members only nineteen of them filed a response to Ecma 376 (Open XML) by the close of the initial 30 day consultation period.
From Brian Jones, whose excellent blog does worthy service to Microsoft's perspective on OOXML:
The 1 month contradiction phase of the 6 month fast track process is now complete. It sounds like about 18 of the 100+ countries reviewing the standard came back with comments.
And finally, an eWeek article, quoting Tom Robertson, Microsoft GM of interoperability and Standards:
There are 103 countries that participated in the ISO process, and each country has a national standards body with the authority to act at the ISO on behalf of that country.
Obviously, these are their talking points. To see why this misleading, we first need to have a quick refresher on how votes are counted, according to the JTC1 Directives.
First, remember that this is not ISO per-se. It is JTC1, a Joint Committee between ISO and the IEC. Not all 100+ ISO members are members of JTC1. So it is not relevant to talk about how many ISO members there are in total. Only the JTC1 members were eligible to raise contradictions.
Second, the critical number to look at is the count of Primary Countries of JTC1, the so-called P-Countries. There are only 30 P-Countries in JTC1. You can see them listed here. If you compare the list that Andy Updegrove posted you will see that 16 of the 19 countries on his list are also P-Countries of JTC1. So 16 of the 30 P-Countries raised contradictions. The other three are Observers, or O-Countries.
Why is the P-Countries designation important? During the 5-month ballot, approval of OOXML will require that two-thirds of the voting P-Countries approve, as well as that no more than one-quarter of all votes cast are negative. This requirement for two-thirds approval from P-Countries is what makes them so critical.
Do the math. One-third of 30 P-Countries is 10. Add 1 to get 11, the magic number. If 11 P-Countries vote against OOXML during the 5-month ballot, then OOXML will fail. If some countries abstain, then this magic number goes proportionately down. Since some NB's have a consensus voting procedure for determining their vote in JTC1, the lack of consensus could lead them to abstain from the 5-month ballot, just as it may have lead some NB's to abstain from the contradiction ballot. So this magic number will likely be less than 11 because of these abstentions.
So, with 16 P-Countries already expressing concerns about OOXML, Microsoft clearly has an uphill battle. If the vote were held today, OOXML would fail in JTC1. To portray the reception of 19 contradictions, 16 of them from P-Countries, as being an average occurrence, or par for the course, or insignificant, is pure spin and and denies the magnitude of the rebuke OOXML has received.
Labels: OOXML
Sunday, February 04, 2007
A Barleywine
The richest and strongest of the English Ales. A showcase of malty richness and complex, intense flavors. The character of these ales can change significantly over time; both young and old versions should be appreciated for what they are. The malt profile can vary widely; not all examples will have all possible flavors or aromas.
Usually the strongest ale offered by a brewery, and in recent years many commercial examples are now vintage-dated. Normally aged significantly prior to release. Often associated with the winter or holiday season.
I started this batch back in November, with Belgian toasted malts (Dingemans Special B and Biscuit) and Target, Cascade and Fuffgle hops. The starting specific gravity (O.G.) was 1.112, which is one seriously heavy wort.
The previous day I had made a yeast starter, building a Wyeast #1056 American Ale 125ml "smack pack" into a 600ml starter (650 ml water 3/4 cup DME boiled for 15 minutes). For high gravity beers this is essential in order to get the fermentation off to a fast start.
After 2 1/2 weeks, the fermentation slowed enough to rack into a carboy where it sat for another month. Today I finally had a chance to bottle this, yielding 11 liters of barleywine. Final gravity was 1.034 giving an estimated ABV of 10.3%, a potent brew indeed. By way of reference, Budweiser is 5%.
An initial taste indicated that it was nicely balanced and hid the high alcohol levels behind the maltiness with forward hints of licorice, vanilla and plum. I will let it bottle condition for another 6-months or so before trying again. This will be a beer to sip and enjoy for several years.
Note that no licorice, vanilla, or plum was ever added to this beer. It is pure beer, according to the German Reinheitsgebot — nothing but water, malted barley, hops and yeast. The rest is the magic of biochemistry, the enzymes released during the malting of the barley that convert the starches into sugar, the carmelization of these sugars during the roasting of the barley, the alcohols and esters produced by the fermenting of the yeast. Even after the yeast has done its work and settled out, the beer will continue to evolve and change over time. Compare the complexity of a serious, living beer like this to the mass-produced, always-the-same pale lagers that fill the store shelves, and you will never go back.
Declaring Bankruptcy
I have a related problem, interesting links I've collected and have meaning to blog about. But my links have accumulated far faster than I have been able to write about them. So I am declaring "link bankruptcy". Here is my fire sale, a set of interesting topics for only pennies on the dollar:
- Glyn Moody has the story about how platform dependencies has impacted one notable British institution.
- Even more startling results in Korea, as reported in The Cost of Monoculture and the Korean Saga.
- It is mainly in Polish, but some in English. More coverage of Open Standards in a new blog from Jacek Łęgiewicz.
- In case you missed it the first time around, here is a wonderful essay by Dan Bricklin on "Software that Lasts 200 Years". It made me think of what ramifications this has for file formats that aspire to longevity as well.
- This looks interesting. A free OpenOffice Calc add-in for doing "fuzzy math" in OpenOffice.
- Sweave adds ODF support to the open source R statistical analysis and graphing platform.
- Docvert, an online REST service for converting Microsoft Word documents into ODF format.
- I know someone was asking for this a few months ago — A Microsoft Works import filter for OpenOffice.
- Office Migration Planning Manager (OMPM) allows bulk conversions of legacy Office binary documents to OOXML. Does anyone have something similar for ODF? Not just bulk conversion, but detection and reporting of possible conversion problems as well.
- The eXtensibility Manifesto has some good schema design advice, including: #3 "Design of a data model focuses on all stakeholders' requirements for the data." #6 "Designs or components are not reinvented, but rather are leveraged where possible."
- "[Expert Witness] Alepin...alleged that the company [Microsoft]had subverted developers who used Microsoft's version of Java 'thinking they were developing multi-platform applications, but were actually developing Windows-specific applications' ". From PC Pro News.
- The Case For ODF -- a recent presentation from OpenOffice Community Manager Louis Suarez-Potts.
- "Office 2007 lacks some features of earlier versions of Office, and so it can't fully support some Office files created in earlier versions. For example, Word 2007 cannot open Word files that contain multiple document versions, a feature supported by Word prior to Word 2007". Anyone know what else is missing? From Directions on Microsoft.
- A few months old — European Cities Do Away with Traffic Signs. Does anyone know how this has turned out?
- Dashed Lines and their uses.
- David Berlind over at ZDNet: "To me, Ecma is not a standards body. As evidenced by the DVD situation (which is ridiculous if you ask me), it's little more than a puppet with a pipeline through which vendors can pump their proprietary technologies into the ISO standardization process (avoiding the rigor that should normally be applied to anything up for consideratoin as an ISO standard). As such, the ISO is sort of a joke too."
- "One trouble spot we encountered using Vista's Explorer metadata organization tools was the lack of support for some of the file types we commonly use. For instance, JPEG files happily take attributes under Vista, but PNG files do not. Along similar lines, Vista would not apply metadata to files we had created in the OpenOffice.org format. And, strangely, our attempts to apply metadata to documents created in OpenOffice.org—in Microsoft Office format—were greeted with an error message." From eWeek.
- What is a standard, according to David Rudin, Microsoft's official Standards Attorney? "A technical specification that enables interoperability between different products and services and is either 1) intended for widespread industry adoption or 2) has achieved wide spread industry adoption." This is a nice write-up.
Friday, February 02, 2007
Introducing ODF 1.1
I played but a bit role in this story, though I watched it unfold with amazement. It was late 2005. A colleague mentioned that there were rumblings of concern in Massachusetts about their recent decision to move to ODF and what impact that would have on persons with disabilities. Although I am not an accessibility expert, I know the basics. (Every programmer should know the basics of accessibility, as well as the basics of internationalization, typography, human factors, performance, security, law, technical writing, project management and how to present and receive business cards in Asia).
Initially, I suggested that a file format has no relevance at all for accessibility. After all, the hard part of accessibility, the integration with screen readers was all at the application and operating system level. What difference could a file format make? The file format is not even involved except when loading or saving the document, right? But since knew ODF, I offered to do a quick spot check and report back. It wasn't long before I was able to demonstrate a handful of places where data necessary to enable accessibility was not described in the existing specification. For example, although an imported image allowed an annotation of alternate text for use by screen readers, an OLE embedding did not.
To err is human, but what happened next was extraordinary. There is a natural tendency to shrink away from criticism, to retreat inward and retrench, and at all costs avoid admitting errors. But I personally believe that every time we are corrected or criticized, it gives us another opportunity to show our character by how we handle it. The unchallenged person may be a gentleman or a scoundrel. You do not know until he is under pressure. So it is notable that the OASIS ODF TC overcame its accessibility problems not with defiance and not with acquiescence, but by enthusiastically embracing the challenge, engaging the critics, including the aggrieved community, bringing in the experts, both from OASIS member companies as well as outside invited experts, and working within an open and transparent standards development process, rolled up its sleeves and got to work.
The OASIS ODF Accessibility Subcommittee first met on January 27th, 2006. They delivered their evaluation report on ODF accessibility in June of 2006, followed by contributions to the ODF 1.1 specification which was approved as an OASIS Committee Specification in October, 2006, and just this week was approved by the OASIS membership as an OASIS Standard. This took a few days over a year, start to finish.
This is what open standards are all about and why they are so damn important. It isn't just about patents and lawyers, though that is certainly part of it. It isn't just about getting your specification approved by ISO, though that is certainly part of it. It isn't just about how little you can do to earn the label of "open standard". It is about how much we can do together to improve some parts of the technological landscape that are broken today for some users, and have been for some time.
Congratulations are due the members of the Accessibility Subcommittee for their diligent efforts. But we haven't heard the last from them. Their charter calls for them to continue their good work, to make additional accessibility improvements to ODF, look at new dimensions of accessibility, consider a wider range of disabilities and create a guide for ODF implementors on the best practices for implementing the accessibility feature of the ODF standard.
I look forward to their contributions to ODF 1.2 and beyond!
Labels: Accessibility, ODF