Standards

ODF at 5 Years

2010/05/01 By Rob

Five years ago today, on May 1st, 2005 OASIS approved Open Document Format 1.0 as an OASIS Standard. I’d like to take a few brief minutes to reflect on this milestone, but only a few. We’re busy at work in OASIS making final edits to ODF 1.2. We’re in our final weeks of that revision and it is “all hands on deck” to help address the remaining issues so we can send it out for final public review. But I hope I can be excused for a short diversion to mark this anniversary.

I won’t talk much about the 5 years since ODF 1.0 was approved. The ODF Alliance and their “ODF Turns Five” [pdf] does a good job there. But I would like to talk a little about ODF and why it is so important that it came about when it did, why it was so timely.

To fully appreciate the significance of ODF you need to understand the market climate in which it was created, and to understand that you need to understand a little of the history of word processors. The following time line illustrates the introduction dates of word processor applications over the past 30 years or so. You will notice some familiar and not-so-familiar names:

We can divide this time line into four time periods, each one driven by a pivotal development.

The first period was the “Pioneering Age”, when the first steps toward the modern word processor were taken. This was research-driven, primarily by Xerox PARC, who developed the first WYSIWYG word processor, Bravo as well as the first GUI word processor, Gypsy. Except for the line editor vi, which still has some adherents among the troglodyte cave dwellers, none of these first-generation applications survived, though their influence did. For example, Charles Simonyi, after working on Bravo at Xerox, went to Microsoft to develop Word. (Ah, the days before software patents…)

The next wave of word processor applications, the “Personal Computer Age” came in the 1980s with the new platforms of the IBM PC (1981) and the Apple Macintosh (1984). New platforms require apps, either new or ported, and you will see several familiar names introduced in that fruitful period.

Then we have a gap. From around 1990 to 1999 we do not see many new word processor introductions. This was the “Lost Decade”. New word processor introductions died off. Unchallenged by competition, even Microsoft Word advanced relatively little in this decade, compared to innovations before or since.

A few forces were at play here. First, there was a platform shift, from MS-DOS to MS-Windows 3.1 (1991) and Windows 95 (1995). Few companies were able to successfully port their applications to Windows. Also, the market changed significantly with the introduction of Microsoft Office as a suite of applications. Suddenly it was not enough to have a good word processor, say WordPerfect, or a good spreadsheet, say 1-2-3, or a good presentation package, say Harvard Graphics. To be competitive you needed to have all three suite components. And few companies did. Finally, there was the preferential access to operating system technical information Microsoft gave to their own applications teams, allowing Microsoft apps to run better on Microsoft operating systems than their competitors could. The decade closed with word processor competition wiped out. Analysts stopped tracking and reporting market share data when Office’s share exceeded 95%. And file formats? There were the binary DOC, XLS and PPT. And the file format documentation was only available under license from Microsoft, and only if you agreed not to make a competing word processor.

That was the shape of the market around 2000. Or more properly the state of the Microsoft monopoly.

So what happened that made ODF possible? In one word, the Internet. Well, not so much the technology of the internet itself, but widespread access to the internet via the web. This enabled the open source movement as we know it today to scale. Although open source existed before the web, unless you were at a major university or research centers, sharing source code and working collaboratively on software was very difficult. But with widespread access to email, ftp, web, eventually version control, we had the tools needed to scale open source from small teams to large teams. And to write a competitor to Microsoft Word you need a substantial team.

Why was open source so important? Because no rational profit-seeking entity would compete against a monopoly, especially one maintained by restricting access to technical information needed to interoperate. Lacking effective government regulation, the market was revived by open source. You see the same thing happen with Linux and with web browsers.

The other thing the internet and the web brought was a new platform based on open standards, HTML, CSS, XML, Javascript, allowing an interactive style of web application called “AJAX”. And since this new platform was based on open standards, Microsoft was less effective in preventing competition in this area. Certainly they tried. From ActiveX to Silverlight, from poor standards support in Internet Explorer, to the infamous memo by Bill Gates in 1998: “One thing we have got to change in our strategy – allowing Office documents to be rendered very well by other peoples browsers is one of the most destructive things we could do to the company. We have to stop putting any effort into this and make sure that Office documents very well depends on PROPRIETARY IE capabilities”, they tried, but ultimately failed to “take back the web” and turn it into a proprietary Microsoft platform.

With the new web application platform came new web-based word processors, some of which are charted above.

The net effect is that since 2000 or so we have a new diversity of word processors, open source, web-based, even the revival of commercial competition. It was against this backdrop, the history of competition and diversity all but wiped out but then restored in the new millennium, that ODF was born. Today every word processor of note supports ODF, including Microsoft Word. As Microsoft’s National Technology Director, and former CIO of Washington State, Stuart McKee said, “ODF has clearly won“. We’ve scaled the steep walls of monopoly and planted a new flag. Our former opponents are now our colleagues, working with us on ODF 1.2. We’ve shown we can win. But now we need to show that we can rule. This is the challenge. We need to continue to evolve ODF to meet user needs — and these are diverse needs — as well as accommodate a wide range of application models, from traditional heavy-weight desktop applications, to mobile apps, to web based apps, while realizing that these platforms themselves are shifting and possibly converging. Standards advance at glacial speed, while technological and competitive forces move at faster speeds. Allowing flexibility and extensibility while at the same time preserving interoperability among ODF implementations — this is a hard task, and one that is not entirely technological. The key value of ODF is to support interoperability in a market of diverse applications. This is the choice that users want.

But enough of the reflection. Time to get back to my work on ODF 1.2. I need to figure out linear depreciation according to the French accounting system so we can specify the AMORDEGRC spreadsheet function properly.

A Standard I Would Use: Auto Unsubscribe

2009/08/19 By Rob 8 Comments

I don’t get a lot of spam, at least not in the traditional sense of “unsolicited commercial email”. But I do get a lot of solicitations from online retailers with whom I have done business. As we all know, even a single order can trigger weekly emails. Multiply that by all places I do business with, and I end up with a lot of unwanted emails.

Certainly, the vast majority of these companies offer an unsubscribe option, with instructions clearly marked at the bottom of the email. These instructions tend to have a URL which I click and one of three things happen:

Link automatically unsubscribes me
Link takes me to a web page that asks for confirmation and maybe a little survey, or a list of mailings which I can opt-in or out of.
Link takes me to a login page, where I need to remember my login id, navigate to a profile and perform several other steps before I can unsubscribe. I have some mailing lists that I have never been able to unsubscribe to at all. I end up defining inbox rules to delete the mailings altogether.

The idea for a standard is this: Can we encode these unsubscribe mechanisms, or at least the first two mechanisms, in a standard way in the mail message itself, so that an email client can allow the user to simply push a button and activate the unsubscribe procedure? If done right, I could even be in my inbox view and select several emails and unsubscribe to those lists all at once. Ideally no further user interaction would be required. And certainly I want to avoid the requirement to hunt through an email for the unsubscribe link.

Since emails can come in a variety of formats, from text, to HTML to RTF, it might make sense to handle this in the mail headers rather than the email body itself.

Of course, we want this to be simple and declarative and not require general-purpose scripting support, for simplicity and security reasons.

I think this would be a relatively simple standard to create. We just need some conventions for an email to declare a RESTful API for unsubscribing to a list.

Of course, maybe there is something like this already out there?

Where’s Rob?

2008/10/07 By Rob 14 Comments

Alex Brown is saddened that I did not attend the JTC1/SC34 Plenary in Korea last week. You can view his photo essay and lament on my absence here.

[A] question being asked along the committee corridors by perplexed NB members is whether IBM has withdrawn its staff from participation SC 34. I have no idea, but IBM people are certainly conspicuous here by their total absence.

Well, I’m truly touched, and by way of reciprocation maybe I can help Alex and any other similarly perplexed attendees understand the situation better.

First, it will help if we start by taking a look at recent SC34 meetings and what the attendance record (publicly accessible) says:

Date	Location	Total Attendance	Size of US Delegation	# of IBM/OASIS Participants	# of Microsoft/ECMA Participants
Nov 2004 Plenary	Washington DC	25	6	0	0
May 2005 Plenary	Amsterdam	28	4	0	0
Nov 2005 Plenary	Atlanta	22	4	1	0
May 2006 Plenary	Seoul	30	4	2	2
Mar 2007 Plenary	Oslo	37	6	0	5
Dec 2007 Plenary	Kyoto	52	3	2	12
Apr 2008 Plenary	Oslo	37	3	1	8
July 2008 Ad Hoc 1	London	20	1	1	10
Oct 2008 Plenary	Jeju Island	35 est.	2 est.	0 est.	12 est. (estimates from Alex Brown, since no official attendance has been published)

To put it in perspective, the US SC34 shadow committee currently has around 20 members. Before Microsoft stuffed it we had around 7. Regardless, the US SC34 mirror committee typically sends a delegation of 2 or 3 people to international meetings. IBM attendance at these meetings has varied from 0 to 2. It really depends on where the meeting is being held. If it is being hosted by an NB where an IBM employee is a member, then he will typically attend. If something is on the agenda that I find interesting, then I’ll typically attend regardless of location.

Now what is really interesting is how Microsoft has increased its attendance over the years, something Alex does not mention and presumably does not find fault with. I remember introducing myself to the first Microsoft attendee at a SC34 Plenary back in 2006. He was an attorney, from Microsoft’s anti anti-trust department. An odd person to send to a technical standards committee meeting, don’t you think?

Since then, Microsoft’s representation has swelled so it now comprises 20-50% of any given meeting. And that does not count those additional “independent” companies and contractors that are employed by Microsoft to create OOXML convertors or to consult with on OOXML matters. I’m only counting those people who explicitly list “Microsoft” or “Ecma” as their corporate affiliations.

I think you’ll find no other case in SC34 attendance records of a single company sending more than a single representative. Everyone else in the world sends one person. IBM once sent two people. Microsoft sends ten or a dozen.

Despite Microsoft’s successful attempt to stuff SC34, as they did NB’s around the world, participation from IBM remains in the range of 0 to 2 participants. I’d be hard pressed to justify the expense of any greater attendance. The real work on ODF goes on in OASIS. That’s where we put our people, where they can be most effective on the technical topics related to ODF.

Alex, of course, misses all this. Sitting in a room full of non-technical Microsoft employees, the only unusual thing worth mentioning is my unclaimed badge. Good job as always, Inspector Clouseau!

In any case, the greatest concern should be given to that last row in the table, giving the attendance of the recent Jeju Island Plenary. Although the resolutions of this meeting have been posted and discussed, they lack any record of the actual attendance of this meeting. It has been the constant practice of JTC1/SC34, for many years, to record the attendance of their meetings and to post this document to the SC34 document repository and to make it publicly accessible. But in this case, the attendance record is missing entirely. It isn’t even available to SC34 members.

What are they afraid to reveal? Exactly how many Microsoft employees were at this meeting? The trend certainly has been upward. But this information is not available. Is Alex, the Convenor of WG1, only going to publicize my absence, but then fail to report who actually attended his own WG meeting? Is Alex going to express pleasure in saying “In the event this went extremely smoothly: all resolutions passed with unanimous consensus” without mentioning who exactly was there to vote for these resolutions?

I hope this is not yet a further sign that JTC1/SC34 has taken a decent into vendor domination and reduced transparency.

Oh, and where was I? I was on vacation. (Yes, I am allowed vacation). I was in Colorado, spending some time above the timberline and among the rocks.

Standards Words

2008/05/06 By Rob 7 Comments

Introduction

There are several words, more widely used than understood, that recur frequently when discussing standards. Specification and standardization requires us precisely to describe technology in such a way that practitioners in that field can achieve the goals set out in the standard. But this precision is only perfectly intelligible to those who share the same code words. What follows is a handful of the more important ones, what they mean, and how they are unintentionally confused or intentionally misused. You are at a distinct disadvantage when reading (or writing) a news article, a blog post, or evaluating an argument if you do not know the correct meaning of the following words.

Standard

Take the definition from ISO/IEC Guide 2:2004, definition 3.2:

[A] document, established by consensus and approved by a recognized body, that provides, for common and repeated use, rules, guidelines or characteristics for activities or their results, aimed at the achievement of the optimum degree of order in a given context.

NOTE Standards should be based on the consolidated results of science, technology and experience, and aimed at the promotion of optimum community benefits.

So, it is a document, a written description, not an embodiment in the form of a product, that is standardized. Its aims are the “achievement of optimum degree of order” and “promotion of optimum community benefits”, and it is achieved through consensus and consolidation.

international standard

According to ISO/IEC Guide 2:2004, definition 3.2.1.1:

[A] standard that is adopted by an international standardizing/standards organization and made available to the public.

International Standard

[An] international standard where the international standards organization is ISO or IEC

Note the distinction. With capital letters only ISO or IEC standards apply. With lowercase, other standards are included. This is a bit self-serving. ISO and IEC Standards are the only International Standards, because ISO says so. Sorry ITU, sorry CEN, sorry W3C.

So think of “International Standards” as a controlled mark of ISO, like “parmigiano reggiano” is a controlled mark of the Northern Italian Cheese Consorzio.

Normative

The normative parts of a standard are those which set out the scope and provisions of the standard. See ISO Directives, Part 2, section 3.8.

Provisions

The provisions of a standard consist of:

Requirements that must be met for conformance
Recommendations
Statements of permissible, possible or capable actions

See ISO Directives, Part 2, section 3.12.

Note that standards have specific words which denote and distinguish requirements, recommendations and capabilities. Different standards organizations have different vocabulary for this, so a W3C Recommendation, an IETF RFC and an ISO Standard may have different ways of stating the same provision. For ISO Standards, the conventions are:

“shall” and “shall not” are the normal terms for expressing requirements.
“should” and “should not” are the normal terms for expressing recommendations.
“may” and “need not” are the normal terms for expressing permission.
“can” and “cannot” are the normal terms for expressing possibility and capability.

This is necessary because of the extreme ambiguity of the English language in the area of modality. Consider the following sentences, using the word “must”:

(On hearing the doorbell ring), “Oh, that must be the mailman!” [expressing likelihood]
(To a misbehaving child) “You must obey your mother” [expressing obligation]

Or the following exchange with a teenage daughter:

Teen: “I shall return by 11pm” [simple future]
Parent: “No, you shall return by 10pm” [expressing a command]

We can be loose and still be understood, in context, in normal conversation, but in standards work we try to be precise and uniform in the use of our control vocabulary.

Conformance

This simply is a question of whether something meets the requirements of the standard. However, for many standards, there are multiple levels, perhaps even multiple classes of conformance. So you need to be very specific about what you are saying.

For example, you should not ask “Does Excel 2007 conform to OOXML?” You should ask “Is Excel 2007 a conforming transitional class SpreadsheetML Producer?” If you count it all up, OOXML probably has at least 18 distinct conformance classes, by various combinations of applications, documents, readers/writers and transitional/strict conformance classes.

Not in particular that conformance does not mean that an application implements the entire standard.

[My definition above is not very satisfactory. Anyone have something better? Is there an ISO definition of conformance?]

Compliance

This is not a typical standards term. The more typical term is “conformance”. Best to avoid it unless you are talking in regulatory or legal context. See ISO Directives, Part 2, section 6.6.1.1:

A document does not in itself impose any obligation upon anyone to follow it. However, such an obligation may be imposed, for example, by legislation or by a contract. In order to be able to claim compliance with a document, the user needs to be able to identify the requirements he/she is obliged to satisfy. The user also needs to be able to distinguish these requirements from other provisions where there is a certain freedom of choice.

Validity

This is an XML term, referring to the relationship between an XML document instance (an XML file) and a schema (the definition of the syntax of the markup language). Generally, an XML document instance is valid if it adheres to the constraints defined in the schema. The precise definition of validity will depend on the schema definition language used.

I’d welcome any suggestions for other words or definitions that should be included here.

Embrace the Reality and Logic of Choice

2008/04/30 By Rob 9 Comments

Another neo-colonialist press release from Microsoft’s CompTIA lobbying arm, this time inveighing against South Africa’s adoption of ODF as a national standard. One way to point out the absurdity of their logic is to replace the reference to ODF with references to any other useful standard that a government might adopt, like electrical standards.

When we do this, we end up with the following.

South Africa Electrical Current Adoption Outdated

South Africa’s recent adoption of the 230V/50Hz residential electrical standard represents a tact that will blunt innovation, much needed for their developing economy. The policy choice – which actually reduces electrical current choice – runs contrary to worldwide policy trends, where multiple electrical standards rule, thus threatening to separate South Africa from the wealth creating abilities of the global electrical industry.

For MonPrevAss, the Monopoly Preservation Association, the overall concern for the global electrical industry is to ensure that lawmakers adopt flexible policies and set policy targets rather than deciding on fixed rules, technologies and different national standards to achieve these targets. Such rigid approaches pull the global electrical market apart rather than getting markets to work together and boost innovation for consumers and taxpayers. “The adoption sends a negative signal to a highly innovative sector” says I.M. Atool, MonPrevAss’s Group Director, Public Policy EMEA.

The “South African Bureau of Standards” (SABS) approved the 230V/50Hz residential electrical standard on Friday 18 April as an official national standard. This adoption, if implemented, will reduce choice, decrease the benefits of open competition and thwart innovation. The irony here is that South Africa is moving in a direction which stands in stark relief to the reality of the highly dynamic market, with some 40 different electrical current conventions available today.

“Multiple co-existing electrical standards as opposed to only one standard should be favoured in the interest of users. The markets are the most efficient in creating electrical standards and it should stay within the exclusive hands of the market”, I.M. Atool explains.

In light of the recent ISO/IEC adoption of the Microsoft 240V/55Hz electrical standard, the South African decision will not lead to improvements in the electrical sector. MonPrevAss urges Governments to allow consumers and users to decide which electrical standards are best. We fear that the choice of just one electrical standard runs the risk of being outdated before it is even implemented, as well as being prohibitively costly to public budgets and taxpayers.

Governments should not restrict themselves to working with one electrical standard, and should urge legislators to refrain from any kind of mandatory regulation and discriminatory interventions in the market. The global electrical industry recommends Governments to embrace the reality and logic of choice and to devote their energies to ensuring interoperability through this choice.

Of course, this is just a rehash of an old logical fallacy, related to the old “Broken Windows” fallacy. It is like saying heart disease is a good thing because you have such a wide choice of therapies to treat it. We would all agree that it is far preferable to be healthy and have a wide choice of activities that you want to do, rather than a wide choice of solutions to a problem that you never asked for and don’t want.

Consumers don’t want a bag of adapters to convert between different formats and protocols. That is giving consumers a choice in a solution to a interoperability problem they didn’t ask for and they don’t want. Consumers want a choice of goods and services.

Observe the recent standards war with Blu-ray and HD DVD. Ask yourself:

Did consumers want a choice in formats, or did they want a wider choice in players and high definition movies?
Did movie studios want a choice in formats and either the uncertainty over choosing the winner, or the expense of supporting both formats? Or did they really just want a single format that would allow them to reach all consumers?
Did the uncertainty around the existence of two competing high definition formats help or hurt the adoption of high definition technologies in general?
Did consumers who make the early choice to go with HD DVD, say Microsoft XBox owners, benefit from having this choice?

If every private individual, and every private business has the right to adopt technology standards according to their needs, why should governments be denied that same right? Why should they be forced to take the only certain losing side of every standards war — implementing all standards indiscriminately — a choice that no rational business owner would make?

How many spreadsheet formats does Microsoft use internally for running their business on? Why should governments be denied choice in the same field that Microsoft itself exerts its right to chose?