≡ Menu

Defining Deviancy Down

Kai Erikson, in his classic study of deviant behavior in early New England, Wayward Puritans, made the important observation that:

…the amount of deviation a community encounters is apt to remain fairly constant over time. To start at the beginning, it is a simple logistic fact that the number of deviancies which come to a community’s attention are limited by the kinds of equipment it uses to detect and handle them, and to that extent the rate of deviation found in a community is at least in part a function of the size and complexity of its social control apparatus. A community’s capacity for handling deviance, let us say, can be roughly estimated by counting its prison cells and hospital beds, its policemen and psychiatrists, its courts and clinics.

In other words, a community’s perception of social deviation is conditioned and limited by their capacity for controlling it. With equal number of punishment cells, equal-sized communities of cloistered monks and bloodthirsty pirates would perceive the same rate of deviancy. Of course the actual deviations would be different: Brother Maynard isn’t praying earnestly enough versus Greybeard slit a crewmate’s throat in the night, without warning the bunkmate below.

The late Senator from New York, Daniel Patrick Moynihan, took this idea and applied it to the social ills that America has increasingly faced since the 1960’s: mental illness, illegitimacy and violent crime. How does society react when the level of deviancy rises unexpectedly and rapidly above accepted norms? He observed, in an essay entitled, “Defining Deviancy Down”:

[…T]he amount of deviant behavior in American society has increased beyond the levels the community can “afford to recognize” and that, accordingly, we have been re-defining deviancy so as to exempt much conduct previously stigmatized, and also quietly raising the “normal” level in categories where behavior is now abnormal by any earlier standard.

I look at the current situation with Office Open XML (OOXML) in a similar way. There is a clearly defined community — JTC1 member National Bodies — with the responsibility for reviewing submitted standards. However, their capacity for exercising control is finite. The JTC1 Directives allow them a fixed period of time to review any submission. They also have a fixed number of volunteers to perform the review, and a fixed (or at least highly constrained) number of meetings to discuss and agree on review comments. So, when presented with a specification of unprecedented length (over 6,000 pages), and rather low quality, what are they to do? Spend hundreds of hours reading the specification? Write up and report thousands of errors? No, the capacity in JTC1 to deal with this level of deviancy does not exist, so the natural way for the community to cope is to to define deviancy down.

How deviant is OOXML? The 6,000+ page length is one aspect. Another is the rate at which it raced through its Ecma review, 20-times the speed of comparable specifications. Certainly, a longer specification will tend to have more problems than a shorter one, and a rushed review will find fewer problems than a thorough one. But that is speaking in generalities. Is there anything we can say for OOXML defect rates?

The Groklaw review, which occurred over a few days found a large number of serious problems. But I think we can quantify this a bit more. I tried an experiment. I used a random-number generator to generate a sample of 20 page numbers in the OOXML specification. I then read each of these pages, looking for technical errors, platform dependencies, lack of extensibility, drafting errors, etc. I did not bother noting spelling, grammatical or usage errors. I recorded how many reportable errors I found on each page. Some pages had zero problems, others had 1, 2 or even 3 problems. I even found one particularly bad error that could send OOXML back to Ecma once reported — more on that another day — but the average errors per page was 1.0. So projecting out to a 6,039 page specification this leads to a prediction of 6,000 +/- 1,000 errors. Reviewing a larger number of pages would reduce the error bars on that prediction, but we seem to be dealing with defects numbering in the thousands.

Are NB’s able to deal with a level of deviancy this great? Do they possibly have the resources to detect and report this number of errors and then verify that they are addressed? If not, the natural reaction is to define deviancy down.

For example, OOXML is currently in a 30-day review period where “contradictions” with existing ISO or IEC standards can be alleged by National Bodies (NB’s). Although the word “contradiction” is not defined in JTC1 Directives, its meaning can be seen from a resolution unanimously adopted at a JTC1 Plenary in 2000:

Resolution 27 – Consistency of JTC 1 Products

JTC 1 stresses the strong need for consistency of its products (ISs and TRs) irrespective of the route through which they were developed. Any inconsistency will confuse users of JTC 1 standards and, hence, jeopardize JTC 1’s reputation. Therefore, referring to clauses 13.2 (Fast Track) and 18.4.3.2 (PAS) of its Directives, JTC 1 reminds ITTF of its obligation to ascertain that a proposed DIS contains no evident contradiction with other ISO/IEC standards. JTC 1 offers any help to ITTF in such undertaking. However, should an inconsistency be detected at any point in the ratification process, JTC 1 together with ITTF will take immediate action to cure the problem.

The clear meaning of this is that contradictions are to be avoided, and that some of the defining characteristics of standards with contradictions are that they are not consistent, that they confuse users, and that they jeopardize JTC1’s reputation.

Further, we have precedents of other contradictions raised within JTC1, such as just last year, when the NB’s of the UK and Germany both alleged contradictions against Microsoft’s C++/CLI specification, then submitted for Fast Track processing from Ecma. The contradiction raised by the German NB (DIN) in that case said in part:

On a technical level, there are some rather different approaches between C++ and C++/CLI which can easily cause considerable confusion when both languages are considered to be “C++” or add unnecessary overhead when trying to write C++ code usable with C++ and C++/CLI. Below are a few example although if there were sufficient time to to thorough analysis of the C++/CLI document more could probably be found.

This is simple, easy to understand, and well within the spirit of the JTC1 Resolution quoted earlier.

But in a notable case of defining deviancy down, we’re starting to see the word “contradiction” defined very narrowly. For example, Microsoft’s Brian Jones suggests contradictions should be looked at this way:

[T]his is where you want to make sure that the approval of this ISO spec won’t cause another ISO standard to break. In the case of OpenXML, there really can’t be a contradiction because it’s always possible to implement OpenXML alongside other technologies. For instance, OpenOffice will soon have support for ODF and OpenXML.

An example of a contradiction would be if there was a standard for wireless technology that required the use of a certain frequency. If by using that frequency you would interfere with folks using another standard that also leverages that frequency, then there may be a contradiction.

To be quite fair, the Chinese WAPI defeat in ISO is also a precedent, but when searching for a definition of “contradiction” all precedents should be considered, not just one. Arguing exclusively from a wireless protocol standard precedent when dealing with the case of an XML markup standard is dubious when contradictions just last year were alleged to a programming language, a technology much closer to OOXML than a wireless protocol is. Surely, since C++/CLI is Microsoft’s technology they would be aware of this precedent? But still they didn’t mention it.

I ask you to consider the impact of taking Microsoft’s definition of “contradiction” and applying it to virtual technologies, like document formats, image formats, presentation formats, programming languages, operating system interfaces, API’s, security protocols, anything in the realm of software rather than hardware. None of these can ever conflict by Microsoft’s definition. Never. Therefor there is never grounds for a contradiction, and JTC1’s own Directives, which adopted the contradiction clause only a few years ago, is a procedural nullity, a no-op, meaningless, a waste of time for a large part of the technologies JTC1 has standards authority for. This is a clear example of defining deviancy down.

Let’s go back in time, 750 years ago to Thomas Aquinas and his Summa Theologica, the 13th century’s God: The Missing Manual. Aquinas had some apt words on contradictions, when discussing whether the powers of God were infinite and omnipotent (Question 25, Article 3):

Therefore, everything that does not imply a contradiction in terms, is numbered amongst those possible things, in respect of which God is called omnipotent: whereas whatever implies contradiction does not come within the scope of divine omnipotence, because it cannot have the aspect of possibility… For whatever implies a contradiction cannot be a word, because no intellect can possibly conceive such a thing.

Aquinas here allows that God can do all things that are possible, but cannot do something which is a contradiction in terms. Going back to Microsoft’s proposed definition of a contradiction, it seems that they are only willing to acknowledge a contradiction if it amounts to a co-existence problem so severe that even God could not resolve it. This seems to be a rather high hurdle to reach, and is clearly not what JTC1 intended. This is defining deviances down, way down.

This is the essential problem JTC1 has with the OOXML submission. It is too large and has too many problems with it for the control mechanisms available to JTC1 (in particular review time and volunteers) for handling the presented level of deviancy. The only recourse available to them is to define deviancy down to the level where they can handle a much smaller number of problems. Of course, this will lead to a much lower-quality ISO Standard than we are accustomed to, but what other choice is there?

This lesson has clear ramifications for Microsoft. The bigger the specification, the less throughly it will be reviewed. If you make it large enough it will barely be reviewed at all. The plan for 2007 should be to combine the .NET, OPC, XPS, JScript, J#, C#, XAML, WPF, HD Photo and whatever other specifications you have handy, put them all into one 50,000 page document, call it the “Open Microsoft Specification” rush it through Ecma and then Fast Track it into ISO. No one can really stop you. JTC1 Fast Track is broken.

{ 10 comments… add one }
  • Steven G. Johnson 2007/01/31, 00:49

    Nice article, thanks Rob! Hopefully, your discouragement about the JTC1 fast-track ballot is premature, and the non-US members will file some objections.

    Since you apparently found an additional 20 problems in the Ecma 376 draft, it would be great if you could add them at the EOOXML Objections Clearinghouse wiki for future reference. If you don’t have time to write them up carefully, it would still be great if you could jot a few quick notes at the Talk (discussion) page.

  • GaryEdwards 2007/01/31, 04:57

    I don’t think i’ve ever encountered a more polite and politically correct way of describing corruption. You make me feel coarse and cynical Rob.

    Sadly I’m not so kind to ANSI/INCiTS for letting Microsoft skate through ISO/IEC without proper review and due diligence. IMHO, it’s a national outrage and a cause for shame. But then, i’m coarse and cynical; suspecting undue influence and the ever present beauracratic willingness to accomodate.

    That deviance thing? Can i really get rid of my tin foil hat if i learn to talk like that?

    Still, i wonder if others see what i do. You have the sharpest dagger, and the most dangerous, if not noble of intentions. The difference is that you cloak your thrust under the guise of such courtly manners and erudite discussion, perhaps the bastardos will not see it coming. Maybe. For sure they don’t know how to handle your assault.

    For me though, it’s through the heart. There is no excuse for the position ANSI/INCiTS has taken on the Ecma 376 fast track ballot.

    If it looks like a duck, quacks like a duck, walks like a duck, and squirts crap like a duck, then ANSI/INCiTS is indeed the corrupt and Microsoft accommodating organization we suspect it to be.

    That they would have adopted a definition for “contradiction” suggested by Microsoft and serving Microsoft’s needs is shameless. I’m embarrassed to be an America.

    ~ge~
    Yankees in the Court of King Arthur, with a Microsoft Agenda

  • Anonymous 2007/01/31, 07:35

    You’re absolutely right that larger specifications will be more difficult to review and thus more error-prone. Look at the US tax code!

    Still, it’s a bit misleading to think of OpenXML’s 6,000 pages as one format. It really specifies three formats for totally different applications (word processing, spreadsheets, and presentation.) Given the complexity of these applications, I do not see how the format could be reduced by more than perhaps 1,000 pages.

    If you have found problems in the OpenXML spec, please post them online–maybe as a marked-up Word or PDF file.

  • Anonymous 2007/01/31, 09:51

    With the ridicoulous amount of scrutiny that is done by IBM, Groklaw and the OSS and free software communities I have no doubt that most deficiencies will be found.

    However so far there is only a few issues found that really warrent any action. And none even that can’t be corrected in a 1.1 version of the spec simular to ODF that is still correcting flaws it it’s ISO specification for future versions.

    I was happy to read
    http://tirania.org/blog/archive/2007/Jan-30.html
    though as it shows even OSS experts are divided a bit in support for OOXML as ISO format.

  • thomas 2007/01/31, 10:03

    This is not so much corruption as a severe impedance mismatch.

    Picture a carnival cruise ship in a standard olympic sized swimming pool. It WILL scrape the bottom. It won’t be able to move.

    I’m not quite sure what “fast track” is specifically for, but I’ve dealt with lots of standards (SAE, IEEE, RFCs) that deal with some very specific function, so in the dozens of pages are easy to analyze. I was also involved (including writing a reference implementation) of RFC 2440 – open PGP, and there was a LOT of discussion and refinement even though the technology and “simple things” like formats was thought to be well understood. Fast track is probably for something where there is already a body of interoperable implementations that all play by the same rules which are well known and there isn’t much debate about and just needs the rubber stamp.

    Fast tracking 6000+ pages that might or might not document one completely new implementation properly is absurd. It would be somewhat less absurd if everything external was also standard (e.g. SVG instead of VML).

    And what is the purpose of making OOXML a standard anyway? Here I don’t care if it competes with ODF or not, merely if it will truly be an interchange standard so that anyone can have a shot at creating an interoperable program. And that is where the “contradictions” become important. You can’t code for something to do one kind of spacing and a completely different kind of spacing at the same time at the same place in the same document.

    And I’ve already voiced my doubts – why bother with OOXML at all when nothing that even gets the 6000+ pages perfectly implemented (assume the programmer is psychic and will know how to resolve the “contradictions”) will be able to properly render what is emitted from Office 2007?

    Microsoft wants the ISO stamp of approval so they can call their software “open” and “standard” – which will be defined downward if this goes through.

    Microsoft could simply support ODF.

    They could have proposed extensions during the long process of standardizing ODF to make it easy – and they were on the committee.

    They can propose an extension NOW to ODF to add whatever they think is missing between Office 2007 and the current ODF standard and it would not run 6000 pages, and might be something that could be fast-tracked legitimately.

    And don’t assume Microsoft properly documented Office 2007 in OOXML – if they made a mistake, and it becomes “standard”, what will they do? Will they correct Office 2007 so it will obey the specification it submitted (or leave it “wrong” like Internet Explorer, and assume everyone will make their implementations compatible with the errors)?

    So when Microsoft Office 20xx supports neither (the current fast tracked or even a fixed) OOXML nor ODF properly what then?

  • Ben Langhinrichs 2007/01/31, 10:18

    Anonymous – ODF managed to define those three different specifications in many fewer pages. Doesn’t mean that the ODF spec was perfect, but it is certainly easy to remove many pages from the spec, and Rob has enumerated some of the ways previously.

    Rob – Great article. The only problem I have is that I don’t know how to follow the actual proceeedings except through commentary elsewhere. Has JTC1 really “accepted” the Microsoft argument, or is that just Brian Jones playing to the court of public opinion. It is not absolutely necessary to define deviancy down. Instead, would it not be possible to simply take a sufficient number of errors and declare the standard void? Perhaps not, due to the idea that you have to have a way to turn a No vote into a Yes vote, but I don’t know. Could you elaborate on the process and why the fast-track ballot could not simply find “enough” problems without having to enumerate all of them?

    Gary – Hot damn! Have at ’em. Much as I like and admire Rob’s way with language, a bit of “through the heart” activism sounds good too.

    – Ben

  • Rob 2007/01/31, 12:03

    Hi Ben, National Body’s vary in what information they make public. In the US, they released the letters they received from the public, but not their own minutes. I know members of several NB’s so I get the inside scoop, but I really can’t release the details. Let me just say that the argument that Brian uses is one I’ve heard several times from several quarters. That’s why I addressed it.

    Hi Anonymous #2, Standards reviews have their own form rules and procedures. These are very detailed and precise and are intended to ensure that the final standard product is also precise.

    A standard is not formally defective just because it is missing a feature that some users may want. The standard may fail in the market because of that, but it is not formally defective in ISO. On the other hand, a specification of great length and detail can be formally defective because of procedural or drafting errors.

  • dave.leigh 2007/01/31, 17:24

    Hi, Rob, It seems to me that one obvious course of action for the JTC is to simply reject the proposed standard outright, for the very reasons you state: too big, too rushed, too many errors. The fact is, it’s simply not suitable for the fast-track.

    The option at that time for Microsoft is to fix the errors, axe the bloat, and come back with something that IS suitable… like extensions to ODF. That way, high standards are maintained and sanity prevails. What are the odds?

  • Anonymous 2007/01/31, 20:46

    Well, I know what I would tend to do with a pile of… refuse… dropped on my desk. I’d file it in the trash can. Can’t they do that, too? Or are there other factors in play which will make that unlikely?

    Frankly, this thing has no business as a standard of any kind. The grammar may be allegedly simple, but the semantics of the stuff inside there are incomprehensible to anything other than Microsoft Office.

  • Anonymous 2007/02/01, 05:52

    Can you please create another post with the page numbers you found and the errors you saw?
    While I don’t doubt you found errors, I would feel better if I could actually verify them. Otherwise, it *seems* unsubstantiated, and is definitely less convincing.

    Thank you for your effort, and keep up the interesting and illuminating writing.

    Yours,

    JustAGuy

Leave a Comment