This is Part III of a four-part post.
In Part I we surveyed of a number of different problem domains, some that resulted in a single standard, some that resulted in multiple standards.
In Part II, we described the forces that tend to unify or divide standards and showed in particular how network effects can drive the adoption of a single standard.
In this Part III we’ll look at the document formats in particular, how we got to the present point, and how and why historically there has been but a single universally-accepted document format.
In Part IV, we’ll tie it all together and show why there should be, and will be, only a single open digital document format.
It is 9:55 on an average Tuesday morning. I’m late (as usual) preparing for a meeting. With 5-minutes to go, I send out an updated meeting invite, with an updated agenda and a URL for the web conference. I also send out another email with an updated presentation attachment. It is the standard last-minute, pre-meeting shuffle that we all do. I expect that an examination of traffic statistics on IBM’s email servers shows a spike 5-minutes before every hour, as we all send out last-minute meeting updates. I login to my web conference and dial into the call. I’ll be meeting with my teammates, some in Westford, some in Raleigh, some in Portsmouth, some in Lexington, some in Dublin and some in Shanghai, a far-flung group. I’ve worked with some of these guys for years but still have never met most of them face-to-face. This is the nature of collaboration in a modern, global company. The call starts and I take a deep breath, push off my slippers and stretch my toes. Yes, I’m leading this meeting from home today.
“Don’t be impatient, Comrade Engineer; We’ve come very far, very fast”, in the words of Yevgraf Zhivago, Alec Guinness’s character in Doctor Zhivago. Let’s flash back 10 years ago and remind ourselves how we worked them…
It is 9:55 on an average Tuesday morning. I’m late (as usual) preparing for a meeting. With 5-minutes to go, I print out the agenda and handouts to the laser printer down the hall. It has printed by the time I arrive, and I sort through the three or four other print jobs to find the one that is mine. I need twelve copies for the meeting, so I join the queue at the photocopier, with everyone else who also waited to the last minute to print out the materials for their meetings. It is the standard last-minute, pre-meeting shuffle that we all do. I expect that an examination of statistics on IBM’s photocopiers shows a spike 5-minutes before every hour. I head over to the conference room and start the meeting. At the end of the call, 80% of the printed materials will be discarded, hopefully into the recycling bin. This was the nature of collaboration in a modern, global company, circa 1995.
What has changed? Why did it change? What does this mean for document formats?
My family in documents
Let me take you on a detour, back in time, to tell a 200-year family story, illustrated with official documents of the period.
I’ll start with the following excerpt from the 1930 Federal Census returns for Abington, Massachusetts, showing my grandmother, Florence Mae Cushing, then age 18, and her parents William and Mary, and household. The columns indicate the following:
- Relationship to the head of household
- Whether they own or rent their dwelling
- Value of their dwelling
- Whether they own a radio
- Whether they own a farm
- Marital condition
- Age at first marriage
- Whether they are in school
The thing that caught by eye about this record is that it lists a, “Damon, Mary K” as William’s mother-in-law, widowed, age 73, living with them. Let’s see what we can find out about this woman. First step is to find her maiden name. A search for her marriage record in Abington failed, so we tried for Mary E. Damon’s birth record, which we did find in Abington’s birth register for in 1887 revealing her mother’s maiden name as, “Chessman”:
This then allows us to find Mary K. Chessman’s birth record, also in Abington, from 1856 listing her parents as Edward and Emily:
And then from here we can go back and find the family in the 1860 Federal Census:
We see the family as owning $500 in real estate and $100 in personal property, having 5 children, the oldest 8 years old. Mary K. is only 3.
But when I skip ahead to the 1870 Census, something is clearly wrong:
As you can see above, Emily is listed as head of household, and there is no Edward. And where is our Mary K? Age age 13, she has moved out and is working as a “domestic servant” with a family of factory workers. Her sister Harriet, age 15, is also living there and working in an “eyelet factory”:
So what happened? Resolving this mystery required a bit more sleuthing, but I eventually found the answer in a response to a records request to the National Archives and Records Administration (NARA):
From this I learned that Edward Blanchard Chessman, Mary K’s father, had served in the Civil War with the Massachusetts 32nd Volunteers and had died of disease in 1863 at a military hospital in Alexandria, Virginia. This along, with a dozen pages of additional documents from NARA, detailed the pension application of his widow, the depositions of witnesses who vouched for their marriage and his service, the periodic requests for pension increases, all the way to 1903 when Emily died and her pension file was closed, marked “DEAD” with a big, bold stamp.
Since I was now tipped off to the value of pension records, I next searched for Edward’s grandfather, Ziba Chessman, who I knew had served in the Revolutionary War. I was able to locate his widow’s pension application as well:
The hand of this writer is not so easy to read, but I’d transcribe the start of it as:
Commonwealth of Massachusetts. Norfolk County. On this twenty second day of August 1838 personally appeared before Herman **** The *** of Probate in **** County, Mehitable Chessman a resident in the Town of Braintree in the County of Norfolk and state of Massachusetts aged seventy three years, who being first duly sworn according to law doth on her oath make the following declaration in order to obtain the benefit of the provision made by the Act of Congress passed July 7th 1838 entitled “An Act Granting Half Pay and Pensions to Certain Widows”, that she is the widow of Ziba Chessman late of Braintree in the County of Norfolk and state aforementioned deceased, who was a Solider in the War of the Revolution; that her said husband Ziba Chessman enlisted into Captain Isaac Thayers or Captain Nathaniel Belchers Company in the year 1775 and served a short period of time as a private with the Massachusetts Militia, around the shores of Boston, according to the best of her knowledge….
I am in awe that these records have been maintained and preserved for so long, and made available to people like me who are researching their family tree. There is a continuity of records in New England that goes back almost 400 years. Birth, education records, draft registration, military service, marriage, court appearances and eventually death and burial. Whenever your personal life crossed paths with the government, it generated a record and this record may last forever, and more importantly, once the physical preservation aspects are taken care of, these records can be read forever.
A brief history of document technology
It is somewhat odd that we’ve been debating document formats for so long and have not really said what they are. I’ll recommend the following for our discussion:
A document format consists of the conventions that allow a document to be fixed in a persistent state and then exchanged with other parties who are able to use these same conventions to read and further edit that document. If you and I understand the same document format, then you and I can exchange documents in that format and we can collaborate using that format.
Since around 1450, with Gutenberg’s first notable success of combining document production and automation, and even before (and since) with manual document production, there has been a single globally relevant interoperable document format — ink on paper. Everyone could create it, everyone could read it, everyone could exchange it. It worked then and it works now.
Some noticeable advances in documents since 1450 include the invention of pre-printed forms, around 1850. These seem obvious now, but for many years we had what were called “formulary documents” which had boilerplate text which the clerk wrote out in full for each document, in addition to the customized language for each specific instance. You can get a sense of this from Ziba Chessman’s pension application quoted earlier. From an engineering perspective you can think of this as reuse of design, but not implementation.
Having a pre-printed form was a step forward in productivity, allowing a greater degree of reuse. The Surgeon General’s form shown above is an early example. Such forms were quickly associated with bureaucracy . In fact, the first written use of the word “form” in the English language (according to the Oxford English Dictionary) was this critical view of a 19th century government office:
The waiting-rooms of that Department soon began to be familiar with his presence, and he was generally ushered into them by its janitors much as a pickpocket might be shown into a police-office; the principal difference being that the object of the latter class of public business is to keep the pickpocket, while the Circumlocution object was to get rid of Clennam. However, he was resolved to stick to the Great Department; and so the work of form-filling, corresponding, minuting, memorandum-making, signing, counter-signing, counter-counter-signing, referring backwards and forwards, and referring sideways, crosswise, and zig-zag, recommenced — Dickens, Little Dorrit (1855)
The telegraph (1837) and teletype (1910) gave new, faster ways of moving documents around. Was Morse Code a new document format? Although the telegraph operators may have worked in Morse Code, the author of the document, and the person who ultimately received and read the document still worked with ink on paper.
The typewriter (1872) increase the speed and uniformity of personal document production. This also lead to a new use for carbon paper, an invention of 1806 originally created as an aid for the blind.
In the late 1880’s, Edison’s “Autographic Printing” was commercialized as the Mimeograph, giving a cheaper method of small batch document production.
Melvin Dewey (of Dewey Decimal fame) invents the hanging file folder (1893), leading to increased efficiency of document storage and retrieval.
The Harris Automatic Press Company is incorporated in 1895, ushering in the commercial use of offset printing and a 10-fold increase in document output rates.
The invention of the Soundex algorithm by Robert Russell of Pittsburgh in 1918 allowed more efficient searching of files and cards indexed by surnames, by grouping together names which were phonetically similar.
In 1924 radio facsimile allows pictures, as well as text, to be transmitted long distances.
In 1948 Xerography gave us document duplication without the use of wet, messy chemicals.
In 1969, IBM’s Charles Goldfarb, Ed Mosher and Ray Lorie invented GML, the Generalized Markup Language, the ancestor of SGML, HTML and XML.
The 1970’s saw the rise of the first computer-based word processors, including Wang’s Office Information System.
In 1974 Xerox PARC engineers create Bravo, the first WYSIWYG word processor.
In 1975, with the rise of office automation systems and early word processors, Business Week boldly proclaimed the “Paperless Office”.
At this point we reach an important fork in the road of history. What role would the computer and office automation mean for the future of documents? Does the paperless office become a reality? Or do we remain with paper-based documents? As Xerox PARC engineers were developing the world’s first WYSIWYG word processor, at the same time they were also developing a system for transporting documents electronically, from one computer to another. But this innovation was dropped because it went against Xerox’s core business, the creation and duplication of paper documents. So the choice was made. Paper still ruled. Paper consumption went up, not down. The word processor made it easier to produce more paper, faster. The paperless office did not happen, at least not yet. More first-hand details on this fascinating topic can be read in Sellen & Harper’s The Myth of the Paperless Office. In their words, “…paper became a surrogate for the network, enabling users with different machines to share documents…”.
And so we continued, for another 20 years, of WYSIWYG word processors, WordStar, MacWrite, Writing Assistant, Manuscript, WordPerfect, Word, WordPro, etc. We all created documents and hid the files away on our hard-drives in incompatible formats. When we needed to work with others we usually just printed out the document and exchanged the printout, using the 500-year old format of ink on paper.
Let’s pause here and make some observations.
First, note the areas of sustained and recurring innovation. These have been consistent throughout the past 500 years and reflect the ongoing nature and practical concerns of business communications:
- Document authoring
- Document duplication
- Document distribution
- Filling out of forms
- Submission of forms
- Processing of forms
- Storage and Retrieval of documents
- Authentication of documents (not mentioned in the history above, but the use of Notary Publics and corporate seals has facilitated this with ink and paper documents, in some forms back to ancient Rome.)
Note also that the engineering progress and increases in efficiencies in these areas occurred without challenging the primacy of a single document format. The universality of ink and paper did not stifle innovation over these 500 years. On the contrary a single standard document format encouraged and focused innovation. We went from documents authored by pen, then set in moveable type, manually pressed, bound and distributed at the speed of a horse, to where we were circa 1995, when I authored documents on a computer, printed to a laser printer and then queued up at the photocopier to make copies of my agenda before the meeting started. Ink on paper — it was the standard document format for 500 years.
But of course, we don’t work this way anymore. Something changed, very recently. I don’t print out agendas any more. I send them via email. I don’t print out reports and review them with a red pen in hand. I mark them up electronically. In fact, unless I need to sign it or staple a receipt to it, I don’t print out anything. I think I can live out the remainder of my professional career on only 2 reams of paper.
What happened then to change this? Why is there less of an emphasis on printed output today? What does this mean for WYSIWYG? And what does this mean for document formats?
These questions and others when I finish up this series in Part IV.
20 April 2007 — Another editing pass, tightening up the language, but still too long. Added link to “The Myth of the Paperless Office”.