DIS 29500, Office Open XML, was submitted for Fast Track review by Ecma as 6,045 page specification. (After the BRM, it is now longer, maybe 7,500 pages or so. We don’t know for sure, since the post-BRM text is not yet available for inspection.) Based on the original 6,045 page length, a 5-month review by JTC1 NB’s lead to 48 defect reports by NB’s, reporting a total of 3,522 defects. Ecma responded to these defect reports with 1,027 proposals, which the recent BRM, mainly through the actions of one big overnight ballot, approved.<\/p>\n
So what was the initial quality of OOXML, coming into JTC1? One measure is the defect density, which we can say is at least one defect for every 6045\/1027 = 5.8 pages. I say “at least” because this is the lower bounds. If we believed that the 5-month review represented a complete review of the text of DIS 29500, by those with relevant subject matter expertise, then we would have some confidence that all, or at least most, defects were detected, reported and repaired. But I don’t know anyone who really thinks the 5-month review was sufficient for a technical review of 6,045 pages. Further, we know that Microsoft worked actively to suppress the reporting of defects by NB’s. So the actual defect density is potentially quite a bit higher than the reported defect density.<\/p>\n
But how much higher? This is the important question. It doesn’t matter how many defects were fixed. What matters is how many remain.<\/p>\n
There are several approaches to answering this question. One approach is to look at defect “find rates”, the number of defects found per unit of time spent reviewing, and fit that to a model, typical an S-curve (sigmoid) and use that model to predict the number of defects remaining. However, we have no time\/effort data for the DIS 29500 review, so we don’t have enough data to create that model. Another approach is to randomly sample the post-BRM text and statistically estimate the defect density by this sample.<\/p>\n
Are there any other good approaches?<\/p>\n
Here is the plan. I will use the second approach. Since I do not actually have the post-BRM text, I need to make some adjustments. I’ll start with the original text, in particular Part 4, the XML reference section, at 5,220 pages, where the meat of the standard is. I’ll then create a spreadsheet and generate 200 random page numbers between 1 and 5,220. For each random page I will review the clause associated with that page and note the technical and editorial errors I find. I will then check these errors to see if any of them were addressed by BRM resolutions.<\/p>\n
Based on the above, I will be able to estimate two numbers:<\/p>\n
So if I find N defects, and 0.9N of those issues were already found during the Fast Track review and were addressed by the BRM, then we can say that the Fast Track procedure was 90% effective in finding and removing errors. Some practitioners would call that the defect removal “yield” of the process. But if we find that only 0.1N of the errors were reported and addressed by the BRM, then we’ll have a different opinion on the sufficiency of the Fast Track review.<\/p>\n
Clear enough? Microsoft is claiming something like 99% of all issues were resolved at the BRM. So let’s see if we get anything close.<\/p>\n
I’m not done with this study yet. I’m finding so many defects that recording them is taking more time than finding them. But since this is topical, I will report what I have found so far, based on the first 25 random pages, or 1\/8th completion of my target 200. I’ve found 64 technical flaws. None of the 64 flaws were addressed by the BRM. Among the defects are some rather serious ones such as:<\/p>\n
As I said, this study is still underway. I’ll list the defects I’ve found so far, and add to it as I complete the task over the next few days.<\/p>\n
That’s as far as I’ve gone. But this doesn’t look good, does it? Not only am I finding numerous errors, these errors appear to be new ones, ones not detected by the NB 5-month review, and as such were not addressed in Geneva. Since I have not come across any error that actually was fixed at the BRM, the current estimate of the defect removal effectiveness of the Fast Track process is < 1\/64 or 1.5%. That is the upper bounds. (Confidence interval? I’ll need to check on this, but I’m thinking this would be based on standard error of a proportion, where SE=sqrt((p*(1-p))\/N)), making our confidence interval 1.5% \u00b1 3%) Of course, this value will need to be adjusted as my study continues. However, it is starting to look like the Fast Track review was very shallow and that detected only a small percentage of the errors in the DIS.<\/p>\n
[20 March Update]<\/p>\n
As one commenter noted, the page numbers I’m using above are PDF page numbers, not the page numbers on bottom of each page. If I used the printed pages then I would need to deal with all the Roman numeral front matter pages as an exception. Simpler to just use the one large domain of PDF page numbers.<\/p>\n
PDF Page Number = Printed Page Number + 7<\/p>\n
I will continue to report new defects, according to the original random number list I generated. I’ll update the statistics every 25.<\/p>\n
Here’s some more for today:<\/p>\n
[End Update]<\/p>\n
I’ll continue to review the remaining 173 pages of my random sample and update the numbers and the defect list as I go. If you want to play along at home, the upcoming random page numbers will be:<\/p>\n
DIS 29500, Office Open XML, was submitted for Fast Track review by Ecma as 6,045 page specification. (After the BRM, it is now longer, maybe 7,500 pages or so. We don’t know for sure, since the post-BRM text is not yet available for inspection.) Based on the original 6,045 page length, a 5-month review by […]<\/p>\n","protected":false},"author":1,"featured_media":0,"comment_status":"open","ping_status":"open","sticky":false,"template":"","format":"standard","meta":{"_genesis_hide_title":false,"_genesis_hide_breadcrumbs":false,"_genesis_hide_singular_image":false,"_genesis_hide_footer_widgets":false,"_genesis_custom_body_class":"","_genesis_custom_post_class":"","_genesis_layout":"","footnotes":""},"categories":[6],"tags":[],"class_list":{"0":"post-161","1":"post","2":"type-post","3":"status-publish","4":"format-standard","6":"category-ooxml","7":"entry"},"_links":{"self":[{"href":"https:\/\/www.robweir.com\/blog\/wp-json\/wp\/v2\/posts\/161","targetHints":{"allow":["GET"]}}],"collection":[{"href":"https:\/\/www.robweir.com\/blog\/wp-json\/wp\/v2\/posts"}],"about":[{"href":"https:\/\/www.robweir.com\/blog\/wp-json\/wp\/v2\/types\/post"}],"author":[{"embeddable":true,"href":"https:\/\/www.robweir.com\/blog\/wp-json\/wp\/v2\/users\/1"}],"replies":[{"embeddable":true,"href":"https:\/\/www.robweir.com\/blog\/wp-json\/wp\/v2\/comments?post=161"}],"version-history":[{"count":0,"href":"https:\/\/www.robweir.com\/blog\/wp-json\/wp\/v2\/posts\/161\/revisions"}],"wp:attachment":[{"href":"https:\/\/www.robweir.com\/blog\/wp-json\/wp\/v2\/media?parent=161"}],"wp:term":[{"taxonomy":"category","embeddable":true,"href":"https:\/\/www.robweir.com\/blog\/wp-json\/wp\/v2\/categories?post=161"},{"taxonomy":"post_tag","embeddable":true,"href":"https:\/\/www.robweir.com\/blog\/wp-json\/wp\/v2\/tags?post=161"}],"curies":[{"name":"wp","href":"https:\/\/api.w.org\/{rel}","templated":true}]}}