You can not select more than 25 topics
Topics must start with a letter or number, can include dashes ('-') and can be up to 35 characters long.
639 lines
30 KiB
639 lines
30 KiB
<?xml version="1.0" encoding="UTF-8"?>
|
|
<!DOCTYPE html PUBLIC "-//W3C//DTD XHTML 1.0 Strict//EN"
|
|
"http://www.w3.org/TR/xhtml1/DTD/xhtml1-strict.dtd">
|
|
<html>
|
|
<head>
|
|
<meta http-equiv="Content-Type" content="text/html; charset=UTF-8" />
|
|
<title>QA Framework Primer</title>
|
|
<!-- <link rel="Stylesheet" href="/QA/2002/12/qa4.css" />-->
|
|
<link rel="Stylesheet" href="/QA/2005/08/qa-style.css" />
|
|
</head>
|
|
|
|
<body>
|
|
|
|
<!-- Header -->
|
|
<div id="Logo">
|
|
<a href="http://www.w3.org/"><img alt="W3C" src="/Icons/WWW/w3c_home" width="72" height="48" /></a>
|
|
<a href="http://www.w3.org/QA/"><img alt="QA" src="/QA/images/qa" width="161" height="48" /></a>
|
|
|
|
<!-- <div id="Header">Be strict to be cool</div> -->
|
|
<div><map name="introLinks" id="introLinks" title="Introductory Links">
|
|
<div class="banner"> <a
|
|
class="bannerLink" title="W3C Activities" accesskey="A"
|
|
href="/Consortium/Activities">Activities</a> | <a class="bannerLink"
|
|
title="Technical Reports and Recommendations" accesskey="T"
|
|
href="/TR/">Technical Reports</a> | <a class="bannerLink"
|
|
title="Alphabetical Site Index" accesskey="S"
|
|
href="/Help/siteindex">Site Index</a> | <a class="bannerLink"
|
|
title="Help for new visitors" accesskey="N"
|
|
href="/2002/03/new-to-w3c">New Visitors</a> | <a
|
|
class="bannerLink" title="About W3C" accesskey="B"
|
|
href="/Consortium/">About W3C</a> | <a class="bannerLink"
|
|
title="Join W3C" accesskey="J"
|
|
href="/Consortium/Prospectus/Joining">Join W3C</a></div>
|
|
</map></div>
|
|
</div>
|
|
|
|
|
|
<!-- menuRight -->
|
|
|
|
<div id="Menu">
|
|
<p><a href="#Introduction">Introduction</a><span class="dot">·</span> <a
|
|
href="#qaf-audience">QA Framework audience</a><span class="dot">·</span> <a
|
|
href="#qaf-primer">Primer</a><span class="dot">·</span></p>
|
|
<hr />
|
|
|
|
<p class="navhead">Nearby:</p>
|
|
|
|
<p><a href="/QA/"><abbr
|
|
title="Quality Assurance">QA</abbr> Homepage</a><span
|
|
class="dot">·</span> <a href="/QA/#latest">Latest News</a><span
|
|
class="dot">·</span> <a href="/QA/#resources">QA Resources</a><span
|
|
class="dot">·</span> <a href="/QA/WG/">QA <abbr
|
|
title="Interest Group">WG</abbr></a><span class="dot">·</span> <a
|
|
href="/TR/qa-handbook/">QA Handbook</a><span class="dot">·</span> <a
|
|
href="/TR/qaframe-spec/">Spec Guidelines</a><span class="dot">·</span></p>
|
|
</div>
|
|
|
|
<!-- content -->
|
|
<div id="Content">
|
|
|
|
<!-- Your content is starting after this -->
|
|
|
|
<h1><a id="spec-title" name="spec-title">QA Framework Primer</a></h1>
|
|
|
|
<dl>
|
|
<dt>Editors:</dt>
|
|
<dd>Lofton Henderson, CGM Open</dd>
|
|
<dd>Lynne Rosenthal, NIST</dd>
|
|
<dd>Mark Skall, NIST</dd>
|
|
<dt>Last Modified:</dt>
|
|
<dd>24 August 2005</dd>
|
|
</dl>
|
|
|
|
<h2><a id="abstract" name="abstract">Abstract</a></h2>
|
|
|
|
<p><cite>QA Framework Primer </cite> provides
|
|
a general orientation to the QA
|
|
Framework. The QA Framework is a set of related documents (a
|
|
Recommendation, WG Notes, FAQ, Matrix, and Wiki) whose goal is to improve the
|
|
quality of specifications and the implementation of these specifications.
|
|
This Primer provides a roadmap to these documents. It introduces and orients
|
|
the reader by presenting the QA Framework from three different
|
|
prespectives - a document view, a role-based view, and a Working Group
|
|
milestones view.</p>
|
|
|
|
<h2><a id="status" name="status">Status of this document</a></h2>
|
|
|
|
<p>This document was previously published as a part of <cite>The QA
|
|
Handbook</cite> — see <a
|
|
href="http://www.w3.org/TR/2004/WD-qa-handbook-20040510/#primer-scenario">section
|
|
6 of first published working draft</a> of that document.</p>
|
|
|
|
|
|
<p><cite>QA Framework Primer</cite> is closely linked with and will continue
|
|
to be coordinated with the other parts of the QA Framework: <cite><a
|
|
href="http://www.w3.org/TR/qa-handbook/">The QA Handbook</a></cite>; <cite><a
|
|
href="http://www.w3.org/TR/qaframe-spec/">QA Framework: Specification
|
|
Guidelines</a></cite>; <cite><a href="2005/01/test-faq">Matrix, Test
|
|
FAQ.</a></cite>; and some more informal materials about testing in the <a
|
|
href="http://esw.w3.org/topic/QA">QA Activity Wiki</a>.</p>
|
|
|
|
<p>Comments on this document may be emailed to <a
|
|
href="mailto:www-qa-wg@w3.org">www-qa-wg@w3.org</a>, the <a
|
|
href="http://lists.w3.org/Archives/Public/www-qa-wg/">publicly archived</a>
|
|
list of the <a href="http://www.w3.org/QA/WG/">QA Working Group</a>.
|
|
Commenters please note that your comments will be <strong>publicly</strong>
|
|
archived and available, do not send information that should not be
|
|
distributed, such as private data.</p>
|
|
|
|
<h2><a id="contents1" name="contents1">Table of contents</a></h2>
|
|
<ol>
|
|
<li><a href="#Introduction">Introduction</a></li>
|
|
<li><a href="#qaf-document">Document View of the QA Framework</a></li>
|
|
<li><a href="#qaf-role">Role-based View of the QA Framework</a></li>
|
|
<li><a href="#qaf-milestone">Milestones View of the QA Framework</a></li>
|
|
<li><a href="#acknowledgments">Acknowledgments</a></li>
|
|
<li><a href="#History">Change History</a></li>
|
|
</ol>
|
|
<hr class="rule" />
|
|
|
|
<h2><a id="Introduction" name="Introduction">1. Introduction</a></h2>
|
|
|
|
<p><cite>QA Framework Primer</cite> provides a general orientation to
|
|
the QA Framework family of documents. The QA Framework
|
|
consists of a set of related documents (a Recommendation, WG Notes, FAQs,
|
|
Matrix, and Wiki) whose goal is to improve the quality of specifications and
|
|
the implementation of these specifications. This Primer provides a roadmap of
|
|
these documents. It introduces and orients the reader by presenting the QA
|
|
Framework from three different prespectives - a document view, a role-based
|
|
view, and a Working Group milestones view.</p>
|
|
<ul>
|
|
<li>The document view provides a brief summary of each QA Framework
|
|
document,</li>
|
|
<li>The role-based view identifies the documents in the QA Framework that
|
|
might interest those who fill the various roles within a Working
|
|
Group,</li>
|
|
<li>The milestones view associates the QA Framework documents with the
|
|
Working Group's progression of events and milestones.</li>
|
|
</ul>
|
|
|
|
<p>By presenting three views into the QA Framework, the reader is able to
|
|
read and focus on areas of interest. The reader is encouraged to jump into
|
|
any section of the document, reading only those sections relevant to the
|
|
reader or reading the Primer in its entirety.</p>
|
|
|
|
<p>The target audience for this Primer is anyone and everyone with an
|
|
interest in finding out about the QA Framework documents - to get a general
|
|
understanding of the documents, to identify which documents are of interest,
|
|
and/or to know when to use which document.</p>
|
|
|
|
<div id="Content1">
|
|
<p>The QA Framework consists of these existing documents:</p>
|
|
<ul>
|
|
<li><cite><a href="http://www.w3.org/TR/qa-handbook/">The QA
|
|
Handbook</a></cite>, a set of good practices that help Working Groups
|
|
improve their deliverables and their schedules</li>
|
|
<li><cite><a href="http://www.w3.org/TR/qaframe-spec/">QA Framework:
|
|
Specification Guidelines</a></cite>, documenting the most important
|
|
points to be addressed when designing and writing a specification</li>
|
|
<li><cite><a href="http://www.w3.org/TR/spec-variability/">Variability in
|
|
Specifications</a></cite>, which models how design decisions affecting
|
|
conformance change the interoperability landscape for a specification</li>
|
|
<li><cite><a href="2005/01/test-faq">Test Development FAQ</a></cite>, which
|
|
provides introductory information about the purpose of testing, how to
|
|
get started, and what the testing process involves.</li>
|
|
<li><a href="http://www.w3.org/QA/TheMatrix">Matrix of W3C
|
|
Specifications</a>, gathers and formalizes QA efforts for W3C
|
|
specifications.</li>
|
|
<li>Various pages in the <a
|
|
href="http://esw.w3.org/topic/QA">QA Wiki</a> on developing a test
|
|
suite</li>
|
|
</ul>
|
|
|
|
<p>In addition to these documents, the QAWorking Group also developed several
|
|
templates to help implement good practices described in these documents.
|
|
These templates are available (via links) from the QA Framework document to
|
|
which it applies. In particular, there is a Charter template available from
|
|
the QA Handbook and How to write a Conformance Clause template available from
|
|
the Specification Guidelines document.</p>
|
|
|
|
<h2><a id="qaf-document" name="qaf-document">2. Document View of the <cite>QA
|
|
Framework</cite></a></h2>
|
|
|
|
<p>The following provides an overview of each document in the <cite>The QA
|
|
Framework</cite>.</p>
|
|
</div>
|
|
|
|
<h3><a href="http://www.w3.org/TR/qa-handbook/">QA Handbook (QAH)</a></h3>
|
|
|
|
<p>This document is a non-normative handbook about the process and
|
|
operational aspects of certain quality assurance practices of W3C's Working
|
|
Groups, with particular focus on testability and test topics. It is a a
|
|
practical guide about applying good practices and quality assurance
|
|
techniques to WG activities, especially developing Recommendations and test
|
|
materials. It flags avoidable pitfalls, and provides techniques, tools, and
|
|
templates that will facilitate and accelerate the work of the groups.
|
|
Guidance in QAH is supported by real-world stories and examples. QAH will
|
|
help Working Groups avoid known pitfalls and benefit from experiences
|
|
gathered from the W3C Working Groups.</p>
|
|
|
|
<h3><a
|
|
href="http://www.w3.org/TR/2005/REC-qaframe-spec-20050817/">Specification
|
|
Guidelines (Spec GL)</a></h3>
|
|
|
|
<p>This document is a guide for editors of W3C specifications. It helps W3C
|
|
editors write better specifications, by making a specification easier to
|
|
interpret without ambiguity and clearer as to what is required in order to
|
|
conform. It focuses on how to define and specify conformance. It addresses
|
|
the most basic and critical topics with respect to conformance, including how
|
|
to convey what is required for an implementation in order to conform. It also
|
|
addresses how a specification might allow variation among conforming
|
|
implementations. The document presents guidelines or requirements,
|
|
supplemented with good practices, examples and techniques. As a result of
|
|
using SpecG L, Working Groups will be able to produce specifications that are
|
|
precise, easier to interpret without ambiguity or confusion, and clearer as
|
|
to what is required in order to conform.</p>
|
|
|
|
<h3><a
|
|
href="http://www.w3.org/TR/2005/WD-spec-variability-20050629/">Variability in
|
|
Specifications</a></h3>
|
|
|
|
<p>This document details some of the most important conformance-related
|
|
concepts evoked in the QA Specification Guidelines, by developing some of the
|
|
analysis that need to be considered while designing a specification and
|
|
providing advanced techniques, particularly for dealing with conformance
|
|
variability and complexity. This document analyzes how design decisions of a
|
|
specification's conformance model may affect its implementability and the
|
|
interoperability of its implementations. To do so, it introduces the concept
|
|
of variability - how much implementations conforming to a given specification
|
|
may vary among themselves - and presents a set of well-known dimensions of
|
|
variability. Its goal is to raise awareness of the potential cost that some
|
|
benign-looking decisions may have on interoperability and to provide guidance
|
|
on how to avoid these pitfalls by better understanding the mechanisms
|
|
affected by variability.</p>
|
|
|
|
<h3><a href="http://www.w3.org/QA/WG/2005/01/test-faq">Test FAQ</a></h3>
|
|
|
|
<p>Test FAQ provides introductory information about the purpose of testing,
|
|
how to get started, and what the testing process involves. This FAQ primarily
|
|
documents what is already considered good testing practice or the norm, but
|
|
it also includes a number of advanced testing goals that have not yet been
|
|
fully achieved by any Working Group. The Test FAQ document is addressed to
|
|
those who develop tests or organize testing efforts. It should also be useful
|
|
to those who develop specifications or who run tests. It stresses early
|
|
planning for testing, defines what makes a good test and examines test
|
|
reporting, publishing and packaging of tests.</p>
|
|
|
|
|
|
<h3><a href="http://www.w3.org/QA/TheMatrix">Matrix of W3C
|
|
Specifications</a></h3>
|
|
|
|
<p>The Matrix is a table of W3C Specifications that are in at least the Last Call
|
|
stage (i.e., at Last Call, Candidate Recommendation, Proposed Recommendation
|
|
or Recommendation). For each specification entry, a symbol indicates if the
|
|
specification has a conformance clause and if there are conformance tools or
|
|
test suites associated with the specification.</p>
|
|
|
|
|
|
<h3><a href="http://esw.w3.org/topic/QA">QA Wiki</a></h3>
|
|
|
|
<p>The Wiki is an open forum that allows anyone to contribute, share and
|
|
develop QA ideas, tools and methods at W3C. Anyone is encouraged to add
|
|
individual opinions, comments, and solutions to problems directly on the
|
|
pages. Topics focus on writing good specifications and building test
|
|
suites.</p>
|
|
|
|
|
|
<div id="Content11">
|
|
<h2><a id="qaf-role" name="qaf-role">3. Role-based View of the <cite>QA
|
|
Framework</cite></a></h2>
|
|
|
|
<p>This section identifies the parts of the <cite>The QA Framework</cite>
|
|
that might interest those who fill the various roles within a Working Group
|
|
or that interact with a Working Group. While each part of framework is
|
|
targeted itself at a specific principal audience, the various parts might
|
|
have somewhat broader interest and applicability.</p>
|
|
<dl>
|
|
<dt>all participants</dt>
|
|
<dd>For any (potential) Working Group participant, the <a
|
|
href="qa-handbook.html#Early">early planning and commitment parts</a>
|
|
of <cite><a href="http://www.w3.org/TR/qa-handbook/">The QA
|
|
Handbook</a></cite> might provide helpful context for understanding
|
|
what the group has committed to deliver. Familiarity with the <cite><a
|
|
href="http://www.w3.org/TR/qaframe-spec/">Specification
|
|
Guidelines</a></cite> will be helpful to any participant who actively
|
|
participates in the advancement of the specifications to
|
|
Recommendation.</dd>
|
|
<dt>spec editors and authors</dt>
|
|
<dd>As for all participants, <cite><a
|
|
href="http://www.w3.org/TR/qa-handbook/">The QA Handbook</a></cite>
|
|
might be interesting, for shedding some light on the context in which
|
|
the group is operating. A good working understanding the Principles and
|
|
Good Practices of <cite>Specification Guidelines</cite>, together with
|
|
its collected examples, tools, and templates, should be a valuable
|
|
resource in choosing document structure, formats, techniques and
|
|
conformance-related designs that will facilitate production of a
|
|
high-quality specification.</dd>
|
|
<dt>Chair</dt>
|
|
<dd>As the person ultimately responsible for all aspects of the group's
|
|
work, a familiarity with the guidance for operations and process of
|
|
<cite><a href="http://www.w3.org/TR/qa-handbook/">The QA
|
|
Handbook</a></cite> should be helpful — Chairs and Staff Contacts
|
|
are the principal intended audience. Some familiarity
|
|
with the advice and guidance of <cite><a
|
|
href="http://www.w3.org/TR/qaframe-spec/">Specification
|
|
Guidelines</a></cite> should be helpful as well, as the Chairs
|
|
ultimately oversees the advancement of the specifications.</dd>
|
|
<dt>Test Task Force participant</dt>
|
|
<dd>Those who are active in building the test materials of the Working
|
|
Group should benefit from reading the <a href="2005/01/test-faq">Test
|
|
FAQ</a> as well as the articles regarding test materials in the <a
|
|
href="http://esw.w3.org/topic/QA">QA Wiki</a>, and from their
|
|
associated examples, techniques, and tools. Because of the close
|
|
dependency of test materials on the functional specifications, a
|
|
familiarity with the <cite><a
|
|
href="http://www.w3.org/TR/qaframe-spec/">Specification
|
|
Guidelines</a></cite> could be useful as well.</dd>
|
|
<dt>Test Task Force lead</dt>
|
|
<dd>The person who manages the Working Group's test suite projects should
|
|
have working understandings of guidance and techniques for
|
|
specifications of <cite><a
|
|
href="http://www.w3.org/TR/qaframe-spec/">Specification
|
|
Guidelines</a></cite> , as well as the main points of the <a
|
|
href="2005/01/test-faq">Test FAQ</a>.</dd>
|
|
<dt>Non-Working Group spec reviewers</dt>
|
|
<dd>Whether from other Working Groups, or the public at large, the
|
|
<cite><a href="http://www.w3.org/TR/qaframe-spec/">Specification
|
|
Guidelines</a></cite> will be helpful to those who review a Working
|
|
Group's specifications, by providing some objective metrics by which to
|
|
measure the specifications.</dd>
|
|
<dt>Reviewers of Activity proposals & charters</dt>
|
|
<dd>For those W3C Members who will be reviewing Activity proposals and
|
|
proposed Working Group charters, and helping to form their Advisory
|
|
Committee Representative's positions, the <a
|
|
href="http://www.w3.org/TR/qa-handbook/#Early">early planning and
|
|
commitment parts</a> of <cite><a
|
|
href="http://www.w3.org/TR/qa-handbook/">The QA Handbook</a></cite>
|
|
might be helpful in evaluating whether or not the group's attention to
|
|
test and quality deliverables is appropriate and consistent with the
|
|
Working Group's overall program of work.</dd>
|
|
</dl>
|
|
|
|
<h2><a id="qaf-milestone" name="qaf-milestone">4. Milestone View of the
|
|
<cite>QA Framework</cite></a></h2>
|
|
|
|
<p>This section identifies which parts (sections and documents) of the QA Framework are useful during the life of a Working Group. It progresses through significant milestones in a Working Group's life, from writing a Charter through publishing Recommendations, and looks at associated test suite and other quality
|
|
assurance activities.</p>
|
|
|
|
<h3><a id="qa-commit" name="qa-commit">First Step — QA
|
|
Commitment</a></h3>
|
|
|
|
<p>Because QA is properly an integral part of the activities of each Working
|
|
Group, each Working Group has to consider and commit to a set of QA
|
|
deliverables appropriate to its work. A spectrum of possibilities are
|
|
discussed and illustrated in the <a
|
|
href="http://www.w3.org/TR/qa-handbook/#Early">early planning and
|
|
commitment</a> module of <cite><a
|
|
href="http://www.w3.org/TR/qa-handbook/">The QA Handbook</a></cite>.</p>
|
|
|
|
<p>If a Working Group is being newly formed, and if the it is able to
|
|
anticipate and agree at Charter time on deliverables like test suites, then
|
|
it should consider documenting those QA deliverables in its Charter, just as
|
|
it does all other deliverables. Again, see the <a
|
|
href="http://www.w3.org/TR/qa-handbook/#Early">early planning and
|
|
commitment</a> section of <cite><a
|
|
href="http://www.w3.org/TR/qa-handbook/">The QA Handbook</a></cite>. A
|
|
Working Group being re-chartered is a similar case to a newly formed group,
|
|
although the scope and direction of its work might be clearer.</p>
|
|
|
|
<p>For an already-chartered Working Group undertaking new test and other
|
|
QA-related activities, if these deliverables are not documented in the
|
|
Charter already, then there are a couple of options. The <a
|
|
href="http://www.w3.org/Consortium/Process">W3C Process Document</a>
|
|
describes how to amend a Charter to accommodate significant new deliverables,
|
|
if it wishes to take this route.</p>
|
|
|
|
<h3><a id="setup-proc" name="setup-proc">Set Up Processes and
|
|
Logistics</a></h3>
|
|
|
|
<p>Once the Working Group is off and running, and assuming that it has
|
|
planned on some test- or other quality-related deliverables, the next step is
|
|
to chose and document the processes and logistics that it will use for its QA
|
|
activities. These include such typical details as:</p>
|
|
<ul>
|
|
<li>QA point-of-contact;</li>
|
|
<li>"/Test/" home page in the group's Web space;</li>
|
|
<li>test materials email discussion list;</li>
|
|
<li>Working Group process document for QA;</li>
|
|
<li>decisions about test materials licenses.</li>
|
|
</ul>
|
|
|
|
<p>The sections within the <a
|
|
href="http://www.w3.org/TR/qa-handbook/#Day-to-day">Day-to-Day operations</a>
|
|
module of <cite><a href="http://www.w3.org/TR/qa-handbook/">The QA
|
|
Handbook</a></cite> give good-practice advice about how to do this, plus
|
|
examples and a handy template for writing a QA Process Document.</p>
|
|
|
|
<h3><a id="plan-write" name="plan-write">Planning and Writing the
|
|
Specification</a></h3>
|
|
|
|
<p>There is a tight bond between how a specification is written on the one
|
|
hand, and on the other hand its testability and its suitability as a basis
|
|
for interoperable implementations.</p>
|
|
|
|
<h4>New specification</h4>
|
|
|
|
<p><cite><a href="http://www.w3.org/TR/qaframe-spec/">QA Framework:
|
|
Specification Guidelines</a></cite> should be applied from very beginning.
|
|
Among the key topics that it addresses are:</p>
|
|
<ul>
|
|
<li>conformance model;</li>
|
|
<li>normative language usage;</li>
|
|
<li>writing with test assertions;</li>
|
|
<li>optional features, extensibility;</li>
|
|
<li>profiles, levels, conformance claims.</li>
|
|
</ul>
|
|
|
|
<p>Consider the advice of <cite><a
|
|
href="http://www.w3.org/TR/qaframe-spec/">QA Framework: Specification
|
|
Guidelines</a></cite> even at the stage of planning the structure and
|
|
presentation style of the spec. Along with W3C "pubrules" and <cite>W3C
|
|
Manual of Style</cite>, spec editors should refer to the spec guidelines
|
|
throughout their work, on topics like testable language, clarity,
|
|
conciseness.</p>
|
|
|
|
<h4>New Edition of specification</h4>
|
|
|
|
<p>A new Edition of the same functional level of a specification is typically
|
|
used for incorporation of errata (e.g., <cite>XML 1.0 Second Edition</cite>).
|
|
Normally, this should not be considered a good time to align a specification
|
|
to <cite>QA Framework: Specification Guidelines</cite> — the changes
|
|
associated with such alignment could significantly disrupt and restructure
|
|
the specification.</p>
|
|
|
|
<h4>New Version of specification</h4>
|
|
|
|
<p>A new Version of the specification refers to a significant functional
|
|
change and enhancement. This presents a good opportunity to improve the
|
|
testability and implementability of the specification, as just described for
|
|
new specifications.</p>
|
|
|
|
<h3><a id="progress-spec" name="progress-spec">Reviewing and Progressing the
|
|
Specification</a></h3>
|
|
|
|
<p>This stage in the specification's life has two significant aspects:</p>
|
|
<ul>
|
|
<li>the advancement of the specification to First Public Working Draft and
|
|
beyond;</li>
|
|
<li>and, the need to synchronize production of test suites and tools more
|
|
closely with the spec progression.</li>
|
|
</ul>
|
|
|
|
<p>When the specification is published in <a href="http://www.w3.org/TR/"
|
|
shape="rect">TR space</a>, then W3C Members not participating to the Working
|
|
Group and the general public begin to review and comment. It would be
|
|
valuable that such reviewers consult and understand the material in <cite>QA
|
|
Framework: Specification Guidelines</cite> — it gives and informed set
|
|
of evaluation criteria about the conformance, testability, and
|
|
interoperability aspects of the specification.</p>
|
|
|
|
<p>Working Group participants and especially Test Task Force participants
|
|
should refer to the good-practice pieces about <a
|
|
href="http://www.w3.org/TR/qa-handbook/#gp-sync-deliverables">advancement
|
|
criteria and synchronization</a> (between specs and test materials) of
|
|
<cite>The QA Handbook</cite>. Projects enter <cite><a
|
|
href="/QA/TheMatrix">The Matrix</a></cite> at Last Call Working Draft (if not
|
|
sooner). A de-facto process convention is emerging, that there should be
|
|
significant conformance test materials before finishing Candidate
|
|
Recommendation phase. This timing coordinates with the explicit process
|
|
requirement of two interoperable implementations.</p>
|
|
|
|
<h3><a id="build-tm" name="build-tm">Designing and Building Test
|
|
Materials</a></h3>
|
|
|
|
<p>There are several scenarios for how the Working Group "builds" its
|
|
conformance test materials:</p>
|
|
<ul>
|
|
<li>design and build test cases and test tools in the group, from
|
|
scratch;</li>
|
|
<li>import completed test materials from an outside group or
|
|
organization;</li>
|
|
<li>assemble the test materials from contributed test cases and
|
|
materials.</li>
|
|
</ul>
|
|
|
|
<p>All these topics are addressed in more details in the <a
|
|
href="2005/01/test-faq">Test FAQ</a>.</p>
|
|
|
|
<h4>Intra-group build</h4>
|
|
|
|
<p>Before starting the development, the Test Task Force participants would
|
|
benefit from a familiarity with the material in the <a
|
|
href="2005/01/test-faq">Test FAQ</a> and optionally in the articles of the
|
|
<a href="http://esw.w3.org/topic/QA">QA Wiki</a> regarding test suites.
|
|
There is useful information for both high-level planning — e.g., does
|
|
the Working Group want breadth-first Basic Effectivity or a fully detailed
|
|
suite? — as well as specific details for building the individual test
|
|
cases.</p>
|
|
|
|
<p>Another aspect of building test materials is an acceptance procedure for
|
|
the individual bits, as they are built. This is addressed in the <a
|
|
href="http://www.w3.org/QA/WG/2005/01/test-faq#review">Test FAQ.</a> with
|
|
examples of review guidelines..</p>
|
|
|
|
<h4><a name="AssembleContributions" id="AssembleContributions">Imported test
|
|
materials</a></h4>
|
|
|
|
<p>Several high-quality test suites have been developed outside of the
|
|
relevant W3C Working Group, and then transferred to the group, or have been
|
|
the result of assembling pieces from various contributions. Groups which are
|
|
considering such a transfer should refer to <a
|
|
href="http://www.w3.org/TR/qa-handbook/#Acquire">test materials
|
|
acquisition</a> module of <cite><a
|
|
href="http://www.w3.org/TR/qa-handbook/">The QA Handbook</a></cite>. Clearly,
|
|
the <a href="http://esw.w3.org/topic/TestMaterialsQualities">quality of the
|
|
candidate test materials</a> should be carefully assessed, but also, the
|
|
licensing issues (as discussed in the <a
|
|
href="http://www.w3.org/TR/qa-handbook/#IANAL">QA Handbook</a> needs careful
|
|
planning.</p>
|
|
|
|
<h3><a id="publish-tm" name="publish-tm">Publication of Test
|
|
Materials</a></h3>
|
|
|
|
<p>Typically, a Working Group Test Task Force will want to publish releases
|
|
of test materials, particularly as the specification advances through its
|
|
final maturity levels (e.g., Proposed Recommendation) towards Recommendation. </p>
|
|
|
|
<p>One hurdle on the way to publication is legal — deciding and
|
|
agreeing on suitable publication licenses. Advice on navigating this
|
|
potential quagmire is presented in the <a
|
|
href="http://www.w3.org/TR/qa-handbook/#IANAL">licensing module</a> of
|
|
<cite>The QA Handbook</cite>.</p>
|
|
|
|
<p>As soon as the test materials become public, then there is definite need
|
|
for a procedure to process challenges to correctness, make determinations,
|
|
and appeal decisions.</p>
|
|
|
|
<p>Publication of test materials often comprises an implicit (or explicit)
|
|
invitation for contributions. The considerations described in <a
|
|
href="#AssembleContributions">"Imported test materials"</a> are equally
|
|
applicable here.</p>
|
|
|
|
<h3><a id="rec-n-beyond" name="rec-n-beyond">Recommendation stage and
|
|
Beyond</a></h3>
|
|
|
|
<p>When the specification reaches Recommendation, there is typically a
|
|
concurrent publication of the test materials. This might be considered a
|
|
"final" publication, or ongoing development may still be planned. In any case, a maintenance procedure must be in place for the test
|
|
materials. Firstly, there are tie-ins between approved specification errata
|
|
and applicability of particular tests. Secondly, there is the above-discussed need for both
|
|
challenge/review/appeal processes. Finally, even if the Working Group ceases
|
|
active development of test materials, it may want to continue to consider
|
|
submissions, and review and integrate them.</p>
|
|
|
|
<h3><a id="life-after" name="life-after">Life after Working Group</a></h3>
|
|
|
|
<p>It is possible that the Working Group and its Test Task Force may disband
|
|
after its charter expires. The various aspects of this situation are <a
|
|
shape="rect" href="http://www.w3.org/TR/qa-handbook/#Life">introduced</a> in
|
|
<cite>The QA Handbook</cite>.</p>
|
|
|
|
<h2><a name="acknowledgments" id="acknowledgments">5. Acknowledgments</a></h2>
|
|
|
|
<p>The following QA Working Group and Interest Group participants have
|
|
contributed to this document:</p>
|
|
<ul>
|
|
<li>Daniel Dardailler (W3C),</li>
|
|
<li>Dimitris Dimitriadis (Ontologicon),</li>
|
|
<li>Karl Dubost (W3C),</li>
|
|
<li>Dominique Hazaël-Massieux (W3C),</li>
|
|
<li>Lofton Henderson (CGM Open),</li>
|
|
<li>Richard Kennedy (Boeing),</li>
|
|
<li>David Marston (IBM Research),</li>
|
|
<li>Patrick Curran (Sun Microsystems),</li>
|
|
<li>Lynne Rosenthal (NIST),</li>
|
|
<li>Mark Skall (NIST),</li>
|
|
<li>Andrew Thackrah (Open Group),</li>
|
|
<li>Olivier Théreaux (W3C),</li>
|
|
<li>Jeremy Carroll (Hewlett-Packard)</li>
|
|
</ul>
|
|
|
|
<h2><a name="History" id="History">6. Change history</a></h2>
|
|
<dl>
|
|
<dt>2005-08-18</dt>
|
|
<dd><ul>
|
|
<li>revised Introduction to reflect 3 views</li>
|
|
<li>added document overview section (view 1)</li>
|
|
<li>renamed sections for views 2 and 3</li>
|
|
<li>updated entire document to read better and to reflect changes to
|
|
Framework documents</li>
|
|
</ul>
|
|
</dd>
|
|
<dt>2004-11-04</dt>
|
|
<dd><ul>
|
|
<li>removed TestGL references, since its development has stopped</li>
|
|
<li>referred to Wiki pages when relevant</li>
|
|
<li>updated with ref to Variability in Spec</li>
|
|
</ul>
|
|
</dd>
|
|
<dt>2004-08-30</dt>
|
|
<dd><ul>
|
|
<li>Initial version created by separation from QAH.</li>
|
|
</ul>
|
|
</dd>
|
|
</dl>
|
|
</div>
|
|
|
|
<!-- Your content is finishing before this -->
|
|
</div>
|
|
<!-- Footer -->
|
|
|
|
<div class="disclaimer">
|
|
<a href="http://validator.w3.org/check/referer"><img
|
|
src="http://validator.w3.org/images/vxhtml10" alt="Valid XHTML 1.0!"
|
|
height="31" width="88" /></a>
|
|
|
|
<p>Last modified $Date: 2005/09/12 17:45:53 $ by $Author dom
|
|
$</p>
|
|
|
|
<p class="policyfooter"><a rel="Copyright"
|
|
href="/Consortium/Legal/ipr-notice#Copyright">Copyright</a> © 2000-2004 <a
|
|
href="/"><acronym
|
|
title="World Wide Web Consortium">W3C</acronym></a><sup>®</sup> (<a
|
|
href="http://www.csail.mit.edu/"><acronym
|
|
title="Massachusetts Institute of Technology">MIT</acronym></a>, <a
|
|
href="http://www.ercim.org/"><acronym
|
|
title="European Research Consortium for Informatics and Mathematics">ERCIM</acronym></a>,
|
|
<a href="http://www.keio.ac.jp/">Keio</a>), All Rights Reserved. W3C <a
|
|
href="/Consortium/Legal/ipr-notice#Legal—Disclaimer">liability</a>, <a
|
|
href="/Consortium/Legal/ipr-notice#W3C—Trademarks">trademark</a>, <a
|
|
rel="Copyright" href="/Consortium/Legal/copyright-documents">document use</a>
|
|
and <a rel="Copyright" href="/Consortium/Legal/copyright-software">software
|
|
licensing</a> rules apply. Your interactions with this site are in accordance
|
|
with our <a href="/Consortium/Legal/privacy-statement#Public">public</a> and
|
|
<a href="/Consortium/Legal/privacy-statement#Members">Member</a> privacy
|
|
statements.</p>
|
|
</div>
|
|
</body>
|
|
</html>
|