You can not select more than 25 topics
Topics must start with a letter or number, can include dashes ('-') and can be up to 35 characters long.
671 lines
30 KiB
671 lines
30 KiB
<?xml version="1.0" encoding="iso-8859-1"?>
|
|
<!DOCTYPE html PUBLIC "-//W3C//DTD XHTML 1.0 Transitional//EN"
|
|
"http://www.w3.org/TR/xhtml1/DTD/xhtml1-transitional.dtd">
|
|
<html xmlns="http://www.w3.org/1999/xhtml">
|
|
<head>
|
|
<meta http-equiv="Content-Type" content="text/html; charset=iso-8859-1" />
|
|
<title>Conformance Testing and Certification Model for W3C
|
|
Specifications</title>
|
|
<link rel="stylesheet" type="text/css"
|
|
href="http://www.w3.org/StyleSheets/TR/W3C-NOTE" />
|
|
</head>
|
|
|
|
<body>
|
|
|
|
<div class="head">
|
|
<a href="http://www.w3.org/"><img height="48" width="72" alt="W3C"
|
|
src="http://www.w3.org/Icons/w3c_home" /></a>
|
|
|
|
<h1>Conformance Testing and Certification Model for W3C Specifications</h1>
|
|
|
|
<h2>W3C Working Draft 2 January 2002</h2>
|
|
<dl>
|
|
<dt>This version:</dt>
|
|
<dd><a
|
|
href="http://www.w3.org/QA/2002/01/Note-qa-certif-20020102">http://www.w3.org/QA/2002/01/Note-qa-certif-20020102</a></dd>
|
|
<dt>Latest version:</dt>
|
|
<dd>http://www.w3.org/QA/qa-certif</dd>
|
|
<dt>Previous version:</dt>
|
|
<dd>This is the first draft from the W3C QA Activity. <br />
|
|
It it based on a NIST White Paper entitled: Conformance Testing and
|
|
Certification Model for Software Specifications, by Lynne Rosenthal,
|
|
Mark Skall, and Lisa Carnahan</dd>
|
|
<dt>Editors:</dt>
|
|
<dd>Daniel Dardailler (<a
|
|
href="mailto:danield@w3.org">danield@w3.org</a>)
|
|
<p></p>
|
|
</dd>
|
|
</dl>
|
|
|
|
<p class="copyright"><a
|
|
href="http://www.w3.org/Consortium/Legal/ipr-notice#Copyright">Copyright</a>
|
|
©2001 <a href="http://www.w3.org/"><abbr
|
|
title="World Wide Web Consortium">W3C</abbr></a><sup>®</sup> (<a
|
|
href="http://www.lcs.mit.edu/"><abbr
|
|
title="Massachusetts Institute of Technology">MIT</abbr></a>, <a
|
|
href="http://www.inria.fr/"><abbr xml:lang="fr" lang="fr"
|
|
title="Institut National de Recherche en Informatique et Automatique">INRIA</abbr></a>,
|
|
<a href="http://www.keio.ac.jp/">Keio</a>), All Rights Reserved. W3C <a
|
|
href="http://www.w3.org/Consortium/Legal/ipr-notice#Legal_Disclaimer">liability</a>,
|
|
<a
|
|
href="http://www.w3.org/Consortium/Legal/ipr-notice#W3C_Trademarks">trademark</a>,
|
|
<a
|
|
href="http://www.w3.org/Consortium/Legal/copyright-documents-19990405">document
|
|
use</a> and <a
|
|
href="http://www.w3.org/Consortium/Legal/copyright-software-19980720">software
|
|
licensing</a> rules apply.</p>
|
|
<hr />
|
|
</div>
|
|
|
|
<h2><a id="abstract" name="abstract">Abstract</a></h2>
|
|
|
|
<p>The use of conformity assessment as a means by which buyers and sellers
|
|
can communicate requirements will increase as information technology systems
|
|
and applications grow more complex. Models for conformance testing and
|
|
certification programs are necessary to understand principles and issues that
|
|
are essential for successful conformity assessment programs. This paper
|
|
presents one such model by identifying key roles, activities and products
|
|
involved in any conformance testing and certification program. This model has
|
|
been successfully used by NIST in helping private-sector organizations
|
|
establish their certification programs.</p>
|
|
|
|
<h2><a id="status" name="status">Status of this document</a></h2>
|
|
|
|
<p>This document is a Note, made available by the W3C Quality Assurance
|
|
Activity (QAA) for discussion on the QA email discussion list. It is
|
|
submitted as to stimulate discussion within the <a
|
|
href="http://www.w3.org/QA/IG">W3C Quality Assurance Interest Group</a>
|
|
regarding guidance to external organizations that may wish to establish
|
|
certification programs.</p>
|
|
|
|
<p>It may form the basis of a W3C Note and as such may be modified, replaced
|
|
or obsoleted by other documents. Publication of this document does not imply
|
|
endorsement by the W3C, its membership or its staff. It is inappropriate to
|
|
use W3C Working Drafts as reference material or to cite them as other than
|
|
"work in progress". Please send comments on the publicly archived list <a
|
|
href="mailto:www-qa@w3.org">www-qa@w3.org</a>.</p>
|
|
|
|
<h2>Table of Content</h2>
|
|
|
|
<p>1. <a href="#Communicat">Introduction</a></p>
|
|
|
|
<p>2. <a href="#Conformanc">Conformance Testing and Certification
|
|
Model</a></p>
|
|
|
|
<p>3. <a href="#Roles">Roles</a></p>
|
|
|
|
<p>4. <a href="#Activities">Activities</a></p>
|
|
|
|
<p>5. <a href="#Products">Products</a></p>
|
|
|
|
<p>6. <a href="#Examples">Examples</a></p>
|
|
|
|
<p>7. <a href="#Conclusion">Conclusion</a></p>
|
|
|
|
<p><a href="#References">References</a></p>
|
|
|
|
<p></p>
|
|
<hr />
|
|
|
|
<h2><a name="Communicat" id="Communicat">1. Introduction</a></h2>
|
|
|
|
<p>As the pervasiveness of information technology increases, so does the
|
|
importance of ensuring the quality of products (i.e., software and systems).
|
|
Conformance testing is defined in ISO/IEC Guide 2, "as any activity concerned
|
|
with determining directly or indirectly that relevant requirements are
|
|
fulfilled". The W3C QA activity's goal is to make sure that all W3C
|
|
specifications are covered by adequate tools supporting their conformance
|
|
testing, but it is not chartered, nor is the W3C itself, to run certification
|
|
program. The QA activity on the other hand is interested in understanding
|
|
principles and issues that are essential for the development of successful
|
|
conformity assessment programs run externally to W3C. The goal of this
|
|
document is to describe a model by identifying key roles, activities and
|
|
products involved in any conformance testing and certification program.</p>
|
|
|
|
<h3>Communication between Buyers and Sellers</h3>
|
|
|
|
<p>In the marketplace, conformance testing provides a vehicle for exchanging
|
|
information between buyer and seller. It increases a buyer's (and/or user's)
|
|
confidence in a product and its ability to meet their needs. It provides an
|
|
independent, objective method for evaluating products and not becoming
|
|
locked-into a single vendor. For sellers (and developers), conformance
|
|
testing can help to substantiate claims that a product meets the given
|
|
specification.</p>
|
|
|
|
<p>Conformance testing is a means of measuring whether a product faithfully
|
|
implements a specification. The level and formality of the testing are
|
|
determined by the market - the requirements of the buyer directly or an
|
|
organization acting on behalf of a community of buyers, or by regulation
|
|
(e.g., safety, health, national security concerns). For example, some
|
|
programs may require a very formal testing and certification approach
|
|
consisting of independent (i.e., third party), nationally accredited testing
|
|
laboratories while others may be more appropriate for self declaration and
|
|
demonstration testing.</p>
|
|
|
|
<p>The sections below describe a generic model for establishing a conformance
|
|
testing and certification program. It describes the processes and procedures
|
|
for establishing, administering a testing program. While much has changed
|
|
regarding conformity assessment given the growth and changes in the software
|
|
industry, the conformance and certification model has not. Examples are used
|
|
to describe how the model is applied to support the changes in the software
|
|
industr</p>
|
|
|
|
<h2><a name="Conformanc" id="Conformanc">2. Conformance Testing and
|
|
Certification Model</a></h2>
|
|
|
|
<p>It is well recognized that conformance testing and certification is a way
|
|
to ensure that "standard-based" products are implemented. The advantage
|
|
afforded by testing and certification are fairly obvious: quality products,
|
|
competitive markets with more choices, commodity pricing, and less
|
|
opportunity to become "locked in" to a particular vendor. Moreover, a testing
|
|
and certification program based on well understood and sound principles will
|
|
be acceptable and credible to its community of users.</p>
|
|
|
|
<p>The conformance testing and certification model described herein contains
|
|
the fundamental roles, activities, and products that are necessary in
|
|
administering and operating a testing and certification program (see Table
|
|
1). By adjusting and modifying the various activities, roles and products,
|
|
the model can be applied and used in establishing any testing and
|
|
certification program. Figure 1 highlights the interactions between the roles
|
|
and activities. The model allows for roles, activities and/or products to be
|
|
consolidated or further partitioned.</p>
|
|
|
|
<p> </p>
|
|
|
|
<table width="90%" cellspacing="0" cellpadding="0" border="2">
|
|
<tbody>
|
|
<tr>
|
|
<th>Roles</th>
|
|
<th>Activities</th>
|
|
<th>Products</th>
|
|
</tr>
|
|
<tr>
|
|
<td>Buyer</td>
|
|
<td>Require Certification</td>
|
|
<td>Specification</td>
|
|
</tr>
|
|
<tr>
|
|
<td>Seller</td>
|
|
<td>Test Implementation</td>
|
|
<td>Implementation Under Test (IUT)</td>
|
|
</tr>
|
|
<tr>
|
|
<td>Test Laboratory (TL)</td>
|
|
<td>Recognize Test Method</td>
|
|
<td>Test Method</td>
|
|
</tr>
|
|
<tr>
|
|
<td>Certificate Issuer</td>
|
|
<td>Recognize Tester</td>
|
|
<td>Test Report</td>
|
|
</tr>
|
|
<tr>
|
|
<td>Control Board (CB)</td>
|
|
<td>Validate Results</td>
|
|
<td>Certification Program Policy</td>
|
|
</tr>
|
|
<tr>
|
|
<td></td>
|
|
<td>Anser Programmatic Queries</td>
|
|
<td>Testing Laboratory Criteria</td>
|
|
</tr>
|
|
<tr>
|
|
<td></td>
|
|
<td>Answer Test Method Queries</td>
|
|
<td>Certificate of Conformance</td>
|
|
</tr>
|
|
<tr>
|
|
<td></td>
|
|
<td>Resolve Test Method Disputes</td>
|
|
<td></td>
|
|
</tr>
|
|
<tr>
|
|
<td></td>
|
|
<td>Validate Conformance</td>
|
|
<td></td>
|
|
</tr>
|
|
<tr>
|
|
<td></td>
|
|
<td>Issue Certificate</td>
|
|
<td></td>
|
|
</tr>
|
|
</tbody>
|
|
</table>
|
|
|
|
<div align="center">
|
|
Table 1: Roles, Activities, and Products</div>
|
|
<br />
|
|
<br />
|
|
|
|
|
|
<div align="center">
|
|
<img src="interact.gif" border="0"
|
|
title="Figure 1: Interactions Among Roles and Activities"
|
|
alt="Interaction among roles and activities" /></div>
|
|
|
|
<div align="center">
|
|
Figure 1: Interaction Among Roles and Activities</div>
|
|
<br />
|
|
<br />
|
|
|
|
|
|
<p>While actual testing and certification can be carried out by various
|
|
organizations, it is essential that there be a centralized sponsor or owner
|
|
of the testing and certification program. The sponsor has a fundamental
|
|
interest in ensuring the success of the program. Typically, the sponsor
|
|
establishes and maintains the conformance testing and certification program.
|
|
It assumes responsibility for insuring that the components of the program are
|
|
in place and becomes the centralized source for information about the
|
|
program. The sponsor may be composed of one or more organizations. Examples
|
|
of sponsors are consortia, trade associations, standards groups, or a
|
|
government agency. More often than not, the sponsor of the program is also
|
|
the Certificate Issuer.</p>
|
|
|
|
<h2><a name="Roles" id="Roles">3. Roles</a></h2>
|
|
|
|
<p>To execute the activities of the model, five roles are defined. In the
|
|
realization of this model, some roles may be combined and performed by a
|
|
single organization or further distributed among several organizations.</p>
|
|
<ul>
|
|
<li><em>Buyer</em> requires conformance to the Specification.</li>
|
|
<li><em>Seller</em> builds the product with the intent of meeting the
|
|
conformance requirement of the purchaser. Products that undergo testing
|
|
are called Implementation Under Test (IUT).</li>
|
|
<li><em>Test Laboratory (TL)</em> performs the operational testing of the
|
|
IUT .</li>
|
|
<li><em>Certificate Issuer (CI)</em>, issues a Certificate of Conformance
|
|
for IUTs that have successfully completed the testing process.</li>
|
|
<li><em>Control Board (CB)</em>, resolves dispute and answers queries on
|
|
behalf of the CI.</li>
|
|
</ul>
|
|
|
|
<h3>Buyer</h3>
|
|
|
|
<p>The Buyer requires that a product be tested for conformance. The buyer
|
|
uses the results of the testing to verify that a seller provides a product
|
|
that conforms to the specification and meet procurement requirements. In
|
|
general, the buyer is the impetus for sellers to undergo conformance testing.
|
|
Specifically, if buyers don't demand that a product be tested and show
|
|
evidence of that testing, it is most likely that sellers will not undertake
|
|
having their products tested.</p>
|
|
|
|
<h3>Seller</h3>
|
|
|
|
<p>The Seller or developer uses the conformance tests and undergoes testing
|
|
to demonstrate that the product adheres to the specification and thus, meets
|
|
established conformance requirements. Additionally, developers may use the
|
|
tests to debug their products prior to market</p>
|
|
|
|
<h3>Test Laboratory</h3>
|
|
|
|
<p>The Test Laboratory (TL) conducts the conformance testing using the
|
|
prescribed test method. The testing is performed on the seller/developer's
|
|
product. A TL can be an organization or individual. A TL can be accredited
|
|
from a formal accreditation organization such as NIST's National Voluntary
|
|
Laboratory Accreditation Program (NVLAP) or recognized by the buyer, seller,
|
|
and certificate issuer, as qualified to perform the testing.</p>
|
|
|
|
<h3>Certificate Issuer</h3>
|
|
|
|
<p>The Certificate Issuer (CI) is responsible for issuing certificates for
|
|
conforming products. The decision to issue a certificate is based on the
|
|
testing results and established criteria for issuing certificates</p>
|
|
|
|
<h3>Control Board</h3>
|
|
|
|
<p>The Control Board (CB) is an impartial body of experts who function on
|
|
behalf of the CI. The CB is responsible for resolving queries and disputes
|
|
related to the testing process.</p>
|
|
|
|
<h2><a name="Activities" id="Activities">4. Activities</a></h2>
|
|
|
|
<p>The activities comprising the model can be categorized into one of four
|
|
areas:</p>
|
|
<ul>
|
|
<li><em>Recognition</em> of competent testing laboratories,</li>
|
|
<li><em>Testing</em>with an approved test method,</li>
|
|
<li><em>Testing process</em>,</li>
|
|
<li><em>Resolution</em> of queries and disputes.</li>
|
|
</ul>
|
|
|
|
<h3>Recognition of Competent Testing Laboratory</h3>
|
|
|
|
<p>A Testing Laboratory (TL) is an entity that provides services to measure,
|
|
examine, test, or otherwise assess conformance of an implementation with its
|
|
specification. Within the buyer/seller model, a TL can be either a
|
|
first-party, (the seller performs the testing), second-party (the buyer
|
|
performs the testing), or third-party (an independent organization performs
|
|
the testing) testing organization. All three types of testing are used in the
|
|
software industry. Often there will be multiple TLs for a conformance testing
|
|
and certification program</p>
|
|
|
|
<p>The Certificate Issuer (CI) as well as Sellers and other interested
|
|
parties, must have confidence in the competency of the TL. Competence is
|
|
based on three concepts,</p>
|
|
<ol>
|
|
<li>the ability to apply the test method correctly,</li>
|
|
<li>the ability to repeat a given test and generate the same results,</li>
|
|
<li>the ability to operate the TL in a manner that maintains objectivity
|
|
and neutrality (obviously, first and second party testing organizations
|
|
are not neutral).</li>
|
|
</ol>
|
|
|
|
<p>The CI defines competence through requirements and criteria. The CI can
|
|
then apply the criteria to a TL, determine its level of competency and, if
|
|
appropriate, recognize the TL as competent to perform testing. This practical
|
|
approach to identifying and recognizing qualified testing organizations is
|
|
appropriate when costs, time and efforts do not warrant seeking accreditation
|
|
from a formal accreditation organization.</p>
|
|
|
|
<p>If a more formal and rigorous approach is appropriate, there exists many
|
|
accreditation bodies exist that are capable of performing this function. The
|
|
National Voluntary Laboratory Accreditation Program (NVLAP) is a NIST
|
|
organization that accredits testing organizations based on the requirements
|
|
of ISO Guide 17025 and additional subject-matter requirements.</p>
|
|
|
|
<p>The purpose of the recognition criteria or accreditation is to assure that
|
|
TLs are capable and competent to meet the needs of the testing and
|
|
certification program. The basic activities to make this determination
|
|
include:</p>
|
|
<ul>
|
|
<li>proficiency testing - demonstration of a TL's competency to
|
|
successfully perform the conformance testing using the test method,</li>
|
|
<li>on-site assessment - visit by a technical expert to determine
|
|
compliance with the recognition criteria and ensure the TL is a legally
|
|
identifiable organization with staff and resource to discharge their
|
|
duties,</li>
|
|
<li>quality assurance - documentation and practices to ensure technical
|
|
integrity of testing and analyses and adherence to quality practices
|
|
appropriate to the testing and certification program.</li>
|
|
</ul>
|
|
|
|
<p>Additional attributes required of a third-party TL include that it:</p>
|
|
<ul>
|
|
<li>ensure that its personnel are free from any commercial, financial and
|
|
other pressures which might adversely affect the quality of their
|
|
work,</li>
|
|
<li>ensure that the protection of sellers' confidential information and
|
|
proprietary rights are protected,</li>
|
|
<li>ensure that sellers are served with impartiality and integrity,</li>
|
|
<li>maintain a functional record keeping system for each seller testing
|
|
process,</li>
|
|
<li>have the adequate facilities and equipment to fulfill the requirements
|
|
of a TL.</li>
|
|
</ul>
|
|
|
|
<h3>Testing with an Approved Test Method</h3>
|
|
|
|
<p>For a Certificate of Conformance to be meaningful, all implementations
|
|
must be tested in the same manner. Testing reflects the essence of technical
|
|
requirements of specifications and measures whether a product faithfully
|
|
implements the specification. A <em>test method</em> is a defined technical
|
|
procedure for performing a test. A test is the technical operation that
|
|
consists of the determination of one or more characteristics of a given
|
|
product, process or service according to a specified procedure. A test suite
|
|
is the collection of tests. Critical to the success of any conformance
|
|
testing and certification program is an appropriate and adequate test
|
|
method.</p>
|
|
|
|
<p>An adequate test method is one that provides test results that give enough
|
|
information for the CI to be satisfied that conformance can be measured. An
|
|
adequate test method meets the requirement of rigor. An appropriate test
|
|
method is one that, while adequate, does not place undue requirements on the
|
|
IUT and is cost justifiable. If the test method is too expensive to employ
|
|
then it will not be used. The definition of adequate and appropriate is left
|
|
to the CI to determine.</p>
|
|
|
|
<h3>Testing Process</h3>
|
|
|
|
<p>The Testing Process is described in a conformance testing and
|
|
certification policy and procedures document. The document identifies the
|
|
administrative as well as testing processes.</p>
|
|
|
|
<p>The testing process initiates with a seller (or anyone desiring to be
|
|
tested) contracting with the TL to have an implementation tested for
|
|
conformance. The seller and TL negotiate the scope of testing, the cost of
|
|
testing, and the timeliness of testing. For a given seller, the TL must not
|
|
be in a position to benefit nor suffer (beyond the testing fees) from the
|
|
resulting pass or failure of the implementation under test (IUT).</p>
|
|
|
|
<p>Using the approved Test Method, the TL tests the IUT for conformance and
|
|
reports the results in a <em>Test Report</em>. The TL forwards the Test
|
|
Report and an indication of pass/fail to the CI. If the IUT successfully
|
|
completes all the tests and meets the criteria for issuing certificates, the
|
|
CI issues a <em>Certificate of Conformance</em>to the seller. Typically, the
|
|
CI maintains and makes available to the public, a listing of products that
|
|
have received certificates of conformance.</p>
|
|
|
|
<h3>Resolution of Queries and Disputes</h3>
|
|
|
|
<p>Queries and disputes involving the test method, procedures, test results,
|
|
and program administration are directed to the Control Board (CB). The
|
|
purpose of the CB is to resolve these issues and communicate the decision to
|
|
all parties involved. The CB acts on behalf of the CI. A query or dispute can
|
|
be initiated by a seller, TL or entity (e.g., developer) at any point in the
|
|
testing process. Queries and disputes should contain a statement of the
|
|
problem, rationale for dispute, and desired resolution. All matters to be
|
|
resolved by the CB should be determined by consensus or as determined by
|
|
documented CB policy and procedures.</p>
|
|
|
|
<p>Additional activities that may be under the auspices of the CB include:</p>
|
|
<ul>
|
|
<li>maintain liaison with appropriate standards bodies and test
|
|
laboratories,</li>
|
|
<li>participate in the assessment of TL's seeking recognition status,</li>
|
|
<li>recommend changes to new versions of the test method or test laboratory
|
|
recognition criteria,</li>
|
|
<li>serve as technical advisor to the CI and TLs,</li>
|
|
<li>maintain the test suite,</li>
|
|
<li>control changes to the conformance testing process.</li>
|
|
</ul>
|
|
|
|
<h2><a name="Products" id="Products">5. Products</a></h2>
|
|
|
|
<p>The following products are used in the model:</p>
|
|
<ul>
|
|
<li>Certification Program Policy,</li>
|
|
<li>Testing Laboroatory Criteria,</li>
|
|
<li>Specification,</li>
|
|
<li>Implementation Under Test (IUT)</li>
|
|
<li>Test Method,</li>
|
|
<li>Test Report,</li>
|
|
<li>Certificate of Conformance</li>
|
|
</ul>
|
|
|
|
<h3>Certification Program Policy</h3>
|
|
|
|
<p>The Certification Program Policy (CPP) defines the certification system.
|
|
ISO/IEC Guide 2 defines a certification system as a system having its own
|
|
rules of procedure and management for carrying out conformity certifications.
|
|
The CPP addresses the following:</p>
|
|
<ul>
|
|
<li>responsibilities of the CI,</li>
|
|
<li>responsibilities of the TLs,</li>
|
|
<li>responsibilities of the seller (the IUT owner),</li>
|
|
<li>policy and procedures for test laboratory recognition,</li>
|
|
<li>policy and procedures for testing process,</li>
|
|
<li>policy and procedures for handling queies and disputes,</li>
|
|
<li>complete deinition of the certificate of conformance.</li>
|
|
</ul>
|
|
|
|
<h3>Test Laboratory Criteria</h3>
|
|
|
|
<p>Testing Laboratory Criteria serves three purposes. The first purpose is to
|
|
define the competence and quality-related requirements that a testing
|
|
laboratory must possess to be designated as a recognized testing laboratory.
|
|
The second purpose is to describe the manner in which the laboratory will be
|
|
assessed against the requirements. The third purpose is to show those who
|
|
want to use the testing laboratory (e.g., sellers), or those who want to
|
|
accept the conformance certificate as evidence of conformance (e.g., buyers)
|
|
the rigor under which the testing laboratory operates</p>
|
|
|
|
<h3>Specification</h3>
|
|
|
|
<p>First and foremost to conformance testing and certification is the
|
|
specification. This paper delineates "standards-based" software specification
|
|
from other types of specification. This is because not all specifications can
|
|
be objectively tested for conformance. We recognize that not all
|
|
"standards-based" specifications can be objectively tested. However objective
|
|
measurement (not necessarily conformance testing per se) is usually a goal in
|
|
these specification development efforts</p>
|
|
|
|
<p>If the specification can not be objectively tested, then a alternate
|
|
approach to conformance testing should be used to measure whether a produce
|
|
faithfully implements the specification. This is because an accepted test
|
|
method cannot be developed, thus repeatability and reproducibility cannot be
|
|
ensured.</p>
|
|
|
|
<h3>Implementation Under Test</h3>
|
|
|
|
<p>The implementation under test (IUT) is the object that is being tested for
|
|
conformance. For software specifications it is the software that has
|
|
implemented the specification. For any certification program, the scope of
|
|
the IUT must be defined and delineated from the rest of the supporting
|
|
software and hardware of the total system (referred to as the system under
|
|
test). In many current certification programs the hardware that is used by
|
|
the software must also be defined. The software and supporting hardware
|
|
constitute the IUT and are listed in both the test report and certificate of
|
|
conformance.</p>
|
|
|
|
<h3>Test Method</h3>
|
|
|
|
<p>The test method must be adequate and appropriate within the conformance
|
|
testing and certification program in which it is used. Beyond these
|
|
properties, test methods (and thus the tests) should be objective, have
|
|
adequate coverage, and correctly implement the specification. In trying to
|
|
meet these requirements, those using and applying the test method should not
|
|
make the common mistake of allowing the test method to become the
|
|
specification. This means that sellers (builders of IUTs) will build the IUT
|
|
to pass the conformance tests, rather than building to the specification.</p>
|
|
|
|
<p>An objective test method allows for test results to be reproducable by the
|
|
same testing laboratory and to be repeatable by a different laboratory.
|
|
Initially some test methods do not quite achieve a sufficient level of
|
|
objectivity. However objectivity should be something that is always strived
|
|
for in the development and ongoing refinement of a test method.</p>
|
|
|
|
<h3>Test Report</h3>
|
|
|
|
<p>A test report contains the results of the testing effort, along with any
|
|
additional information required by the CI. The test report should provide
|
|
enough information that, if necessary, the testing effort could be
|
|
duplicated. The testing report should contain:</p>
|
|
<ul>
|
|
<li>a complete description of the IUT,</li>
|
|
<li>the name of the testing laboratory,</li>
|
|
<li>the signature of a testing laboratory official,</li>
|
|
<li>the date that the testing was completed,</li>
|
|
<li>the name and version number of the test method (and test suite),</li>
|
|
<li>the results of the test method,</li>
|
|
<li>an unambiguous statement indicateing pass or fail.</li>
|
|
</ul>
|
|
|
|
<h3>Certificate of Conformance</h3>
|
|
|
|
<p>The certificate of conformance is typically a summation of the test
|
|
report. Since it is often used in the procurement process, it includes
|
|
information most pertinent between the busyer and the seller.</p>
|
|
|
|
<p>The certificate includes statements made by the CI. These statements
|
|
articulate what the CI is asserting as being conformant. Typically these
|
|
statements indicate that "this IUT was tested in this environment, on this
|
|
day, using this test method: the test results produced were consistent with
|
|
expected test results". The certificate also includes the signature of a CI
|
|
official.</p>
|
|
|
|
<h2><a name="Examples" id="Examples">6. Examples</a></h2>
|
|
|
|
<h3>ATA Computer Graphics Metafile (CGM) Conformance Testing Program</h3>
|
|
|
|
<p>The <a href="http://www.air-transport.org">Air Transport Association
|
|
(ATA)</a> CGM Program was originally established and operated by NIST to
|
|
support the ATA 2100 Specification, Graphics Exchange (a.k.a. ATA CGM
|
|
profile). The testing program is a critical component of the ATA's program to
|
|
represent maintenance manuals in digital form and move to completely on-line
|
|
maintenance manuals. Testing is done to ensure that the fidelity and quality
|
|
of the digital information is sufficient to satisfy the airline companies'
|
|
safety and quality concerns. The program is a means whereby a seller of a CGM
|
|
implementation can formally demonstrate conformance to the ATA CGM
|
|
profile.</p>
|
|
|
|
<p>NIST is currently working with the ATA in its assumption of the testing
|
|
program. The ATA CGM Conformance Testing Program will consist of recognized
|
|
Testing Laboratories to conduct the testing and a Control Board to handle
|
|
disputes and serve as an advisor to the ATA. The ATA will act as the sponsor
|
|
and administrator of the program. The ATA or an ATA designate will issue
|
|
certificates of conformance. The roles, activities, and products as described
|
|
in the generic model apply here with little modification. The Control Board
|
|
takes on the additional activity of assessing the testing laboratories
|
|
according to pre-established criteria. Additionally, the ATA Technical
|
|
Information Communication Committee's Graphics Working Group serves as a
|
|
technical advisor to both the ATA and the control board.</p>
|
|
|
|
<p>The test method consists of a NIST developed test suite and test
|
|
procedures. The test method has been accepted and used by the community. It
|
|
is publicly available along with other program documents.</p>
|
|
|
|
<h3>IEEE POSIX Validation Service</h3>
|
|
|
|
<p>The IEEE established a validation service for the POSIX (Portable
|
|
Operating System Interface). The <a
|
|
href="http://standards.ieee.org/regauth/posix/index.html">IEEE Validation
|
|
Service</a> uses accredited POSIX testing laboratories, issue certification
|
|
of validated test results, and maintains a register of accredited
|
|
laboratories and successfully tested products. The laboratories are
|
|
accredited by the NVLAP under its POSIX program</p>
|
|
|
|
<p>The requirement for testing is buyer driven. Initially, federal agencies
|
|
in their requests for procurement (RFP) of POSIX systems required
|
|
certificates of validation prior to purchase. However, the benefits of POSIX
|
|
testing and its acceptance in the industry has resulted in sellers requesting
|
|
to be tested as a matter of course, rather than a procurement requirement.</p>
|
|
|
|
<p>The test suite was produced in a joint effort between NIST and several
|
|
computer vendors. The original testing policy and procedures produced by NIST
|
|
have been adopted by the IEEE.</p>
|
|
|
|
<p></p>
|
|
<hr />
|
|
|
|
<h2><a name="Conclusion" id="Conclusion">7. Conclusion</a></h2>
|
|
|
|
<p>This model describing the conformance testing and certification process
|
|
has been used many times over in certification programs for standards-based
|
|
software specifications. The examples above illustrate just a few of these
|
|
programs. It will continue to be used as a communication mechanism between
|
|
buyers and sellers.</p>
|
|
|
|
<p>Test method developers must continue to develop test methods that have
|
|
adequate coverage with regard to the specification; are well defined in terms
|
|
measurement (i.e., what does each test case prove); and be adequate and
|
|
appropriate as defined by the Certificate Issuer.</p>
|
|
|
|
<p>As the industry moves toward component based software, the challenge will
|
|
be to develop test methods and associated certification programs that can
|
|
provide meaningful measurement in this environment</p>
|
|
|
|
<h2><a name="References" id="References">References</a></h2>
|
|
<dl>
|
|
<dd>Breitenberg, Maureen, The ABC's of the U.S. Conformity Assessment
|
|
System, NISTIR 6014, April 1997.</dd>
|
|
<dd>Breitenberg, Maureen, The U.S. Certification System from a Government
|
|
Perspective, NISTIR 6077, October, 1997.</dd>
|
|
<dd>Carnahan, Lisa, Developing Federal Standards and Accreditations for
|
|
Data Protection Products, Proceeding of SPIE Conference, October,
|
|
1995.</dd>
|
|
<dd>Dashiell, William H., L. Arnold Johnson and Lynne S. Rosenthal,
|
|
Overview of Model for United States Geological Survey Recognition of
|
|
Spatial Data Transfer Standard Certification System, NIST IR 6124, May
|
|
1998.</dd>
|
|
<dd>Horlick Jeffrey, and Lisa Carnahan, Cryptographic Module Testing,
|
|
Handbook 150-17, April, 1995.</dd>
|
|
<dd>ISO/IEC Guide 2: 1996, Standardization and Related Activities:
|
|
General Vocabulary.</dd>
|
|
<dd>ISO/IEC Guide 17025: 1999, General Requirements for the Competence of
|
|
Calibration and Testing Laboratories.</dd>
|
|
<dd>NIST, Derived Test Requirements for FIPS 140-1, Security Requirements
|
|
for Cryptographic Modules, March, 1995.</dd>
|
|
<dd>NIST, Procedures and Requirements, NIST Handbook 150, March 1994.</dd>
|
|
</dl>
|
|
</body>
|
|
</html>
|