JTC1/SC22/WG15 - RCT N056




    Date:			  October 26, 1993

    Supersedes: 		  N/A

    Title/Topic:		  Guide to Conformance Testing
				  Methodology

    Source:			  United Kingdom

    _____________________________________________________________________
    Action Requested(check one):

    _____Review/Comment by:

    _____Respond by:

    _____For discussion/consideration @ RGCT meeting

    _____For distribution

    _____For information

    _____Other(see details)


    Details: This is a replacement for WG15 N347

    JTC1/SC	22/WG15 RCT N056
    _______________________________________________________________________
     Title (subject to be covered and type of standard, e.g. terminology,
     method of test, performance requirements, etc.):

      Guide to Conformance Testing Methodology
    _______________________________________________________________________
      Scope (and field of application):

     Those areas of standardisation covered by the ISO/IEC JTC1/SC22 WG15
     work item and other areas of standardisation within JTC1 that define
     Application Program Interfaces (APIs).  Platform External Interfaces
     (PEIs) are outside the scope of this work item.
    _______________________________________________________________________
     Purpose and justification:

      GENERAL

     "POSIX" testing methodology is less mature than its OSI equivalent.
     Current IEEE work on conformance testing has concentrated on the
     issue of test suite derivation from base standards (test assertions,
     the concept of test suite conformance).  It is necessary to identify
     an overall model for OSE Conformance Testing which complements CD
     13210 (IEEE 1003.3-1991), and which is as far as possible compatible
     with existing OSE work. The conformance testing of OSE profiles
     containing both OSI and non-OSI components must be handled in as
     coherant and consistent a way as possible, in order that users have
     confidence in procuring against such composite ISPs. The production
     of an overall reference model will assist in this process.

      SPECIFIC AREAS FOR STANDARDISATION

      General Terminology
     One of the barriers to harmonisation of OSE Conformance Testing is
     the lack of a common vocabulary.  The first task should be to
     identify common terminology in as many areas as possible and to
     identify unavoidable areas of difference.

      The Conformance Assessment Process
     CD 13210 (IEEE 1003.3-1991) contains no overall model of the
     Conformance Assessment Process.  This is a quality hazard. Without
     an overall model it is not obvious that test results are repeatable
     or reproduceable. Therefore this TR will identify overall models of
     the Conformance Assessment Process.

      Validation and Maintenance of Tests.
     CD 13210 (IEEE 1003.3-1991) lays down conformance requirements for
     tests but does not say how conformance can be established and
     maintained. This is also a quality hazard. Therefore this TR will
     identify methods for validation and maintenance of tests.

    _______________________________________________________________________
      Programme of work:

     A multi-part Technical Report (TR) consisting of 3 parts, which will
     be Terminology, Conformance Assessment Model and Validation and
     Maintenance of Tests.  The approach to be taken for each of the
     three parts of this TR is as follows:

      Part 1  Terminology

     A detailed comparison of terms used in CD 13210 and the latest draft
     of IEEE 2003, IEEE 2003.1-1992, IEEE 2003.2 (Draft),  ISO 9646-1 and
     the most recent draft of TR 10000-3 will form the basis of this
     work, to identify areas of overlap and disagreement in the use of
     conformity assessment terminology and thereby to create a TR, which
     as far as possible explains any unnecessary difference between terms
     used in the conformity assessment of APIs and terms used in the
     conformity assessment of OSI.

      Part 2 Conformance Assessment Model

     Using existing practice as a starting point, develop an API
     Conformance Assessment Model which takes into account ISO 9646-5 and
     CD 13210 and identify other areas of concern and overlap.

      Part 3 Validation and Maintenance of Tests

     This multi-part TR will take as input work from OIW and research
     projects investigating the automatic production and   validation of
     tests.

    _______________________________________________________________________
     Relevant documents to be considered:

     The ISO 9646 series of standards, EN 45001 to EN 45014, CD 13210
     (IEEE 1003.3-1991), IEEE 2003.1-1992, and the TR 10000 series.

    _______________________________________________________________________
      Co-operation and liaison:
      ISO/IEC JTC1 SGFS
      ISO/IEC JTC1 SWG_CA
      ISO/IEC JTC1/SC22/WG15
      ISO/IEC JTC1/SC21/WG1
      IEEE P2003
      IEEE 1003.0
      EWOS
      OIW
    _______________________________________________________________________
     Preparatory work offered with target dates:

     Working Draft Q2 1995
    _______________________________________________________________________
     Will the services of a maintenance agency or registration authority
     be required? - No

     Are there any known requirements for coding? - No

     Does the proposed standard concern known patented items? - No
    _______________________________________________________________________