From rpl@ditc.npl.co.uk Mon Jan 23 15:24:33 1995 Received: from ditc.npl.co.uk by dkuug.dk with SMTP id AA16609 (5.65c8/IDA-1.4.4j for ); Mon, 23 Jan 1995 16:24:11 +0100 Message-Id: <6625.199501231524@ditc.npl.co.uk> Received: from elgar by ditc.npl.co.uk; Mon, 23 Jan 1995 15:24:05 GMT X-Sender: rpl@vivaldi Mime-Version: 1.0 Content-Type: text/plain; charset="us-ascii" Date: Mon, 23 Jan 1995 15:24:33 +0000 To: sc22wg15@dkuug.dk From: rpl@ditc.npl.co.uk (Richard Lampard) Subject: Re: App Conformance X-Mailer: X-Charset: ASCII X-Char-Esc: 29 >Richard, > >I saw your email to the POSIX panel. Thanks for the response. > >Any information on application conformance would be potentially useful, >perhaps a break down of the work the NPL is doing in the area ? > >We have security programs, and MHS conformance programs also running that >may stand to benefit, as well as the POSIX work I am interested in. > >The NHS has just procured a managed national MHS network. This has several >possible access methods, including dial up. Dial up may be done via PSTN or >ISDN. As far as GPs are concerned dial up access is likely to be the only >method of access. So, there is an obvious security link there with the work >you mentioned in your email. > >Basically, yes we would be interested ! > >I think I need a little more information about NPL work/approach and then >perhaps we can do something. > >Yours, > >Rob Smith >NHS Information Management Centre > Please find below a bit more information on our research, and the results we've so far had. We are actively looking for avenues to continue our research at the moment. Richard. A new approach to testing IT security Bronia Szczygiel Data Security Group December 1994 1. INTRODUCTION The DSG has proposed the use of the strict conformance testing methodology (SCT), which combines an extended form of conformance testing with a developers quality system procedures, as a means of increasing the cost-effectiveness and objectivity of security development and evaluation [1]. In order to test the proposed methodology, a trial was conducted which involved the development of a strict conformance test suite for a real product. This paper outlines the results of this trial and the further development of the SCT methodology. The SCT methodology has several distinct elements, which combine to address the requirements of ITSEC evaluation, to gain assurance in the security of a product or system. Those elements are: a) An abstract security target for the standard implemented. b) An extended set of conformance tests for the standard implemented. c) Requirements on a developers quality system procedures. Many experts in the field believe that it is impossible to develop conformance test suites for security standards or that any test suites developed would not identify vulnerabilities in implementations and therefore would be of little value. The DSG aimed to determine the feasibility and value of the SCT approach through this study. Progress in each of the above areas is summarised in the following sections. THE SECURITY TARGET The Common Criteria suggest that standardised Protection Profiles (PPs) should be developed to address common functionality. The SCT approach recognises the value of PPs and the recent trial included information based on the standard which could be incorporated into a PP. Security targets were developed for the standards used in the implementation. These security targets can be profiled for the implementation to give a security target which addresses some of the ITSEC requirements. We envisage a building block system in which security targets for standards could be developed, profiled and combined to produce a product or system security target, in the same way that the base standards are combined. Similarly the test suites can be combined to produce a product or system test suite. Since the standardised security target has undergone expert scrutiny it should meet ITSEC requirements. Recognition of this fact within the ITSEC scheme would lead to a reduction in the cost of evaluation for an implementation which uses that security target. A comparison was made of the security target developed for ISO/IEC 9798-4 and that of the implementation. This highlighted the omission of details from the implementation security target and showed the value of having guidance of this sort available to developers. THE STRICT CONFORMANCE TEST SUITES The product on which the trial is based incorporates an implementation of two security standards, namely ISO/IEC 9798-4 and the Secure Hash Algorithm (SHA). The DSG has developed strict conformance test suites for both standards using the semi-formal test case notation TTCN (defined in [2]) for the former and the formal language Z for the latter. Hence, the DSG has demonstrated that the development of such test suites is possible. The value of those test suites had then to be demonstrated. The original intention of the exercise was to allow the product to undergo development and testing, including ITSEC evaluation to E4 level, whilst the DSG conducted a parallel strict conformance test campaign. There were two objectives: To determine the value of SCT in detecting errors which would increase the costs to the developer in remedial work and further evaluation effort. To compare the results, time scales and costs of strict conformance testing of the product and formal evaluation. The progress made towards achieving each objective is summarised here. THE VALUE OF SCT IN DEVELOPMENT The developer allowed an assessment of the product to be conducted early in the design stage in an attempt to identify errors. The results of that exercise are summarised here. The SCT test suite incorporates a set of test purposes developed according to the principles laid down in ISO/IEC 9646, the OSI conformance testing standard [2]. The preliminary design of the product was analysed by the DSG to assess whether the test purposes would have been satisfied. The results indicated several non-conformances in the design at this early stage, one of which proved to be serious. These problems, had they not been identified and corrected, would have resulted in an increase in the vulnerability of the product. The design was consequently modified by the developer and an even more serious error, caused by the modification, was identified by the DSG when the SCT exercise was repeated. The developer corrected this error, so that an ITSEC E4 certification is possible. This exercise proves the case that strict conformance testing can identify vulnerabilities in implementations of security standards. It also shows that analysing a design in the early stages of development against an abstract test suite is a valuable exercise. This step has now been incorporated into the suggested methodology. Running an executable version of the test suite against a real implementation would have established whether the potential problems identified at the design stage had appeared in the product. Hence the first objective was fully achieved. A COMPARISON OF SCT AND EVALUATION The second objective could not be achieved because, owing to the length of the product development time scales (which were, of course, outside the control of the DSG), the DSG was unable to complete the comparison of results within the time remaining for that part of the project. Although the direct comparison between SCT and evaluation could not be completed, the results obtained from the project have shown the value of the approach in reducing development and evaluation costs and time scales. The above exercise demonstrated that the SCT methodology can identify problems far earlier than the evaluation process, since pre-evaluation does not involve any assessment of the design. In addition, the evaluator could not guarantee that the problem found through the SCT exercise would have been found with evaluation. In contrast, the SCT methodology is guaranteed to identify such problems. The developer has estimated that the identification of the problem in the early stages of the design process, rather than towards the end of the testing programme, has resulted in a saving of up to 10% of the cost of development. This includes the costs associated with re-submitting the product to the CLEF after rectifying the error. In addition, the development and evaluation time scales have been shortened since, had the error been discovered late in the evaluation process, correction and re-evaluation would have been a lengthier process. The use of a standardised security target could also contribute to a reduction in time and costs as indicated above. It is therefore clear that, had a direct comparison been possible, it would have demonstrated a reduction in costs and time scales through using the SCT methodology. QUALITY ASSURANCE Although, for the trial, the developers quality system was not explored, the presence of a set of procedures relevant to the development of secure products and systems is an essential part of the SCT methodology. DSG has completed a study on the role a suitable Quality Assurance system could play in reducing the costs of development and evaluation against ITSEC. This gave rise to two issues, ITSEC requirements via Quality Assurance and the mechanism for ensuring conformity. ITSEC REQUIREMENTS VIA QUALITY ASSURANCE The purpose behind ITSEC evaluation is to gain assurance for security products. A mechanism which is based upon the supplier's process will generally reduce costs, since the means of ensuring the process is correct can be applied to all the suppliers products. Savings can be made if appropriate QA procedures are available. Most of these are in the area of validation The procedures would reflect the requirements at the level for which they are designed. The cost of producing some of these procedures could be quite high but once developed they can be used repeatedly. The extent to which some property of the Target Of Evaluation (TOE) can be assured by means of a Quality System depends upon the property in question and how well the QA procedure is written. As the evaluation level goes up the required degree of formality increases, therefore objectivity is greater and QA procedures can more easily provide a high level of assurance. QA procedures are already essential. Some aspects of ITSEC can only be met by means of QA procedures, such as E2.23. One limiting factor to any savings is the requirement to `search for errors'. Such a process can not be adequately specified by a QA procedure, since it is subjective. MECHANISMS FOR ENSURING CONFORMITY For those ITSEC requirements which could be satisfied by a QA procedure, the following means of conformity could be envisaged: Self-certification. A supplier follows its own QA procedure which is seen to satisfy a specific requirement in ITSEC. On evaluation, the supplier would merely state that requirement En:m is satisfied, since it is the requirement of its own procedure. This option is the simplest to administer. Independent audit. An independent audit of the suppliers QA system is undertaken. The audit checks any claims the company cares to make. This option would require a special form of audit. ISO 9001 audit. An ISO 9001 audit checks, by sampling, that a company follows its QA procedures. Such an audit does not check that a specific procedure satisfies a specific requirement of ITSEC. Registration after testing. Regular ITSEC evaluation is undertaken for the first product. If the Test laboratory establishes that an ITSEC requirement is met as a consequence of a QA procedure, the company registers that procedure so that on any subsequent evaluations this specific requirement would not be re-tested. The register could be held by Test Laboratories or by a certification body. This option seems to be the best, since it relies upon the current scheme, but provides for an element of cost-reduction. This process is similar to that of NAMAS accreditation. It might be claimed to be unwise to rely upon the supplier's QA system, since there is evidence that procedures are not necessarily followed. The risks can be reduced by requiring documentary evidence that a procedure has been followed. The evidence of compliance is therefore similar to that of an evaluation report. Outline procedures could be produced to ensure conformity with some of the specific ITSEC requirements. These procedures would be similar to those already being devised by the evaluators and could be accredited. The revised model for evaluation would place much greater reliance upon procedures developed by the supplier to ensure that security products met most of the specific requirements of ITSEC. The changes would therefore be: The supplier must develop procedures to validate their products against specific requirements of ITSEC. The evaluators must assess the supplier's procedures when they are offered for evaluation to reduce subsequent evaluation costs. The validators must undertake minimal checks when a procedure is being used by a supplier to satisfy a specific ITSEC requirement, given that this procedure has already been evaluated. A registration system must be established if there is a requirement for suppliers to have their products evaluated by more than one evaluator using this revised model. CONCLUSION The DSG approach is to develop standardised conformance test suites for security standards. These can then be used in both the development and evaluation process highlighting vulnerabilities early in the design stage, leading to a reduction in the cost associated with designing functional tests, demonstrating adequate coverage and allowing claims of conformance to standards to be evaluated. The DSG work has demonstrated that it is possible to develop such test suites and that they are of value in reducing development and evaluation time scales and costs. The presence of a developers quality system can help to reduce the costs of subsequent evaluations if certain procedures are accredited and registered as part of the first evaluation. Significant savings could be made by placing some reliance upon the supplier's quality assurance procedures, given that these procedures are well written. For an initial assessment, the evaluation would need to take into account the suppliers QA procedures as well as the TOE. This implies a greater cost for the evaluation, say perhaps 20% higher. Subsequently, if the supplier has been successful in demonstrating that specific QA procedures ensure that some ITSEC requirements are being met, then the evaluation should be significantly cheaper. The strict conformance testing methodology combines the development of a standardised security target and strict conformance test suite with the use of the developers accredited quality procedures to give a level of assurance in the security of the product or system. This combination can cover approximately 60 - 80% of the evaluator actions required for ITSEC E2 level evaluation. Further work should be directed at producing SCT test suites for commonly used security standards. We hope to set up collaborative projects with the DSG developing the abstract test suites for these standards and partners deriving executable versions for use on real implementations. REFERENCES [1] Lampard R.P. Strict conformance testing (or: "How to get through an evaluation with less pain"). DSG/D/306. [2] International Standards Organisation. Information Technology - Open Systems Interconnection - Conformance testing methodology and framework. ISO/IEC 9646.