Date: Thu, 7 Jul 1994 21:06:07 +0200 From: D.Cannon@exeter.ac.uk To: " (SC22/WG15 Posix)" Subject: (wg15-uk 522) (SC22WG15.363) Posix Apps conformance I have been actioned by the UK BSI IST/5/-/15 (POSIX) panel to forward the following paper to WG15 for its consideration. The paper expresses the concerns of a major IT user in the UK, whose expenditure on IT components per year runs to hundreds of millions of pounds. The English NHS is concerned that its investment in POSIX-conformant platforms may be undermined by a lack of compliant applications: In the UK POSIX applications testing has been available for the last two years - but the tests are not available through a NAMAS-accredited testing laboratory, and the problem here is one of insufficient demand. (Developers tend not to offer conformance to standards unless their customers have a requirement for it). The problem may be one of POSIX-user education: users should be moving towards the use of POSIX compliant applications. They should be pressuring their suppliers to show some commitment to conformance, but they need the hooks in the standards to hang their demands on, and something identifiable on the product's shrink-wrapper to tell them they're buying POSIX compliance. IST/5/-/15 is interested in debating any proposed mechanisms which might lead to the 'certification' of POSIX-worthy applications. David Cannon Convener, BSI IST/5/-/15 4-July-1994 ________________________________________________________________ The English National Health Service Requirement for Application Conformance Testing The open systems movement has been driven largely by market requirement for greater choice. However, there are a large number of political and technical problems that still need addressing. This short paper will outline some of those problems related to open applications, with particular reference to POSIX, from the English NHS point of view. When a standard is adopted it should be done primarily for sound business reasons. There needs to be a demonstrable benefit. The English NHS believes that there is a business case for adopting the POSIX standard (investment protection in hardware and operating systems, market choice, protection of staff expertise). These benefits can be realised because the majority of operating systems now conform to POSIX.1 and POSIX.2 (or will do so). The vendors of operating systems conform because they have been motivated by market pressure. First and third party testing has allowed operating system vendors to gain competitive advantage by demonstrating conformance and has given purchasers confidence to buy operating systems that conform. Unfortunately there is little evidence that application developers are writing to POSIX standards. Unless they do so the whole exercise will have been a waste of effort. Until application developers start to actively make use of POSIX standards, and tell customers about it, the primary goal in developing POSIX will no be reached. It is quite likely, if application developers do not start to use the standard and advertise that fact, that operating system vendors will cease to support it. The English NHS mandates POSIX in all relevant purchases above GBP 5000 (with appropriate derogations). This level of commitment requires support in training and guidance for purchasers, technical expert help, procurement help, and maintenance and development of POSIX standards requirements. Without applications to support POSIX there is little business reason to continue to mandate its presence, especially when the cost of mandating it is taken into consideration. Two further problems complicate the situation. (i) Potential purchasers of open applications can not be expected to have the expertise needed to accurately assess the conformance (or lack of it) in an application. (ii) As there is no application testing available, public bodies may derogate from the EC Standards Decision where applications are of concern. Application conformance testing is necessary in order to encourage purchasers to require POSIX conformant applications. Application conformance testing gives purchasers the ability to accurately assess the conformance claims of a vendor. Application testing may be necessary to persuade application developers to adopt the standard. The situation for conforming applications is the reverse of that for platforms, ie very little support for the standard. For platforms there is adequate first and third party testing available, and support for the standard is almost universal. The POSIX standard is being developed "to support application portability at the source level" (1). This support will not be achieved without application conformance testing. There is an urgent requirement to develop application test benches for POSIX before users of the standard start to seriously question the worth of continuing to use it. Robert Smith NHS Information Management Centre (1) Portable Operating System Interface (POSIX) - Part 1: System Application Program Interface (API) [C language] ISO 9945-1. Date: Thu, 7 Jul 94 13:23:43 BST From: knosof!derek@xopen.co.uk (Derek M Jones) To: wg15-uk@pyramid-technology.co.uk Subject: (wg15-uk 521) All, >From: D.Cannon@exeter.ac.uk >Subject: (wg15-uk 520) POSIX App Conformance Testing >Date: Wed, 6 Jul 1994 21:19:33 +0100 (BST) > > Two further problems complicate the situation. (i) Potential > purchasers of open applications can not be expected to have the > expertise needed to accurately assess the conformance (or lack > of it) in an application. (ii) As there is no application > testing available, public bodies may derogate from the EC > Standards Decision where applications are of concern. Applications testing has been available for the last two years. You can say that testing by a NAMAS accredited testing laboratory is not available. If there were sufficient demand we would become NAMAS accredited. > > > Application conformance testing is necessary in order to > encourage purchasers to require POSIX conformant applications. > Application conformance testing gives purchasers the ability to > accurately assess the conformance claims of a vendor. Very true. > Application testing may be necessary to persuade application > developers to adopt the standard. No it isn't. Developers conform to standards because their customers require it. > > The POSIX standard is being developed "to support application > portability at the source level" (1). This support will not be > achieved without application conformance testing. There is an > urgent requirement to develop application test benches for POSIX > before users of the standard start to seriously question the > worth of continuing to use it. > This paper implies that the problem is somebodies elses fault. In fact it is the NHS's fault (along with lots of other large corporate users). The NHS should create a public policy of moving towards the use of POSIX compliant applications. They should start to put pressure on their suppliers to show some commitment to eventual conformance. We have all sorts of conformance testing and certification procedures in place for a wide variety of standards. Many of them fail to make much headway. The reason is that users do not demand proof of conformance. I don't see what setting up an applications testing program will do for POSIX unless users start to ask pointed questions of their suppliers. derek ps. The above does not mean that I am not willing to support this proposal. Given the current status anything is better than nothing. pps. What has happened to my porposal to add "Rigorous Conformance" to POSIX? This concept would help to tie up a lot of loose ends that currently exist in testing applications. Date: Fri, 15 Jul 94 13:23:39 EDT From: stephe@mks.com (Stephen Walli) To: D.Cannon@exeter.ac.uk, sc22wg15@dkuug.dk Subject: (wg15-uk 527) (SC22WG15.368) Re: Posix Apps conformance Hello All, I am a little concerned by the opinions expressed in the letter from the British National Health Service, and statements of support for this point of view from colleagues in the POSIX community. I would like to offer a different view on the NHS's concerns, and why they and others in similar situations may wish to take a different approach to the problem. POSIX.1's scope states that it `` defines a standard operating system interface and environment to support application portability at the source-code level. It is intended to be used by both application developers and system implementors.'' The intended users of POSIX.1 are application developers and system implementors. Not purchasers of applications and platforms. As a customer of an application, I would like to know that: -- it runs on every platform upon which I wish to install it. -- I want to know that the application users do not need retraining when they use the application on a different platform. -- I want to know that data created by one instance of the application on one platform can be used by another instance of the application on another platform, possibly across a network, although there are many reasons why it may not *need* to be supported across platforms. I do not care one little bit about whether or not they used POSIX.1, ANSI C, Tcl/Tk, or any other *implementation* technique. I want it to work. Prior to POSIX.1 and ANSI C existing, many application vendors, of applications such as relational databases and system analysis tools, offered tools on multiple platforms. How they did this was of no concern to me as a user of these applications. One might not even care about the number of platforms an application runs on, if one lives in a networked environment where one can run the application on a single machine in a window on an X-terminal. (This becomes more an issue of what a vendor charges per licence, and whether they charge different amounts for licences on different machines.) As a large purchasing agent, such as I am presuming the NHS is, I would probably encourage my application vendors that I am purchasing particular platforms, based on particular criteria, and that I will put my application dollars where my platform dollars are going. I would point out that the particular criteria is platforms that support POSIX.1, ANSI C, window manager foobar, and so on, but this is information that is helpful, not necessary. It allows your supplier to understand what platforms you, their customer, feel are important, and where you will likely place your platform dollars in the future, so they themselves can begin making future plans. How the application vendors choose to implement their applications on those platforms, current and future, is their business. As an example, you certainly would NOT want to have a relational database vendor create a product that was SLOWER by necessity to make it conformant, by insisting they use certain conformant calls rather than using their own knowledge of how a pariticular architecture accomplishs certain things. Indeed, if they came to you beforehand, because you were a good customer, and said: ``We *will* deliver it on platform Y, but we can make it 15% faster if we don't have to ensure the application uses only standard Z's calls.'' What would you tell them? You don't care how. You care that they deliver it on platform Y. You want to be able to buy the application. Indeed, they would likely choose application speed over conformance without asking you because they need to remain competative against other vendors selling similar applications, and you are but one customer. Application vendors that use portability models based on portability standards will likely have an easier job porting their software. This is a business decision that they care about, especially if you are telling them you are purchasing applications that run on platforms because they support these particular standards. It would be a terrible waste of my tax dollar if my country's government started spending money supporting any sort of application conformance testing and certification program. Likewise, it would be a disservice to national application vendors to insist they develop completely conformant (certifiable) applications. This will cost them money to develop applications that are conformant rather than competative, and I would rather the software industry was healthy and paying taxes, and paying software developers like me (so that I may unfortunately pay taxes). It would be a special kind of disservice to POSIX to try to ``put teeth into it'' for application conformance. This will thrust it once more into the middle of a marketing battle, because once more it will be perceived as a threat rather than a tool by vendors. POSIX.1 is an excellent base for a portability model, used in conjunction with ANSI/ISO C. One can even write useful and interesting applications using just these two standards. One will likely want to expand their model to include specifications (standard or otherwise) for a GUI API, possibly a GUI look-and-feel, and a network IPC mechanism. This immediately puts the application at odds with just POSIX.1 and its amendments, with respect to applications conformance. Currently, I work for an application vendor. We know a lot about POSIX, as we sell POSIX.2 and implement it using POSIX.1. But, just because I have machines that implement these specifications does not mean that my entire application base uses them. There are places where we need to deliberately, by design, write non-portable code. My application architecture is such that I still know where those pieces of source code are. My application portability model also takes into account the tools and process I need to manage and build my application source across all the machines I need to support. POSIX.1 is a source code portability standard. It is to be used by application developers writing source code. It needs to be implemented by system implementors on their architectures to be useful to those application developers. It extends a useful portability model. I believe discussions of POSIX.1 applications conformance and testing and certification are nasty consumers of resources (people and time), that offer no genuine benefit. My two cents. Kindest regards, stephe Date: Sun, 17 Jul 94 19:56:12 BST From: knosof!derek@xopen.co.uk (Derek M Jones) To: wg15@pyramid-technology.co.uk Subject: (wg15-uk 528) All, >Date: Fri, 15 Jul 94 13:23:39 EDT >From: stephe@mks.com (Stephen Walli) > >POSIX.1's scope states that it >`` defines a standard operating system interface and environment >to support application portability at the source-code level. >It is intended to be used by both application developers and >system implementors.'' > >The intended users of POSIX.1 are application developers and system >implementors. Not purchasers of applications and platforms. So I guess 1003.3 have been wasting their time. Just because a standard is developed with an intended audience in mind does not preclude other interests using it. > >As a customer of an application, I would like to know that: >-- it runs on every platform upon which I wish to install it. ... But can you say today what platforms you will be interested in tomorrow. Software vendors showing some level of proof of conformance to POSIX are likely to have fewer porting problems than those who don't investigate the issues. Vendor proof of conformance provides a feel good factor to customers that there are no hidden purchasing problems (software not available on platform X) ahead. Also how are you the customer? If you are buying source code along with the binary then conformance is probably of great importance. > >Application vendors that use portability models based on portability >standards will likely have an easier job porting their software. >This is a business decision that they care about, >especially if you are telling them you are purchasing applications >that run on platforms because they support these particular standards. > Yes. I think it is important that we separate out the internal software engineering portability issues (which if done right will save a compnay lots of money) from the marketing issues "Our software conforms to Open Systems standards". >It would be a terrible waste of my tax dollar if my country's >government started spending money supporting any sort of application >conformance testing and certification program. It would be a bigger waste of your tax dollar if they bought applications that did not conform and later had to pay through the nose to change vendors or have them ported. Given that NIST is already up an running the incremental cost of adding applications testing is small. I have yet to hear of the US Government directly funding develoment of test suites. >Likewise, >it would be a disservice to national application vendors to insist they >develop completely conformant (certifiable) applications. >This will cost them money to develop applications >that are conformant rather than competative, >and I would rather the software industry was healthy and paying taxes, >and paying software developers like me (so that I may unfortunately pay taxes). > Again this is a business issue. The advantage of a testing service is that it offers an independent assessment of the claims being made by applications developers. Conformance to standards is one of many priorities that purchasers have the weigh up before deciding what products to select. >It would be a special kind of disservice to POSIX to try to >``put teeth into it'' for application conformance. >This will thrust it once more into the middle of a marketing battle, >because once more it will be perceived as a threat rather than a tool >by vendors. So why does POSIX go to such lengths to define the concept of applications conformance? Given that most vendors claim some sort of conformance to standards this issue of obviously important. The only threat that I see is that many claims will be shown to be not worth the hot air taken to say them. Don't purchasers really want to have some means of independently verifying vendors claims? > >POSIX.1 is an excellent base for a portability model, >used in conjunction with ANSI/ISO C. One can even write useful and >interesting applications using just these two standards. >One will likely want to expand their model to include specifications >(standard or otherwise) for a GUI API, possibly a GUI look-and-feel, >and a network IPC mechanism. This immediately puts the >application at odds with just POSIX.1 and its amendments, >with respect to applications conformance. The existing POSIX conformance model handles this situation. There are various places where things could be fine tuned. Lets not throw the baby out with the bath water. > >POSIX.1 is a source code portability standard. It is to be used >by application developers writing source code. It needs to be implemented >by system implementors on their architectures to be useful to those >application developers. It extends a useful portability model. >I believe discussions of POSIX.1 applications conformance and >testing and certification are nasty consumers of resources >(people and time), that offer no genuine benefit. > The same thing might be said about standards. But having decided to create a standard I think a responsible committee would also look to see that claims of conformance could be tested and ought to encourage work to carry out such testing. derek jones ps. I shall be leaving for a WG14 meeting in 12 hours (yes it is 9 days away, but ...) so will not be able to follow up any responses to this posting. Date: Mon, 18 Jul 1994 16:03:48 +0100 To: D.Cannon@Exeter.ac.uk From: p.tanner@xopen.co.uk (Paul Tanner) Subject: (p.janecek 8957) Application Portability Testing From Paul Tanner, X/Open To Dave Cannon, IST/5/-/15 Dave, Here are some comments on Rob Smith's paper. I thought it best to send them to you for circulation - or whatever. Regards, Paul Comments on App Conformance Testing ----------------------------------- We applaud Rob Smith's initiative to bring this subject to the fore. We agree that, if this issue is not addressed, POSIX and the whole concept of portability through use of standard APIs will be brought into disrepute. User demand, expressed though procurement policies could be a major factor in avoiding this outcome. However, the problem may be more complex than described in Rob's note. I would therefore like to make a few supplementary points in the hope that we can make sure the cure is not worse that the illness: 1. More and more software is subject to an appropriate "buy-not-build" policy. As such, the Independent application Software Vendor (ISV) is the primary purveyor of software. If ISVs cannot be convinced then software policies will surely fall on stony ground. This is why X/Open is striving to open a specific dialogue with precisely this constituency. (Note that we refer to a specific type of ISV here.) 2. Like all vendors, ISVs would submit their products for certification by accredited laboratories if there were sufficient market demand. However, that is *extremely* unlikely to happen (for reasons too numerous to mention here). They are, however, prepared take measures internally to maximise quality and minimise support costs. That should be our first goal for seeking their cooperation. 3. In discussions with many developers, both ISVs and major corporates, we have seen that there is a vast amount of knowledge about what it takes to make software portable. There is a willingness to explore the possibilities for practical means of improving on the ad-hoc approaches that have been used in this area. However, it will take a lot more than the kind of testing tools that exist today to get them (the real people) actively involved. 4. We have begun to compile a list of the problems that need to be solved in order to get the active attention of application developers. It is too early to give a definitive list but here are some key issues: - experienced developers have learnt the hard way that there is no one API set that solves all portability problems. If any enforcement were to be acceptable (or even useful) it would need to be adapted to developers' specific needs. There is therefore a need for flexibility in profiling of APIs within any tools they are asked to adopt. - they also know that it takes much more than POSIX to isolate an application from its underlying infrastructure. An API set at least as broad as XPG4 (including planned extensions) - a manor superset of POSIX - would be needed to afford any real degree of portability. The bottom line: This is an important initiative that needs ISV support to give it value. It is essential that their requirements be taken into account - better still they should be active contributors to any initiatives in this area. If we cannot get this support at this time, I think we would do well to keep our powder dry until we can. But, in case I am sounding negative, this is something important that must be addressed ASAP! Paul Tanner Bus. Development Mgr., Test Systems p.tanner@xopen.co.uk +44 734 508311 x2266 ------------------------------------------ Date: Wed, 3 Aug 94 23:51:31 EDT From: stephe@mks.com (Stephen Walli) To: sc22wg15@dkuug.dk Subject: (wg15-uk 542) (SC22WG15.379) More thoughts on Application Conformance (long-ish) Hello All, Thank you to those that took time to respond to my concerns. Hopefully, I can address some of the additional concerns raised as I appear to have described some of my concerns poorly. Most of this follow-up is based around Derek Jones' reply. Thanks, Derek, for taking the time. I appreciate your concerns and expertise w.r.t. application testing issues too well. dmj> Date: Sun, 17 Jul 94 19:56:12 BST dmj> From: knosof!derek@mks-gate.mks.com (Derek M Jones) dmj> To: wg15@pyra.co.uk dmj> dmj> srw> Date: Fri, 15 Jul 94 13:23:39 EDT dmj> srw> From: stephe@mks.com (Stephen Walli) dmj> srw> dmj> srw> POSIX.1's scope states that it dmj> srw> `` defines a standard operating system interface and environment dmj> srw> to support application portability at the source-code level. dmj> srw> It is intended to be used by both application developers and dmj> srw> system implementors.'' dmj> srw> dmj> srw> The intended users of POSIX.1 are application developers and system dmj> srw> implementors. Not purchasers of applications and platforms. dmj> dmj> So I guess 1003.3 have been wasting their time. dmj> dmj> Just because a standard is developed with an intended audience in mind dmj> does not preclude other interests using it. I certainly hope 1003.3 was not wasting its time! I have long used the example of P1003.3's work (both as a process and as specifications) in both IEEE Std 1003.3-1991 and IEEE Std 2003.1-1992 as examples of how test methods standards *should* be created. I am reading your reply to mean that P1003.3, as neither application developers nor system implementors made use of the POSIX.1 specification in their work. Absolutely true. If this is what you mean, however, I don't believe this is a good example of an alternate ``use'' of the standard. IEEE Std 2003.1-1992's purpose is ``to specify the test assertions and related test methods for measuring conformance of an implementation of POSIX.1{3}.'' It is essentially a collection of ``statement(s) of functionality ... derived from the POSIX standard being tested ....'' Others may read the specifications for other purposes, but I believe they stand as the blueprint between the participants identified in the scope. dmj> srw> dmj> srw> As a customer of an application, I would like to know that: dmj> srw> -- it runs on every platform upon which I wish to install it. dmj> ... dmj> dmj> But can you say today what platforms you will be interested in tomorrow. No. I don't know what will be available tomorrow. I would like to think they provide a POSIX.1 interface and an ISO C compiler and libraries. That will help protect my current application development investment as much as possible, without guarantees. As an ISV, I probably do not have sufficient buying clout personnally with the hardware vendors to ensure they will continue to support standards of interest to me. (There are very few ISVs such as Lotus, with sufficient SELLING clout to call the shots.) As a large MIS shop, or better yet, HUGE procurement concern, I probably have better clout with my hardware vendors. dmj> Software vendors showing some level of proof of conformance to dmj> POSIX are likely to have fewer porting problems than those who don't dmj> investigate the issues. ``Some level of proof of conformance to POSIX'' worries me. Which ``POSIX''? And with which conformance options? And with which functional options? And how does one distinguish? Your statement is absolutely correct w.r.t. ``likely to have fewer porting problems'' and an understanding of the issues. This was why I got involved with the IEEE POSIX standards committees in the first place, as an applications developer in a big-iron MIS department. I now work for an ISV. As a software vendor of an application that we want to be as portable as possible, we continue to be as knowledgeable as possible on POSIX.1 and its amendments in the process pipeline for exactly these reasons. dmj> Vendor proof of conformance provides a dmj> feel good factor to customers that there are no hidden purchasing dmj> problems (software not available on platform X) ahead. I believe this feel good factor is REALLY dangerous. POSIX.3 went out of its way to ensure its definitions were ALWAYS couched in terms of ``measuring conformance''. Never did they gaurantee it, nor give the illusion that test procedures built around the POSIX.3 specifications could be a guarantee of conformance. This is not a problem when dealing with implementations: it could be a very large problem w.r.t application conformance. Many useful applications will make use of features beyond IEEE Std 1003.1-1990 as described by FIPS 151-2 (which I use as a convenient short-hand for all the conformance options and specified limits.) This means that were one to be able to test the model used by an application suite, it would likely test as a ``conforming application with extensions''. Okay. What extensions? Are they also ISO or IEEE standards, possibly completely unrelated to POSIX in functionality because they are embedded SQL, or a GUI toolkit? Or are they a proprietary service, or a completely different industry spec? dmj> dmj> Also how are you the customer? If you are buying source code along dmj> with the binary then conformance is probably of great importance. dmj> As both you and Luigi point out, we are a little thin on definitions here. I am speaking from the experience of a application developer, spending most of the past 15 years in corporate MIS cultures, and developing applications by writing source code. In that culture I was also a purchaser and user of programmer analyst tools, such as, relational databases, CASE tools, and 4GLs. I'm not sure I would welcome debate/discussion on terms yet. dmj> .... I think it is important that we separate out the internal dmj> software engineering portability issues (which if done right will save dmj> a compnay lots of money) from the marketing issues "Our software dmj> conforms to Open Systems standards". I agree. I don't know if the market will allow us this luxury. But for the purposes of this discussion I hope we can. With respect to internal software portabillity issues, I agree that there are many issues that would be of interest to people who are also interested in POSIX based portability models. These issues probably range as broad a spectrum as ANDF, and Deploy technologies, through to better source management and construction techniques. I do not see why the efforts to standardize the source code interface to operating system services is necessarily involved. dmj> .... dmj> Given that NIST is already up an running the incremental cost of dmj> adding applications testing is small. I have yet to hear of the dmj> US Government directly funding development of test suites. Exactly. NIST's group of people are very experienced in the POSIX arena. Yet they are not particularly vocal on this issue. Any comments from NIST employees that participate? Official or otherwise? Especially on the incremental cost statement. dmj> dmj> srw> Likewise, dmj> srw> it would be a disservice to national application vendors to insist they dmj> srw> develop completely conformant (certifiable) applications. dmj> srw> This will cost them money to develop applications dmj> srw> that are conformant rather than competative, dmj> srw> and I would rather the software industry was healthy and paying taxes, dmj> srw> and paying software developers like me (so that I may unfortunately pay taxes). dmj> srw> dmj> dmj> Again this is a business issue. The advantage of a testing service is that dmj> it offers an independent assessment of the claims being made by dmj> applications developers. Conformance to standards is one of many dmj> priorities that purchasers have the weigh up before deciding what dmj> products to select. I remain unconvinced w.r.t. applications and how they're implemented. I guess I need more data, examples, etc. dmj> dmj> srw> It would be a special kind of disservice to POSIX to try to dmj> srw> ``put teeth into it'' for application conformance. dmj> srw> This will thrust it once more into the middle of a marketing battle, dmj> srw> because once more it will be perceived as a threat rather than a tool dmj> srw> by vendors. dmj> dmj> So why does POSIX go to such lengths to define the concept of dmj> applications conformance? Given that most vendors claim some sort dmj> of conformance to standards this issue of obviously important. I disagree. I'm not sure how obvious it is. I have not seen people claim conformance for their applications w.r.t. POSIX.1, only implementations. (I appreciate that this probably reflects more on my lack of reading of trendy magazines lately. If people have seen such claims, I would genuiely like to see them itemized, i.e. who, what, and where.) Implementation conformance is important w.r.t. purchasing platforms upon which to develop more portable applications. I don't see it as clearly defined when discussing applications. I may be really naive, but I can not imagine the form of a conformance claim for an application, and how that will effect my purchasing of platforms (implementations) 5 years from now that may conform to a more functional amendment of POSIX, that the application vendors may not yet have made use of .... The rationale for the POSIX.1 application conformance section makes passing reference to ``users or adaptors of applications''. As a member of a porting team I appreciate the ``adaptors of applications'' reference. Any one with direct experience as a ``user of applications'' actually making use of this information sucessfully, please step forward. It would certainly give me a better understanding of how I can think about this. dmj> The dmj> only threat that I see is that many claims will be shown to be dmj> not worth the hot air taken to say them. Don't purchasers really dmj> want to have some means of independently verifying vendors claims? If an application vendor says their application runs on these n platforms, then they wouldn't survive long if they didn't. If an application vendor says that their application will run on certain additional platforms in the future, this is marketing. Saying that the version of the application today supports a set of standards might be useful, if for example, it was a database tool, and I know as a user of the application I can type in a line of SQL and it will work. Saying that the version of the application today is implemented against a particular set of standards isn't particularly useful: - the application will not remain the same. It will evolve. Software just does that, based on user input and vendor marketing. - the application may make use of different functional groups of routines not in a standard (i.e. extensions) that will NOT be supported by future platforms. Saying that it is easier to port is not helpful. As a consumer paying money for an application, ``might port easier'' is meaningless. - Easier to port is a nasty ambiguous term. POSIX standards provide an interface definition against which to write applications to make them more portable. It doesn't address the myriad of details w.r.t to the process of managing and constructing the application on more than one platform, today or ever. dmj> dmj> srw> dmj> srw> POSIX.1 is an excellent base for a portability model, dmj> srw> used in conjunction with ANSI/ISO C. One can even write useful and dmj> srw> interesting applications using just these two standards. dmj> srw> One will likely want to expand their model to include specifications dmj> srw> (standard or otherwise) for a GUI API, possibly a GUI look-and-feel, dmj> srw> and a network IPC mechanism. This immediately puts the dmj> srw> application at odds with just POSIX.1 and its amendments, dmj> srw> with respect to applications conformance. dmj> dmj> The existing POSIX conformance model handles this situation. I'm not sure how. I've read what I believe are the appropriate sections of the standard. Please point me to the text and how it helps. dmj> srw> dmj> srw> POSIX.1 is a source code portability standard. It is to be used dmj> srw> by application developers writing source code. It needs to be implemented dmj> srw> by system implementors on their architectures to be useful to those dmj> srw> application developers. It extends a useful portability model. dmj> srw> I believe discussions of POSIX.1 applications conformance and dmj> srw> testing and certification are nasty consumers of resources dmj> srw> (people and time), that offer no genuine benefit. dmj> srw> dmj> dmj> The same thing might be said about standards. But having decided to dmj> create a standard I think a responsible committee would also look dmj> to see that claims of conformance could be tested and ought to encourage dmj> work to carry out such testing. The IEEE standards efforts are based on the voluntary participation of individual professionals in the industry, coming together in a consensus process to develop engineering specifications in their respective areas of interest. The IEEE Standards Board is not a funded testing house. And many of the volunteers that give of their time, and and their respective employers that support them, have taken the responsibility that, where they have specific economic concerns w.r.t. conformance of implementations, they have acted, committing people and resources, to ensure their needs are served. In some cases, this was as grand as X/Open's and NIST's, but then they have larger needs. In others, it was to ensure they were informed and educated consumers. lb> My thirtytwo lire (approximately two US cents). Hmmm, if we're using U.S. currency in this discussion, I guess this just became my three Canadian cents :-) cheers, stephe Date: Mon, 8 Aug 94 10:10:39 +1200 From: Keith Hopper Subject: (wg15-uk 549) (SC22WG15.384) Conformance -- to what? To: sc22wg15@dkuug.dk Hi, I have followed with interest the recent notes passing to and fro on the issue of conformance and have felt that perhaps I have been missing something -- at the very least my understanding of conformance seems to have been somehow dented in the rush. I remember at some time past a very impassioned plea from the UK delegation to consider defining conformance to POSIX in a different way, involving a new classification which, I believe, they termed rigorous. This seemed quite reasonable to me until I began to work on the M-2 binding (see WG13/D205 -- or SC22/16?? - mislaid the number, sorry). It was then that I began to see also some of the things which Steve has recently been mentioning in regard to applications. WG13 discussed this whole business at great length -- which is reflected in the conformance clauses in the binding WD -- much along the following lines :-- (1) An application is written in some source language X (say) and this is presumably translated by an implementation of a conforming language system. Some measure of this conformance rubs off into everything else used by that application. (2) If the application writer wishes to use package Y (which could happen to be POSIX of course) then there will be some binding defined for this package. Now the binding will have had to be specified in such a way that not only does the syntax and static semantics have to conform with language X, but so that in addition to any peculiarities of the dynamic semantics of language X, the dynamic semantics of Y are made available. Conformnce to this binding has therefore to be speciried in terms of the necessary mappings between X and Y. (3) The conformance to the package Y itself is not necessarily available to the application writer except as specified in the mapping for the binding -- which may not be total, of course. With these things in mind, I see the topic of conformance to POSIX itself as being something which has to be specified within the dynamic behaviour described in the standard. The syntactic/static semantic matters are the subject of bindings to various programming (or other languages), with the result that we can express something which WG13 decided to call 'Application Program Portability' rather than 'conformance' per se. In doing this we found that the UK suggestion of levels of decreasing portability was quite useful -- a. Rigorous --- uses a conforming implementation of the binding standard AND translated in standard mode by a conforming translator for language X AND runs using a conforming implementation of pakcage Y (eg POSIX) WITHOUT ANY IMPLEMENTATION-DEFINED EXTENSIONS. b. Strict --- uses a conforming implementation of the binding standard without using any permitted implementation-defined extensions, but need not have been translated in standard mode by a translator for language X and may use library material defined neither by the language nor binding standards. (Note that this still implies conforming to the semantics of package Y for which the binding is defined). c. Standard --- uses the binding, language X and package Y standards in any permitted combination and manner. In terms of what the recent discussion has been about as I have interpreted it, the 'Standard' level of portability which I have just outlined is, as Steve points out, not really very much use except for marketing 'hype' (dare I use that term?). The questions which pop up here then with regard to conformance testing relate to the combination used in producing an application, rather than the individual language/binding/package conformance. Are we in WG15 necessarily in the business of 'system integration' as opposed to 'component' testing. Frankly I'm not sure -- but I leave the question open, because it IS one which customers are going to want to know something about when they go shopping! Over to you! Keith Date: Sun, 7 Aug 94 16:48:13 BST From: knosof!derek@xopen.co.uk (Derek M Jones) To: wg15@pyramid-technology.co.uk Subject: (wg15-uk 548) (SC22WG15.383) Applications testing All, >Date: Wed, 3 Aug 94 23:51:31 EDT >From: stephe@mks.com (Stephen Walli) > >dmj> srw> >dmj> srw> As a customer of an application, I would like to know that: >dmj> srw> -- it runs on every platform upon which I wish to install it. >dmj> ... >dmj> >dmj> But can you say today what platforms you will be interested in tomorrow. >No. I don't know what will be available tomorrow. >I would like to think they provide a POSIX.1 interface and an ISO C >compiler and libraries. That will help protect my current application >development investment as much as possible, without guarantees. >As an ISV, I probably do not have sufficient buying clout personnally >with the hardware vendors to ensure they will continue to support standards >of interest to me. (There are very few ISVs such as Lotus, with sufficient >SELLING clout to call the shots.) >As a large MIS shop, or better yet, HUGE procurement concern, I probably >have better clout with my hardware vendors. > Who is the customer and what are they buying? A customer buying source is likely to be very interested in standards conformance. After all they are likely to be porting and maintaining (or paying somebody else to do it) their purchase. Customers buying binaries are unlikely to be that interested in the standards conformance of the source code. It is somebody else's problem. Knowing that the source conforms to standards will create a warm feeling in some people, but they are unlikely to be willing to pay much for it. >dmj> Software vendors showing some level of proof of conformance to >dmj> POSIX are likely to have fewer porting problems than those who don't >dmj> investigate the issues. > >``Some level of proof of conformance to POSIX'' worries me. >Which ``POSIX''? And with which conformance options? And >with which functional options? And how does one distinguish? Which ever bits of Posix are required. One useful side effect of checking applications is that it hightlights those optional parts of Posix that are used. My customers then feed this information back to their hardware procurement people (it also acts as a useful handle for management to control what the developers are assuming will exist on all platforms). > >Many useful applications will make use of features beyond >IEEE Std 1003.1-1990 as described by FIPS 151-2 (which I use as >a convenient short-hand for all the conformance options and specified >limits.) This means that were one to be able to test the model >used by an application suite, it would likely test as a ``conforming >application with extensions''. Okay. What extensions? Are they also >ISO or IEEE standards, possibly completely unrelated to POSIX in >functionality because they are embedded SQL, or a GUI toolkit? >Or are they a proprietary service, or a completely different industry spec? > Exactly, what options? An important part of the conformance checking process is the uncovering and documentation of these options. >dmj> >dmj> So why does POSIX go to such lengths to define the concept of >dmj> applications conformance? Given that most vendors claim some sort >dmj> of conformance to standards this issue of obviously important. > >I disagree. I'm not sure how obvious it is. Clause 1.3.2 "Applications Conformance" is on page 4. >I have not seen people claim conformance for their applications w.r.t. >POSIX.1, only implementations. (I appreciate that this probably reflects >more on my lack of reading of trendy magazines lately. If people >have seen such claims, I would genuiely like to see them itemized, >i.e. who, what, and where.) ... They usually use the blanket term Posix. What you need to do is talk to software developers and ask them simple questions in a non threatening way, like; what standards does your application conform to (I would not waste my time asking this question of people in the DOS world)? >dmj> >dmj> srw> >dmj> srw> POSIX.1 is an excellent base for a portability model, >dmj> srw> used in conjunction with ANSI/ISO C. One can even write useful and >dmj> srw> interesting applications using just these two standards. >dmj> srw> One will likely want to expand their model to include specifications >dmj> srw> (standard or otherwise) for a GUI API, possibly a GUI look-and-feel, >dmj> srw> and a network IPC mechanism. This immediately puts the >dmj> srw> application at odds with just POSIX.1 and its amendments, >dmj> srw> with respect to applications conformance. >dmj> >dmj> The existing POSIX conformance model handles this situation. > >I'm not sure how. I've read what I believe are the appropriate sections >of the standard. Please point me to the text and how it helps. > Clause 1.3.2.3 "Conforming POSIX.1 Applications Using Extensions" Why are you so against the testing of applications? Like most standards activities only those that are interested in it need be involved. In terms of industrial take, conformance certification will take to battle it out with other user requirements. At the moment there are a number of large organisations who want some form of applications certification. The Portable Applications Standards Committee ought to be responding to this need. derek Date: Tue, 9 Aug 94 13:22:18 BST From: knosof!derek@xopen.co.uk (Derek M Jones) To: wg15@pyramid-technology.co.uk Subject: (wg15-uk 551) (SC22WG15.386) Applications conformance and Modula 2 (not) All, >Date: Mon, 8 Aug 94 10:10:39 +1200 >From: Keith Hopper > > I remember at some time past a very impassioned plea from the UK delegation >to consider defining conformance to POSIX in a different way, involving a new >classification which, I believe, they termed rigorous. It is not a different way. It is simply a new classification to be added to the existing list. At the moment Posix only addresses the API. Companies interested in conforming applications also want the language in which the application is written to conform to standards. Rigorous Conformance is defined in terms of API and language. > This seemed quite >reasonable to me until I began to work on the M-2 binding (see WG13/D205 -- >or SC22/16?? - mislaid the number, sorry). Modula 2 has botched the whole topic of conformance. This work now seems to be driven solely by the requirements of the formal definition. Indeed it now appears to be purely an exercise in formally defining a large system with little connection to the real world. I responded to a WG13 public review with a request for a statement on applications conformance. After all Modula 2 is supposed to be the language in which one writes highly reliable, "safe" software. I was told that after a lot of discussion this concept did not fit into the current methods used to define the language. derek Date: Tue, 9 Aug 94 14:40:50 EDT From: arniep@VNET.IBM.COM To: d.cannon@exeter.ac.uk, dekek@knosof.uucp Subject: POSIX Testing Dave I did read your note on the testing of POSIX applications that you sent out some time ago. I really haven't followed the UK discussion (stephe and derek) but I feel we have a problem here somewhere and that we have to address it. In your note you describe a situation where a user organization has a real and valid concern re their investment in POSIX. I sense, and not just from this discussion, that we have to take action to preserve the credibility of the POSIX effort. Testing is definitely one aspect of this but I'm not convinced that there is any serious effort going into the development of POSIX conforming applications today. I don't know if testing and conformance processes would stimulate any effort but somehow we have to encourage POSIX application development. Otherwise we will end up creating several thousand pages of documentation that will either not be used. The benefits to users will be at best minimal. So I hope you've established an agenda item for the October meeting. Regards Arnie Date: Wed, 10 Aug 94 09:09:34 +1200 From: Keith Hopper Subject: (wg15-uk 552) (SC22WG15.387) What is conformance? To: sc22wg15@dkuug.dk Hi, Mmmmmm! I detect a certain amount of dissatisfaction in message 386 about what is perceived to be the "Modula-2 view of conformance". Having a hand in both camps I feel that there has to be a misunderstanding somewhere, since I have a single view -- which fits both the POSIX version being developed and the M-2 one in the DIS. I believe that conformance is about the ability to assign a meaning to a program. In terms of a programming language standard the conformance of an implementation either allows this or it doesn't. We have all seen meaningless programs, I'm sure. In terms of a binding conformance says that an implementation must bridge the gap (small or large) between the semantics of the package to which a binding is being made and the semantics of meaningful code written in the language X. It is a Janus-like conformance requirement. For a package, the matter of conformance is independent of the language and is therefore more of an open contract between package provider and user --- nevertheless only ascribing meaning to some action if the user meets any pre-conditions -- his part of the contract. This view of conformance seems to be consistent in that at the language level there is either a meaningful chunk of source text -- or not, a binding either provides transparent mapping of meaning -- or not, a package either satisfies itd 'crontactual' obligations -- or not. Testing for these is not something which can be done independently -- there has to be some form of integration testing, particularly for bindings. Doesn't this seem sense? Regards, Keith Date: Wed, 10 Aug 94 13:39:02 BST From: knosof!derek@xopen.co.uk (Derek M Jones) To: wg15@pyramid-technology.co.uk Subject: (wg15-uk 553) (SC22WG15.388) What does conformance show? All, >Date: Wed, 10 Aug 94 09:09:34 +1200 >From: Keith Hopper > > I believe that conformance is about the ability to assign a meaning to a >program. In terms of a programming language standard the conformance of >an implementation either allows this or it doesn't. We have all seen meaningless >programs, I'm sure. > I don't see what meaning has to do with conformance. Conformance is about an application or implementation following the rules laid down in a standards document. The intent is that following such rules will result is a known set of actions happening. Because standards do not fully define everything (they specify some constructs as resulting in undefined, implementation defined or unspecified behaviours) there are various levels of conformance. At the top we have strictly conforming. Such an application does not rely on any construct whose behaviour may change. So it should give the same result on all implementations. A Conforming application only uses those constructs defined by the standard and uses some of those whose behaviour may change. Such an application may give different results on different implementations. Further on down the conformance hierarchy we have applications depending on other standards. derek