From stephe@mks.com Wed Aug  3 19:51:31 1994
Received: from mks-gate.mks.com by dkuug.dk with SMTP id AA07084
  (5.65c8/IDA-1.4.4j for <sc22wg15@dkuug.dk>); Thu, 4 Aug 1994 05:51:43 +0200
Received: from mks.com by mks-gate.mks.com (8.6.8.1/MKS-1.1)
	id XAA04133; Wed, 3 Aug 1994 23:51:30 -0400
Received: by mks.com (4.1/GIGA-1.2)
	id AA01090; Wed, 3 Aug 94 23:51:33 EDT
Date: Wed, 3 Aug 94 23:51:31 EDT
From: stephe@mks.com (Stephen Walli)
Message-Id: <9408040351.AA01090@mks.com>
To: sc22wg15@dkuug.dk
Subject: More thoughts on Application Conformance (long-ish)
Content-Length: 13962
X-Charset: ASCII
X-Char-Esc: 29

Hello All,

Thank you to those that took time to respond to my concerns. 
Hopefully, I can address some of the additional concerns raised 
as I appear to have described some of my concerns poorly. 
Most of this follow-up is based around Derek Jones' reply. 
Thanks, Derek, for taking the time.  I appreciate your concerns
and expertise w.r.t. application testing issues too well. 

dmj> Date: Sun, 17 Jul 94 19:56:12 BST
dmj> From: knosof!derek@mks-gate.mks.com (Derek M Jones)
dmj> To: wg15@pyra.co.uk
dmj> 
dmj> srw> Date: Fri, 15 Jul 94 13:23:39 EDT
dmj> srw> From: stephe@mks.com (Stephen Walli)
dmj> srw> 
dmj> srw> POSIX.1's scope states that it 
dmj> srw> `` defines a standard operating system interface and environment
dmj> srw> to support application portability at the source-code level. 
dmj> srw> It is intended to be used by both application developers and
dmj> srw> system implementors.''
dmj> srw> 
dmj> srw> The intended users of POSIX.1 are application developers and system
dmj> srw> implementors.  Not purchasers of applications and platforms.
dmj> 
dmj> So I guess 1003.3 have been wasting their time.
dmj> 
dmj> Just because a standard is developed with an intended audience in mind
dmj> does not preclude other interests using it.

I certainly hope 1003.3 was not wasting its time! 
I have long used the example of P1003.3's work (both as a process
and as specifications) in both 
IEEE Std 1003.3-1991 and IEEE Std 2003.1-1992 as examples of
how test methods standards *should* be created.  

I am reading your reply to mean that P1003.3, 
as neither application developers nor system implementors
made use of the POSIX.1 specification in their work. 
Absolutely true.  

If this is what you mean, however, I don't believe this is a 
good example of an alternate ``use'' of the standard. 
IEEE Std 2003.1-1992's purpose is ``to specify the test assertions and
related test methods for measuring conformance of an implementation of
POSIX.1{3}.''  It is essentially a collection of ``statement(s) of 
functionality ... derived from the POSIX standard being tested ....''

Others may read the specifications for other purposes, 
but I believe they stand as the blueprint between the participants 
identified in the scope.

dmj> srw> 
dmj> srw> As a customer of an application, I would like to know that: 
dmj> srw> -- it runs on every platform upon which I wish to install it. 
dmj> ...
dmj> 
dmj> But can you say today what platforms you will be interested in tomorrow.
No. I don't know what will be available tomorrow. 
I would like to think they provide a POSIX.1 interface and an ISO C
compiler and libraries.  That will help protect my current application
development investment as much as possible, without guarantees. 
As an ISV, I probably do not have sufficient buying clout personnally 
with the hardware vendors to ensure they will continue to support standards
of interest to me. (There are very few ISVs such as Lotus, with sufficient
SELLING clout to call the shots.) 
As a large MIS shop, or better yet, HUGE procurement concern, I probably
have better clout with my hardware vendors. 

dmj> Software vendors showing some level of proof of conformance to
dmj> POSIX are likely to have fewer porting problems than those who don't
dmj> investigate the issues.  

``Some level of proof of conformance to POSIX'' worries me. 
Which ``POSIX''?  And with which conformance options? And 
with which functional options? And how does one distinguish?

Your statement is absolutely correct w.r.t. ``likely to have fewer
porting problems'' and an understanding of the issues.  This was 
why I got involved with the IEEE POSIX standards committees in the 
first place, as an applications developer in a big-iron MIS department.
I now work for an ISV.  As a software vendor of an application that we want 
to be as portable as possible, we continue to be as knowledgeable as 
possible on POSIX.1 and its amendments in the process pipeline for 
exactly these reasons. 

dmj> Vendor proof of conformance provides a
dmj> feel good factor to customers that there are no hidden purchasing
dmj> problems (software not available on platform X) ahead.

I believe this feel good factor is REALLY dangerous. 
POSIX.3 went out of its way to ensure its definitions were ALWAYS
couched in terms of ``measuring conformance''.  Never did they gaurantee
it, nor give the illusion that test procedures built around the 
POSIX.3 specifications could be a guarantee of conformance. 
This is not a problem when dealing with implementations:
it could be a very large problem w.r.t application conformance.

Many useful applications will make use of features beyond 
IEEE Std 1003.1-1990 as described by FIPS 151-2 (which I use as
a convenient short-hand for all the conformance options and specified
limits.)  This means that were one to be able to test the model 
used by an application suite, it would likely test as a ``conforming
application with extensions''.  Okay. What extensions? Are they also 
ISO or IEEE standards, possibly completely unrelated to POSIX in 
functionality because they are embedded SQL, or a GUI toolkit?
Or are they a proprietary service, or a completely different industry spec?  

dmj> 
dmj> Also how are you the customer?  If you are buying source code along
dmj> with the binary then conformance is probably of great importance.
dmj> 
As both you and Luigi point out, we are a little thin on definitions 
here. 

I am speaking from the experience of a application developer,
spending most of the past 15 years in corporate MIS cultures, 
and developing applications by writing source code. In that
culture I was also a purchaser and user of programmer analyst tools, such as,
relational databases, CASE tools, and 4GLs. 

I'm not sure I would welcome debate/discussion on terms yet. 

dmj> .... I think it is important that we separate out the internal
dmj> software engineering portability issues (which if done right will save
dmj> a compnay lots of money) from the marketing issues "Our software
dmj> conforms to Open Systems standards".

I agree. I don't know if the market will allow us this luxury. 
But for the purposes of this discussion I hope we can. 
With respect to internal software portabillity issues,
I agree that there are many issues that would be of interest to 
people who are also interested in POSIX based portability models. 
These issues probably range as broad a spectrum as ANDF, and 
Deploy technologies, through to better source management and 
construction techniques.  I do not see why the efforts to 
standardize the source code interface to operating system services
is necessarily involved. 

dmj> ....
dmj> Given that NIST is already up an running the incremental cost of
dmj> adding applications testing is small.  I have yet to hear of the
dmj> US Government directly funding development of test suites.

Exactly. NIST's group of people are very experienced in the POSIX
arena.  Yet they are not particularly vocal on this issue. 
Any comments from NIST employees that participate? Official or otherwise?
Especially on the incremental cost statement. 

dmj> 
dmj> srw> Likewise, 
dmj> srw> it would be a disservice to national application vendors to insist they
dmj> srw> develop completely conformant (certifiable) applications. 
dmj> srw> This will cost them money to develop applications 
dmj> srw> that are conformant rather than competative,
dmj> srw> and I would rather the software industry was healthy and paying taxes,
dmj> srw> and paying software developers like me (so that I may unfortunately pay taxes).
dmj> srw> 
dmj> 
dmj> Again this is a business issue.  The advantage of a testing service is that
dmj> it offers an independent assessment of the claims being made by
dmj> applications developers.  Conformance to standards is one of many
dmj> priorities that purchasers have the weigh up before deciding what
dmj> products to select.
I remain unconvinced w.r.t. applications and how they're implemented.
I guess I need more data, examples, etc. 

dmj> 
dmj> srw> It would be a special kind of disservice to POSIX to try to 
dmj> srw> ``put teeth into it'' for application conformance. 
dmj> srw> This will thrust it once more into the middle of a marketing battle,
dmj> srw> because once more it will be perceived as a threat rather than a tool
dmj> srw> by vendors.
dmj> 
dmj> So why does POSIX go to such lengths to define the concept of
dmj> applications conformance?  Given that most vendors claim some sort
dmj> of conformance to standards this issue of obviously important.  

I disagree.  I'm not sure how obvious it is. 
I have not seen people claim conformance for their applications w.r.t.
POSIX.1, only implementations. (I appreciate that this probably reflects
more on my lack of reading of trendy magazines lately.  If people 
have seen such claims, I would genuiely like to see them itemized, 
i.e. who, what, and where.)  Implementation conformance is important
w.r.t. purchasing platforms upon which to develop more portable
applications.  I don't see it as clearly defined when discussing 
applications.  I may be really naive, but I can not imagine the
form of a conformance claim for an application, and how that will
effect my purchasing of platforms (implementations) 5 years from now 
that may conform to a more functional amendment of POSIX, that
the application vendors may not yet have made use of ....

The rationale for the POSIX.1 application conformance section makes passing 
reference to ``users or adaptors of applications''. As a member of 
a porting team I appreciate the ``adaptors of applications'' reference.
Any one with direct experience as a ``user of applications'' 
actually making use of this information sucessfully, please step forward.
It would certainly give me a better understanding of how I can 
think about this. 

dmj> The
dmj> only threat that I see is that many claims will be shown to be
dmj> not worth the hot air taken to say them.  Don't purchasers really
dmj> want to have some means of independently verifying vendors claims?

If an application vendor says their application runs on these
n platforms, then they wouldn't survive long if they didn't. 
If an application vendor says that their application will run
on certain additional platforms in the future, this is marketing.

Saying that the version of the application today supports a set of 
standards might be useful, if for example, it was a database tool,
and I know as a user of the application I can type in a line of 
SQL and it will work. 

Saying that the version of the application today is implemented 
against a particular set of standards isn't particularly useful:
- the application will not remain the same. It will evolve. Software
  just does that, based on user input and vendor marketing. 
- the application may make use of different functional groups of
  routines not in a standard (i.e. extensions) that will NOT
  be supported by future platforms.  Saying that it is easier to 
  port is not helpful. As a consumer paying money for an application,
  ``might port easier'' is meaningless. 
- Easier to port is a nasty ambiguous term. POSIX standards provide
  an interface definition against which to write applications to make
  them more portable.  It doesn't address the myriad of details w.r.t
  to the process of managing and constructing the application on
  more than one platform, today or ever. 

dmj> 
dmj> srw> 
dmj> srw> POSIX.1 is an excellent base for a portability model, 
dmj> srw> used in conjunction with ANSI/ISO C. One can even write useful and 
dmj> srw> interesting applications using just these two standards.
dmj> srw> One will likely want to expand their model to include specifications
dmj> srw> (standard or otherwise) for a GUI API, possibly a GUI look-and-feel,
dmj> srw> and a network IPC mechanism.  This immediately puts the 
dmj> srw> application at odds with just POSIX.1 and its amendments,
dmj> srw> with respect to applications conformance. 
dmj> 
dmj> The existing POSIX conformance model handles this situation.  

I'm not sure how.  I've read what I believe are the appropriate sections
of the standard.  Please point me to the text and how it helps. 

dmj> srw> 
dmj> srw> POSIX.1 is a source code portability standard.  It is to be used 
dmj> srw> by application developers writing source code.  It needs to be implemented
dmj> srw> by system implementors on their architectures to be useful to those
dmj> srw> application developers.  It extends a useful portability model.
dmj> srw> I believe discussions of POSIX.1 applications conformance and
dmj> srw> testing and certification are nasty consumers of resources
dmj> srw> (people and time), that offer no genuine benefit.
dmj> srw> 
dmj> 
dmj> The same thing might be said about standards.  But having decided to
dmj> create a standard I think a responsible committee would also look
dmj> to see that claims of conformance could be tested and ought to encourage
dmj> work to carry out such testing.

The IEEE standards efforts are based on the voluntary participation of
individual professionals in the industry, coming together in a consensus
process to develop engineering specifications in their respective areas 
of interest.  The IEEE Standards Board is not a funded testing house. 

And many of the volunteers that give of their time, and and their respective
employers that support them, have taken the responsibility that, where they
have specific economic concerns w.r.t. conformance of implementations,
they have acted, committing people and resources, 
to ensure their needs are served. In some cases, this was as grand as 
X/Open's and NIST's, but then they have larger needs. 
In others, it was to ensure they were informed and educated consumers. 

lb>  My thirtytwo lire (approximately two US cents).
Hmmm, if we're using U.S. currency in this discussion, 
I guess this just became my three Canadian cents :-)

cheers,
stephe