We are still interested in application conformance and some form
	of testing to enable ISVs and developers to know that the s/w
	they build really does conform. Furthermore, we wish our
	purchasers to know that when they purchase an application they
	can know with some confidence that 1) The application is not
	overly tied to a particular platform, 2) That the purchasers are
	not overly tied to the vendor of the application (especially in
	the NHS, we have many small vendors who will not be in business
	as long as the NHS), 3) that the application will fit within
	information management and technology strategies, and will
	demonstrate VFM over a period when the platform required to run
	the s/w may change.

	In the vast majority of cases NHS organisations do not have the
	cash to replace a single application (and the platform that
	supports it) in one go, let alone replace a whole hospital IT
	system in one go. This means that we must buy a steady amount of
	replacement platform/application each financial year, that is
	within budegtary limits, and fits within an overall strategy for
	continuous replacement. A strategy such as this requires
	standards (and also matches the client/server model well). The
	area is too complex for the word of the developer to be taken.
	Applications require testing just as platforms do.

	Openness is about value for money and reduction of risk, not
	technology, standards, testing, etc. But to demonstrate VFM and
	to really reduce the risk to a purchaser testing must be
	available on both sides of the API equation (platform and
	application).

	Application testing itself is a complex issue, and one that I do
	not think is solvable thru' 2nd or 3rd party testing. There are
	too many applications (in different configurations on different
	operating systems) to test in this way. Further, applications
	really require testing for standards conformance as part of the
	development cycle. This indicates that the only reasonable route
	to application conformance testing is thru' 1st party testing.
	The conformance test software needs to be available on an fair
	basis to all ISVs and developers. For example, in the NHS we
	have many very small ISVs (employing maybe less than 5-10
	people). In the majority of these cases the NHS is the only
	customer and they may be one of only two or three (suppliers of
	that software). ISVs of this type could not pay the fees
	associated with most conformance s/w (cf: Single UNIX spec test
	s/w, POSIX test bench s/w , etc) and would not. We do not want
	an application conformance regime that favours the big ISV
	vendors over the small ISVs for no reasons except their
	respective bank balance.

	This creates a diffcult situation. We want a fair and equitable
	testing arrangement that is available to all developers and
	doesn't cost so much that it will cost small ISVs out of the
	market. Perhaps there is a soultion based on testing "houses",
	set up on the Internet, with cost associated with membership
	and/or a charge per test run. But, there is a difficulty here.
	How many ISVs do you know that would be willing to send their
	source code over the Internet to be tested by someone else and
	then trust them to delete all copies of the source sent to them
	(and not read it while it is being tested and not leak it
	anywhere). Not many, I think is the answer. Whatever model is
	chosen for conformance testing it must be confidential and
	secure, until the software has been published with its brand.
	ISVs must be confident that the process is trustworthy.

	As a user organisation, most of our IT investment is tied up in
	components upwards from the operating system (i.e. DB,
	application, application interoperability, people, etc). It
	therefore makes sense to be concerned about ensuring the best
	VFM and least potential risk when investing in these areas. This
	leads to a requirement to be able to demonstrate to the
	satisfaction of all parties involved that the best VFM and least
	risk options, that fit within the IM&T strategy of the
	purchasing organisation, have been selected.


	Hope some of this has been useful,

	- Rob Smith.