Created: 12 Jan 2005
Last revised by $Author: kahan $ on $Date: 13 April 2005 - 19:02:22$
In order to move forward the XKMS specification from CandidateRecommendation to Proposed Recommendation status, the XKMS Working Group (WG)had to give enough implementation experience to satisfy the PR entrancecriteria. The method adopted by the WG was to create anXKMSAssertions and Test Collection. This document ennumerate all theassertions on the XKMS specification, specify tests for these assertions or,otherwise, give rationale as to why an assertion could not be tested. Thetests were grouped into four categories: X-KISS, X-KRSS, Compound, andOptional, corresponding to the two basic XKMS services, combining multiplerequests, and finally, tests for optional features, respectively.
The following Entrance Criteria were proposed in the XKMS CandidateRecommendationannouncement (you need a W3C member password to browse this link):
The Working Group succesfully tested all of the HTTP Transport andSoap 1.2/1.1 bindings, As a result of the feedback received, theSecurity Binding sections was made more concise and clear and nowlists a number of security considerations and how these considerationscould be taken into account by the XKMS payload security features, theTLS ones, or by a combination of both of these. Although there were nospecific tests for the payload security bindings, all of the XKMSfeatures on which these bindings depend were included in one or moreof the tests. The TLS bindings were not tested as the developers feltit was out of scope for the XKMS interoperability test; TLS securityproperties have already been tested and discussed in other fora and areindependent from XKMS.
During the CR period, the WG further refined the two serverimplementation exit criteria to request that there are at least twoservers that are contacted by two or more clients. Likewise, the WGadded an extra criteria requesting that there be at least one clientthat contacts two different servers. The WG felt these refinementsgave a better proof of interoperability.
The Working Group defined a total of 36testscenarios (18 X-KISS, 14 X-KRSS, 1 Compound, 3 Optional). The results aresummarized inAppendix A. Two clients (VM, GA)implemented all the tests. An additional client (TL) implemented reportedsuccess on all the tests, except for the Optional ones as ourreporting rules didn't allow for a developer to reportresults against his own server. Two servers (TL, SQL Data) supported all thetests except for the Optional ones. Only one server (TL ) supported theOptional tests. Both servers were tested against two clients at least. Thesetests satisfy the interoperability entrance criteria to ProposedRecommendation.Appendix A gives a breakdown of thetests.
The Working Group received a total of 43issues. Allbut three suggested changes to the XKMS specification which wereaccepted by the WG (see the Changelog of theXKMSspecification and itsbindings).
The Working GroupG declined the following three issues, the first two as beingdeliberately not addressed and the third one as being out of scope ofthe specification. All the reviewers agreed to the Working Groups responses:
The Environmental Information Exchange Network (EIEN)of the United States is in aprocess of deploying aliveXKMS 2.0 service. It will go live in a couple of months. The ExchangeNetwork is a web service network that links information systems in the stategovernments and federal government agencies, and allows automated and securedata exchanges between Network Node (the service endpoint). The projectstarted about 3 years ago, currently there are 32 states participating inlive data exchanges, many more are in the development and testing stage. Thegoal is to have all 50 states to join the Exchange Network. It is perhaps thelargest web service network in the US.
The Exchange Network has a centralized security service - NetworkAuthentication and Authorization Services (NAAS), the idea is to have a liveXKMS service and move toward public key authentication with signedauthentication messages, at least between Network Nodes:
Two of the interoperability participants prepared and contributed a set of XKMS messages that were exchanged during the XKMS CR interoperability phase.These messages are to be seen as advisory, rather than normative.
XKMS client developers were asked to report their success or failure ofrunning the tests described in the theXKMSAssertions and Test Collection document against a number of XKMS serversusing an onlinequestionnaire,that was open from 2004-09-14 to 2005-01-28 (you need to have a W3C member or public password in order to browse that link). The following table is a summaryof thequestionnaireresults from the point of view of the entrance criteria. Only the clientsand servers that were reported as succesful in the XKMS CR test suite reportwere taken into account. Some developers built both client and servers. Inthose cases, only the tests of those clients against other servers were takeninto account.
The following information was reported directly by the developers.It does not necessarily represent the latest state of any givenimplementation over this or later specifications.
The WG defined 18 tests. These tests and their results satisfy theREQUIRED entrance criteria as stated above: at least two client and twoserver implementations of each feature; for each feature, at least twoservers contacted by at least two different clients; there should be at leastone client that contacts two different servers.
We had four server implementations, of which two (TL, SQL) implemented allthe tests and were contacted by at least two different clients each. We hadup to seven client implementations of which three implemented all the tests(TL, GA, VM).
test | server implementations | client implementations | at least two servers contacted by at least two clients | at least one client contacts at least two servers | comments |
XKISS-T1: Locate | 3 TL Server, SQL Data, ASF-XKMS | 5 BL, VM, YZ, TL, GA | yes | yes | |
XKISS-T2: Validate | 4 TL Server, Entrust, SQL Data, ASK-XKMS | 7 | yes | yes | |
XKISS-T3: Locate - not found | 3 TL Server, SQL Data, ASF-XKMS | 5 BL, YZ, VM, TL, GA | yes | yes | |
XKISS-T4: Validate an expired cert | 4 TL Server, Entrust, SQL Data, ASF-XKMS | 6 RL, BL, YZ, VM, TL, GA | yes | yes | |
XKISS-T5: Validate a revoked cert | 4 TL Server, Entrust, SQL Data, ASF-XKMS | 6 RL, BL, YZ, VM, TL, GA | yes | yes | |
XKISS-T6: Two Phase | 3 TL Server, SQL Data, ASF-XKMS | 5 BL, YZ, VM, TL, GA | yes | yes | |
XKISS-T7: Asynchronous | 3 TL Server, SQL Data, ASF-XKMS | 5 BL, YZ, VM, TL, GA | yes | yes | |
XKISS-T8: Two Phase + Asynchronous | 3 TL Server, SQL Data, ASF-XKMS | 5 BL, YZ, VM, TL, GA | yes | yes | |
XKISS-T9: Compound | 3 TL Server, SQL Data, ASF-XKMS | 5 BL, YZ, VM, TL, GA | yes | yes | |
XKISS-T10: Two Phase Compound | 3 TL Server, SQL Data, ASF-XKMS | 5 BL. YZ, VM, TL, GA | yes | yes | BL's self-test answer ignored. |
XKISS-T11: Asynchronous Compound | 2 TL Server, SQL Data | 4 YZ, VM, TL, GA | yes | yes | |
XKISS-T12: Compound with inner asynchronous requests | 2 TL Server, SQL Data | 3 VM, TL, GA | yes | yes | |
XKISS-T13: Soap 1.1 | 4 TL Server, Entrust, SQL Data, ASF-XKMS | 6 BL, RS, YZ, VM, TL, GA | yes | yes | TL: Used T2 for Entrust,, but not important as we were testing the Soap 1.1 bindings. |
XKISS-T14: Soap 1.2 | 3 TL Server, SQL Data, ASF-XKMS | 5 BL, YZ, VM, TL, GA | yes | yes | |
XKISS-T15: Opaque Client Data | 3 TL Server, SQL Data, ASF-XKMS | 5 BL, YZ, VM, TL, GA | yes | yes | |
XKISS-T16: Request Signature Value | 3 TL Server, SQL Data, ASF-XKMS | 5 BL, YZ, VM, TL, GA | yes | yes | |
KISS-T17: Unsuccessful Request Signature Value | 3 TL Server, SQL Data, ASF-XKMS | 4 BL, VM, TL, GA | yes | yes | |
XKISS-T18: Response Limit | 3 TL Server, SQL Data, ASF-XKMS | 5 BL, YZ, VM, TL, GA | yes | yes |
The WG defined 14 tests. These tests and their results satisfy theREQUIRED entrance criteria as stated above: at least two client and twoserver implementations of each feature; for each feature, at least twoservers contacted by at least two different clients; there should be at leastone client that contacts two different servers.
We had two server implementations (TL, SQL); both of them implemented allthe tests and were contacted by at least two different clients each. We hadup to four client implementations of which three implemented all the tests(TL, GA, VM). The remaining client implemented all the tests except for one(XKRSS-T13).
test | server implementations | client implementations | at least two servers contacted by at least two clients | at least one client contacts at least two servers | comments |
XKRSS-T1: Register Client Generated Key | 2 TL Server, SQL Data | 4 YZ, VM, TL, GA | yes | yes | |
XKRSS-T2: Register Service Generated Key | 2 TL Server, SQL Data | 4 YZ, VM, TL, GA | yes | yes | |
XKRSS-T3: Reissue | 2 TL Server, SQL Data | 4 YZ, VM, TL, GA | yes | yes | |
XKRSS-T4: Recover | 2 TL Server, SQL Data | 4 YZ, VM, TL, GA | yes | yes | |
XKRSS-T5: Revoke | 2 TL Server, SQL Data | 4 YZ, VM, TL, GA | yes | yes | |
XKRSS-T6: Revoke with shared secret | 2 TL Server, SQL Data | 4 YZ, VM, TL, GA | yes | yes | |
XKRSS-T7: Two Phase | 2 TL Server, SQL Data | 4 YZ, VM, TL, GA | yes | yes | |
XKRSS-T8: Asynchronous | 2 TL Server, SQL Data | 4 YZ, VM, TL, GA | yes | yes | |
XKRSS-T9: Asynchronous + Two Phase | 2 TL Server, SQL Data | 4 YZ, VM, TL, GA | yes | yes | |
XKRSS-T10: Compound | 2 TL Server, SQL Data | 4 YZ, VM, TL, GA | yes | yes | |
XKRSS-T11: Two Phase Compound | 2 TL Server, SQL Data | 4 YZ, VM, TL, GA | yes | yes | |
XKRSS-T12: Asynchronous Compound | 2 TL Server, SQL Data | 4 YZ, VM, TL, GA | yes | yes | |
XKRSS-T13: Compound with inner asynchronous requests | 2 TL Server, SQL Data | 3 VM, TL, GA | yes | yes | VM reported a different behavior on SQL Data, hut this is without incidence on th entrance criteria. |
XKRSS-T14: Unsuccessful authorization | 2 TL Server, SQL Data | 4 YZ, VM, TL, GA | yes | yes |
The WG defined one test. This test and its result satisfy the REQUIREDentrance criteria as stated above: at least two client and two serverimplementations of each feature; for each feature, at least two serverscontacted by at least two different clients; there should be at least oneclient that contacts two different servers.
Two servers (TL, SQL) and three clients (VM, TL, GA) implemented thetests.
test | server implementations | client implementations | at least two servers contacted by at least two clients | at least one client contacts at least two servers | comments |
Compound-T1: XKISS and XKRSS | 2 TL Server, SQL Data | 3 VM, TL, GA | yes | yes |
The WG defined three tests. These tests and their results satisfy theOPTIONAL entrance criteria as stated above: at least one client and oneserver implementations of each feature.
One server (TL) and two clients (VM, GA) implemented all the tests.
test | server implementations | client implementations | two client requests to each server | a client contacts at least two servers | comments |
Optional-T1: Authentication with Private Key | 1 TL Server | 2 VM, GA | yes | no | |
Optional-T2: Authentication with NotBoundAuthentication | 1 TL Server | 2 VM, GA | yes | no | |
Optional-T3: Validate with RetrievalMethod | 1 TL Server | 2 VM, GA | no | no |