Questions for TSC Regarding the ONC Conformance Testing Project

TSC Questions from the Initial Report Submitted to the ONC

HL7’s proposed approach at this point in time is to require that its implementation guides provide relevant conformance statements that are machine computable and robust enough to enable conformance tooling vendors to develop conformance tooling concurrent with the development of the implementation guide. HL7 feels very strongly that it should not develop conformance testing tools internally but should provide robust conformance statements that allow tooling experts to do so.

ONC has also requested that HL7 identify a means to collect and report implementation/usage data at the message level on an annual basis. To achieve the desired future state HL7 will need to:

1. Develop a policy that requires conformance statements in all implementation guides
HL7 must develop, implement and enforce a policy requiring that all implementation guides balloted through the organization include relevant machine-processable conformance statements that conform to a pre-developed set of criteria. These conformance statements will likely be balloted as part of the underlying implementation guides.

• These open issues will be presented to the HL7 Technical Steering Committee for continued discussion and resolution:
  • Since the HL7 community at large is not well versed in writing conformance statements, the organizations will need to decide who ballots these statements. We may need to introduce a new class of voting members (conformance tooling vendors) who vote separately but simultaneously on the statements for some period of time until the HL7 community at large develops some expertise in this area. This would ensure that the conformance statements receive sufficient review from a community of voters who are knowledgeable and have appropriate skills. It would also ensure a feedback loop from the tooling vendors to the standards developers.
  • Should balloting of the conformance statements be rolled into the implementation guide ballot at some point in the future? If so, what is the appropriate time for doing this?
  • Requiring that all implementation guides contain appropriate conformance statements will likely impact the speed at which implementation guides are developed and ready for ballot.

1. Develop a guidance document that defines how to develop robust conformance statement
   1. Issues related to this item:
      1. Which internal bodies will have input on this document and oversee its development.
      2. Which internal body (e.g. the Technical Steering Committee, the entire HL7 Community, etc) will approve this document and how that will be achieved.
      3. HL7 will need to secure funding (whether from the HL7 budget or an external funder) and develop

1. Train work groups to develop the conformance statements
   1. Issues related to this item:
      1. Do we need multiple levels of training (e.g., one level for general membership and a more rigorous program for conformance facilitators).
      2. HL7 will need to secure funding (either from the HL7 budget or an external funder) to develop the training.

1. Recruit conformance facilitators
   1. Issues related to this item:
      1. Do we have enough volunteers to develop the role fully
      2. Who would be in charge of the conformance facilitator development program
      3. Will development of this program require financial resources beyond those needed for the training discussed above

1. Develop a plan for creating conformance statements for all or some subset of extant implementation guides for which there is not conformance tooling.
2. Develop a certification program to certify conformance tools and/or tooling vendors
3. Develop and maintain a tooling catalog
4. Continue efforts to create examples (CDA) and models (CIMI) for representing data
5. Identify method to collect and report implementation/usage metrics annually

Original Questions to the TSC

1. The general approach that was discussed several months ago with the TSC chair, CTO and staff was that HL7 should not be in the business of creating conformance testing. Rather, we should create conformance satements as part of our IG that are then rolled up and distributed to external toolsmiths capable of creating conformance testing tools (probably at the same time that the standards are created). While my statement of the process is likely over simplified, does the TSC agree that this is the preferred approach?

• F.Hall - yes, and this has worked for the US Realm Lab IGs, conformance statements from IGs used by National Institute of Standards and Technology (NIST) to build validation tools. However, conformance statements must be machine computable.
• J.Roberts - Yes; at our best our IGs contain computable conformance statements. Where that is not true we need to work.
• Feb 22, 2016 TSC Call: Karen asks if anyone disagrees with the approach stated above; no one states disagreement

2. Assuming that the answer to question 1 is yes, we would likely need to come forward with a new requirement that all IGs provide these conformance statements. Where would such a statement go? Is there a document that currently defines what is required for IGs?

• F.Hall - Would include in multiple documents since we never know where a person will look and we have no definitive source of truth recognized by all. For example: GOM, Co-chair handbook, BAM (?), V2 Style Guide, V3, FHIR (?) TSC could also monitor (or require?) existing role of "Conformance Facilitator" for IG projecs, thus would prompt Work Group to plan for how to achieve conformance statements, even if they assign the WG to complete
• J.Roberts - Agree with Freida: there is no source of truth for all our standards products.
• Feb 22, 2016 TSC Call: Ken: we have not been good at tagging conformance statements in V2 and they're in different places. How do we tag them and note them elsewhere? Austin states it is in place in CDA but it's not required. Melva: not everyone is putting conformance statements in and we want it to be required. Karen: yes, we would need to have a new rule, and before that we need a guidance document for each family.

3. If we make conformance statements a requirement, how should this be rolled out? Would the TSC/Board ask each WG to update the current IGs to produce the conformance statements? If so, would this be rolled out over the course of a couple of years? While we could make certain this happens going forward, I believe our plan would need to cover the IGs that have already been published. At least those that are used frequently.

• F.Hall - Suggest follow the "Introducing New Processes to HL7" methodology to roll out. Could create expectation that all IGs need CS created, unless exception granted by TSC. That allows WGs to petition for exception and help identify what is not "used frequently".
• J.Roberts - I'd favor a "grandfather" approach: Require conformance statements to standard at reaffirmation or update time and for any Information or other non-expiring documents, set a sunset date, maybe five years?
• M. Peters - The attached document was developed as part of the Conformance Project in 2014. There are some Lessons Learned and Next Steps in this document that may be relevant. File:HL7 Conformance Lessons Learned and Next Steps - v0.3.pdf

4. Would a particular WG or WGs in HL7 be responsible for developing a document that provides guidance and direction on how to create the conformance statements? Are there other considerations? For example, to do this correctly would each committee need a conformance facilitator in the same way that we have vocabulary and modeling facilitators?

• F.Hall - I think Conformance & Guidance for Implementation/Testing Work Group (CGIT) is best candidate to develop, their mission is to "... support all conformance activities of users of the HL7 standards." They would likely need input from other groups, could be done as a peer review or comment only ballot on their proposal. On the project scope statement we already have a slot for "Conformance facilitator (for IG projects)", defined as: The Conformance Facilitator, recommended for Implementation Guide projects, is a member of the integration team with in-depth knowledge of interoperability standards and conformance/constraint rules. This individual advises the project team on the use of HL7 standards and the constraint(s) requested to support the stakeholder requirements.
• J.Roberts - CGIT.

5. Do we need to develop a program to certify the certifiers? If so, which WG within HL7 do you see being responsible for this task?

• F.Hall - Would promote consistency in approach, seems CGIT should supervise this work.
• J.Roberts - Yes but with care that we do not compete nor set up adversarial relatonships with our own members.

6. ONC would like us to suggest a way that we can deliver annually a way to report on implementation/use of our IGs. I’ve had a very brief discussion with Pew Charitable Trust on this but would like to get input from the TSC on any suggestions they have for collecting this information, including whether the standards maturity model would provide what is needed.

• J.Roberts - If we did do something along these lines it would be an opportunity to identify subjects / domains where standards are lacking and places where competing specifications are in place either in addition to or in lieu of an HL7 Interoperability standard.
• Feb 22, 2016 TSC Call: Ken: Seems like the downloaders would be our audience for a poll; they could fill out a document so we can gather more information. Lorraine: the vast majority of implementations probably don't download it from HL7 directly, they likely get it from the affiliate; this method would only provide a subset of implementers. Karen: that's why we feel like we need an organization that really knows how to poll. Ken: couldn't we ask that the affiliates also provide that information? Pat agrees that is a great idea. Austin: we may have to scope it down to a variety of US Realm products in order to make it manageable.

• Feb 22, 2016 TSC Call - Additional Discussion:
  • Austin: is this a topic that we should send to the SGB? The question of what constitutes an implementation guide and what needs to be in it? Melva notes that Duteau Design did a project a few years ago that had a best practices/lessons learned document that may be helpful to reference. Melva will post it on the wiki page. Ken: what would be the timing for SGB to take a look at this? Karen: final report is due at the end of September.
  • Lorraine: in the past we've struggled with getting volunteers with enough bandwidth to get these things all written down. Karen: one of our recommendations would be that the ONC fund the development of the guidance documents. Austin: Is this guidance intended only for US Realm, or is it universal? Would the ONC be comfortable funding something that is universal? Paul: From an SGB standpoint we can set down the principles and then they could write the documents. Austin: CGIT may be involved with the nuts and bolts but they won't have the governance part of it. Paul: CGIT could work with the paid resources.
  • Karen: would the guidance document need to be balloted? Question over ballot level. By default if an IG is normative, then that would indicate that the conformance statements are normative. Karen asks Paul and Austin through SGB to pull together guidance on what would need to be done to create the guidance document. Karen will check with the ONC on the issue of funding Universal vs. US Realm.

Summary of Feb 22, 2016 TSC call:

• Create a Guidance Doc; ballot informative or Normative (need to finalize that decision)
• ‘Grandfather planning’…conformance stmts required going forward in IGs; volunteer bandwidth can’t expeditiously add them to already published IGs
• Going forward, would conformance statements only be added to US Realm IGs, or also to Universal realm IGs? TSC prefers Universal realm IGs; is ONC okay with that?
• Link to Best Practices/lessons learned from Conformance testing project