View source for Questions for TSC Regarding the ONC Conformance Testing Project

==TSC Questions from the Initial Report Submitted to the ONC ==
On April 1, 2016, HL7 submitted a 50 page [http://gforge.hl7.org/gf/download/docmanfileversion/9157/14168/%48%4c%37%20%4f%4e%43%20%54%65%73%74%69%6e%67%20%52%65%70%6f%72%74%2e%70%64%66 Initial Report to the Office of the National Coordinator] for the Standards Development Organization Collaboration to Enhance Standards Alignment, Testing, and Measurement grant project.

Page 3 contains the list of implementation guides covered by this project.

Copied below, for the TSC's review and feedback, is Section 5 - Proposed approach:

HL7’s proposed approach at this point in time is to require that its implementation guides provide relevant conformance statements that are machine computable and robust enough to enable conformance tooling vendors to develop conformance tooling concurrent with the development of the implementation guide.  HL7 feels very strongly that it should not develop conformance testing tools internally but should provide robust conformance statements that allow tooling experts to do so. 

ONC has also requested that HL7 identify a means to collect and report implementation/usage data at the message level on an annual basis.
To achieve the desired future state HL7 will need to:

'''1. Develop a policy that requires conformance statements in all implementation guides''' <br/>
HL7 must develop, implement and enforce a policy requiring that all implementation guides balloted through the organization include relevant machine-processable conformance statements that conform to a pre-developed set of criteria. These conformance statements will likely be balloted as part of the underlying implementation guides.
* ''These open issues will be presented to the HL7 Technical Steering Committee for continued discussion and resolution: ''
** Since the HL7 community at large is not well versed in writing conformance statements, the organizations will need to decide who ballots these statements. We may need to introduce a new class of voting members (conformance tooling vendors) who vote separately but simultaneously on the statements for some period of time until the HL7 community at large develops some expertise in this area. This would ensure that the conformance statements receive sufficient review from a community of voters who are knowledgeable and have appropriate skills. It would also ensure a feedback loop from the tooling vendors to the standards developers.
** Should balloting of the conformance statements be rolled into the implementation guide ballot at some point in the future? If so, what is the appropriate time for doing this? 
** Requiring that all implementation guides contain appropriate conformance statements will likely impact the speed at which implementation guides are developed and ready for ballot.
<br/>
'''2. Develop a guidance document that defines how to develop robust conformance statement''' <br/>
HL7 will need to undertake development of a guidance document, probably authored by an external consultant with input from HL7’s Technical Steering Committee, our Conformance & Guidance for Implementation/Testing work group and the HL7 Standards Governance Board,  that defines how to develop these conformance statements and the required content/format, etc. The HL7 Technical Steering Committee recommends that this guide be written by a hired consultant who will be tasked with researching best practices, other industry literature and work with the HL7 groups noted above to ensure that the guidance reflects that knowledge and is suitable for HL7 standards.
* ''Issues related to this item:''
** Which internal bodies will have input on this document and oversee its development.
** Which internal body (e.g. the Technical Steering Committee, the entire HL7 Community, etc) will approve this document and how that will be achieved.
** HL7 will need to secure funding (whether from the HL7 budget or an external funder) and develop 
<br/>
'''3. Train work groups to develop the conformance statements''' <br/>
Obviously, HL7 will need to develop and provide free training targeted to the HL7 work groups on the process and criteria involved in developing conformance statements. While we have a few conformance tooling vendors that participate in HL7, we do not perceive wide-spread expertise in this area among the current HL7 community. Training will need to be ongoing for some period of time, probably at least two years. Training can be offered via webinar (both live and recorded) and at the HL7 Working Group Meetings. While we envision introducing a conformance facilitator role to the organization that can lead conformance statement development within the work groups, the general membership does need some education on the purpose and importance of conformance statements and a cursory understanding of how to develop them.
* ''Issues related to this item:''
** Do we need multiple levels of training (e.g., one level for general membership and a more rigorous program for conformance facilitators).
** HL7 will need to secure funding (either from the HL7 budget or an external funder) to develop the training.
<br/>
'''4. Recruit conformance facilitators'''<br/>
Requiring a conformance facilitator for each implementation guide,  much like the current requirement for vocabulary, modeling and publishing facilitators for other projects, will ensure development of cadre of volunteers well versed in the development of conformance statements and that this skill  becomes ingrained in all work groups. It is noteworthy that the HL7 project scope statement already provides a conformance facilitator field for implementation guide projects; obviously at some time the organization was clearly thinking about developing conformance statement skills more broadly across the organization. This role is currently defined as follows: “The Conformance Facilitator, recommended for Implementation Guide projects, is a member of the integration team with in-depth knowledge of interoperability standards and conformance/constraint rules. This individual advises the project team on the use of HL7 standards and the constraint(s) requested to support the stakeholder requirements”. This above statement would need to change “recommendation” to a “requirement”. 
* ''Issues related to this item:''
** Do we have enough volunteers to develop the role fully
** Who would be in charge of the conformance facilitator development program
** Will development of this program require financial resources beyond those needed for the training discussed above
<br/>
'''5. Develop a plan for creating conformance statements for all or some subset of extant implementation guides for which there is not conformance tooling.'''<br/>
One of the most critical steps will be the development and rollout of a plan for creating conformance statements for all or some set of the implementation guides that currently have no conformance tooling support. The Technical Steering Committee will be tasked with making a recommendation to be approved by the HL7 Board regarding how this will be implemented. There are several choices:
# Require that all implementation guides without conformance tooling be updated within some period of time to produce the needed conformance statements and related tooling. This may be an unpopular decision within the work groups and may require that the work be done for hire. Given that 34 implementation guides fall into this category, it is unlikely this would be the decision.
# Require that only come implementation guides be updated within some period of time to produce the needed conformance statement and related tooling. This may be an unpopular decision within the work groups and may require that the work be done for hire.
# Require that no implementation guides without conformance tooling be updated to produce the needed conformance statements and related tooling but do require that all implementation guides going forward produce the required conformance statements. 
* ''Issues related to this item: ''
**Requiring work groups to update extant implementation guides may be unpopular as work within HL7 is typically generated bottom up vs. top down.
**Given the bullet above, updating extant implementation guides may need to be hired out. Should that be the case, HL7 will need to secure funding, either internally or from an external source, to complete this work.
<br/>
'''6. Develop a certification program to certify conformance tools and/or tooling vendors'''<br/>
Much like programs that certify MU tools, HL7 needs a program whereby we certify various tools that adhere to criteria that we establish. Work needed to create this program would include development of a practical guide for conformance tooling vendors on expected format and output for tools. We could develop these to be aligned to the ONC Health IT Certification Program test methods. Much more additional work needs to be undertaken by HL7 in this area to fully understand the work involved. 
* ''Issues related to this item:''
**Very little thinking has been done to date on how to implement this program. We sense this will involve a fair amount of effort and financial resources
**HL7 will need to secure funding, either internally or from an external source, to complete this work
**Although likely appreciated by implementers, certifying tools or tooling vendors may not be a popular decision among the tooling vendors 
<br/>
'''7. Develop and maintain a tooling catalog <br/>
As recommended by the contractor hired to complete the gap analysis, HL7 implementers would greatly appreciate a tooling catalog that would serve as an authoritative, central index of conformance tools. This would provide a mechanism for tool developers to be advised of the need for new tools, to allow implementers to flag issues with tools (out of date, errors, etc) and even rate them to steer implementers to the best tools for any given implementation guide.
* ''Issues related to this item:''
** While much of the analysis for the catalog has been completed by the contractor hired to undertake the tools gap analysis, there is still a significant amount of work to be completed to get the catalog online.
** HL7 has a webmaster and other technical resources that could undertake the needed work but given other responsibilities, the timeframe for completing this work is currently unknown.
** There are a number of issues around maintaining the catalog and keeping it current. It is unclear whether this work would be done by internal HL7 resources or outsourced.
** If the catalog is to be developed and maintained by an outside resource, HL7 will need to find funding, either internally or externally, to complete the work.
<br/>
'''8. Continue efforts to create examples (CDA) and models (CIMI) for representing data'''<br/>
Efforts by the Structured Documents Examples task force and the CIMI work group should be continued. The organization should consider merging and expanding these efforts. 
<br/><br/>
'''9. Identify method to collect and report implementation/usage metrics annually'''<br/>
ONC has also requested as part of this grant that HL7 identify a method to accurately collect and report implementation/usage metrics annually. Our understanding of the request is that ONC would like the data collected and reported on a transaction (message) basis.
This is one of the most difficult aspects of this grant as there is no obvious source for which to collect and report the requested data. HL7 tracks the number of times an implementation guide is downloaded from its website. While this data was used as part of the gap analysis (to determine popularity of a particular IG) it is not scientific (e.g., anyone can download the same implementation guide multiple times and our count mechanism doesn’t account for that; all facilities that implement/use an implementation guide may not secure the implementation guide directly from the HL7 website, etc).  For some transactions, vendors may be able to supply part or all of the data. For other transactions, the end user may be able to supply part or all of the data. Both sources would have some of the data. Thus, getting the data and structuring a methodology to interpret it (as some of the data might be reported by both the vendor and the end user) is difficult.
HL7 has done some work with the Pew Charitable Fund over the last several months and we approached them to determine if their polling arm would be interested in participating in a project to undertake this work. Unfortunately, Pew is not able to participate in projects that are funded by the US government. They did, however, suggest a few polling agencies that they have worked with in the past that might be interested, including:
*Hart Research
*Public Opinion Strategies
There are a number of other organizations that could assist in the development of a poll and the affiliated methodology for collecting data and interpreting the results, including organizations such as Westat, which maintains an organizational membership within HL7. 
The HL7 Technical Steering Committee has had some discussion around this issue but has not made any progress in identifying a means or methodology for getting the desired data. HL7’s Orders and Observations work group and the FHIR work groups (at least to some extent) are working on a standards maturity model that attempts to get a sense of how widely the specifications are implemented/used. While interesting, the standards maturity model will not provide the needed information.
Even if we could find a reliable source to provide the data, reporting that would likely require a significant amount of time and effort on the part of the organization responsible for reporting the data.
HL7 will need to do additional research to determine possible methods for collecting the desired data. As of the writing of this report, HL7 will commit to contacting Hart Research, Public Opinion Strategies and Westat to determine if they believe this is something they could and would be willing to undertake and what the timeline/cost might be.
Optionally, the government could enact a policy requiring vendors and/or providers to report this data annually and report that data on a public website.
Regardless of which method is selected, HL7 is of the opinion that it does not have the expertise to collect the data and would therefore recommend that the government either outsource the project or fund HL7 to oversee a project that it outsourced to a qualified entity.

==Original Questions to the TSC ==

1. The general approach that was discussed several months ago with the TSC chair, CTO and staff was that HL7 should not be in the business of creating conformance testing. Rather, we should  create conformance satements as part of our IG that are then rolled up and distributed to external toolsmiths capable of creating conformance testing tools (probably at the same time that the standards are created).  While my statement of the process is likely over simplified, does the TSC agree that this is the preferred approach?
*F.Hall - yes, and this has worked for the US Realm Lab IGs, conformance statements from IGs used by National Institute of Standards and Technology (NIST) to build validation tools.  However, conformance statements must be machine computable.
*J.Roberts - Yes; at our best our IGs contain computable conformance statements.  Where that is not true we need to work.
*Feb 22, 2016 TSC Call: Karen asks if anyone disagrees with the approach stated above; no one states disagreement

2. Assuming that the answer to question 1 is yes,  we would likely need to come forward with a new requirement that all IGs provide these conformance statements. Where would such a statement go? Is there a document that currently defines what is required for IGs?
*F.Hall - Would include in multiple documents since we never know where a person will look and we have no definitive source of truth recognized by all.  For example:  GOM, Co-chair handbook, BAM (?), V2 Style Guide, V3, FHIR (?)  TSC could also monitor (or require?) existing role of "Conformance Facilitator" for IG projecs, thus would prompt Work Group to plan for how to achieve conformance statements, even if they assign the WG to complete
*J.Roberts - Agree with Freida: there is no source of truth for all our standards products.
*Feb 22, 2016 TSC Call: Ken: we have not been good at tagging conformance statements in V2 and they're in different places. How do we tag them and note them elsewhere? Austin states it is in place in CDA but it's not required. Melva: not everyone is putting conformance statements in and we want it to be required. Karen: yes, we would need to have a new rule, and before that we need a guidance document for each family. 

3. If we make conformance statements a requirement, how should this be rolled out? Would the TSC/Board ask each WG to update the current IGs to produce the conformance statements? If so, would this be rolled out over the course of a couple of years? While we could make certain this happens going forward, I believe our plan would need to cover the IGs that have already been published. At least those that are used frequently.
*F.Hall - Suggest follow the "Introducing New Processes to HL7" methodology to roll out.  Could create expectation that all IGs need CS created, unless exception granted by TSC. That allows WGs to petition for exception and help identify what is not "used frequently".
*J.Roberts - I'd favor a "grandfather" approach: Require conformance statements to standard at reaffirmation or update time and for any Information or other non-expiring documents, set a sunset date, maybe five years?
* M. Peters - The attached document was developed as part of the Conformance Project in 2014.  There are some Lessons Learned and Next Steps in this document that may be relevant. [[File:HL7 Conformance Lessons Learned and Next Steps - v0.3.pdf]]

4. Would a particular WG or WGs in HL7 be responsible for developing a document that provides guidance and direction on how to create the conformance statements? Are there other considerations? For example, to do this correctly would each committee need a conformance facilitator in the same way that we have vocabulary and modeling facilitators?
*F.Hall - I think Conformance & Guidance for Implementation/Testing Work Group (CGIT) is best candidate to develop, their mission is to "... support all conformance activities of users of the HL7 standards." They would likely need input from other groups, could be done as a peer review or comment only ballot on their proposal.  On the project scope statement we already have a slot for "Conformance facilitator (for IG projects)", defined as: The Conformance Facilitator, recommended for Implementation Guide projects, is a member of the integration team with in-depth knowledge of interoperability standards and conformance/constraint rules. This individual advises the project team on the use of HL7 standards and the constraint(s) requested to support the stakeholder requirements. 
*J.Roberts - CGIT.

5. Do we need to develop a program to certify the certifiers? If so, which WG within HL7 do you see being responsible for this task?
*F.Hall - Would promote consistency in approach, seems CGIT should supervise this work.
*J.Roberts - Yes but with care that we do not compete nor set up adversarial relatonships with our own members. 

6. ONC would like us to suggest a way that we can deliver annually a way to report on implementation/use of our IGs. I’ve had a very brief discussion with Pew Charitable Trust on this but would like to get input from the TSC on any suggestions they have for collecting this information, including whether the standards maturity model would provide what is needed.
*J.Roberts - If we did do something along these lines it would be an opportunity to identify subjects / domains where standards are lacking and places where competing specifications are in place either in addition to or in lieu of an HL7 Interoperability standard.
*Feb 22, 2016 TSC Call: Ken: Seems like the downloaders would be our audience for a poll; they could fill out a document so we can gather more information. Lorraine: the vast majority of implementations probably don't download it from HL7 directly, they likely get it from the affiliate; this method would only provide a subset of implementers. Karen: that's why we feel like we need an organization that really knows how to poll. Ken: couldn't we ask that the affiliates also provide that information? Pat agrees that is a great idea. Austin: we may have to scope it down to a variety of US Realm products in order to make it manageable.


*Feb 22, 2016 TSC Call - Additional Discussion:
**Austin: is this a topic that we should send to the SGB? The question of what constitutes an implementation guide and what needs to be in it? Melva notes that Duteau Design did a project a few years ago that had a best practices/lessons learned document that may be helpful to reference. Melva will post it on the wiki page. Ken: what would be the timing for SGB to take a look at this? Karen: final report is due at the end of September. 
**Lorraine: in the past we've struggled with getting volunteers with enough bandwidth to get these things all written down. Karen: one of our recommendations would be that the ONC fund the development of the guidance documents. Austin: Is this guidance intended only for US Realm, or is it universal? Would the ONC be comfortable funding something that is universal? Paul: From an SGB standpoint we can set down the principles and then they could write the documents. Austin: CGIT may be involved with the nuts and bolts but they won't have the governance part of it. Paul: CGIT could work with the paid resources. 
**Karen: would the guidance document need to be balloted? Question over ballot level. By default if an IG is normative, then that would indicate that the conformance statements are normative. Karen asks Paul and Austin through SGB to pull together guidance on what would need to be done to create the guidance document. Karen will check with the ONC on the issue of funding Universal vs. US Realm. 



Summary of Feb 22, 2016 TSC call: 
*Create a Guidance Doc; ballot informative or Normative (need to finalize that decision)
*‘Grandfather planning’…conformance stmts required going forward in IGs; volunteer bandwidth can’t expeditiously add them to already published IGs
*Going forward, would conformance statements only be added to US Realm IGs, or also to Universal realm IGs?  TSC prefers Universal realm IGs; is ONC okay with that?
*Link to Best Practices/lessons learned from Conformance testing project