Versioning and Backwards Compatibility in V3

Notes for V3 Task Force on Data Types and Backwards Compatibility

Version 3 Principles on Compatibility

Document balloted in HL7 in late 90's to express the intended principles for V3. The principles are published as section 5.2 of the Notes to Readers in each V3 Ballot.

Upward Compatibility Among V3

HL7 will provide the maximum degree possible of interoperability among systems operating on older and newer versions of HL7 protocols in the V3 family through compatible enhancement.

• A message structure that is modified in a newer version of the protocol must be acceptable to a system designed for the prior V3 release. However, a system built for the prior release will only extract the information that was defined for that prior release.

• A message structure created in accordance with an older version of the V3 protocol must be acceptable to a system designed for a later V3 release. In some cases, this will mean that the system built for the newer release will not receive certain information fields because they were not a part of the older version of the message structure in use by a specific sender.

• Where compatible enhancement is not possible, HL7 will require evolution in the protocol to happen gradually, so that users can introduce the change into their networks gradually.

• The messages associated with all interactions that are newly defined in a version of HL7 SHALL NOT be sent to a receiver that conforms to the older version.

• A message structure MAY be declared obsolescent in one release, with a stated alternative message structure. Both the obsolescent message structure and its alternative can be used by all systems supporting that release.
  • WoodyBeeler 15:56, 23 October 2012 (UTC) The second sentence above begs a number of questions: (i) Did it really men "Both" can be used or should it be "either" (one or the other)? (ii) If "both" does this mean systems MUST send both or that they MAY send both? (iii) Should there be a requirement to provide a transform (in the case of XML) to convert "obsolescent" to "alternative"?

• The obsolescent message structure MAY be declared obsolete and dropped when still another HL7 version is issued.

• An obsolescent message structure SHALL NOT be declared obsolete for at least two years from the date of the version that first declared it obsolescent.

The above notwithstanding, if a new Implementation Technology Specification (ITS) is introduced, HL7 may specify that conformance to the ITS does not require dealing with message structures that are obsolescent when the new ITS is introduced. To the maximum degree possible, these restrictions should not impose limitations on the evolution of the overall reference model. There are no restrictions on making changes to the information model if those changes do not impact the message structures that were described in a prior version of the Standard.

Actual "Versioning" in V3

Subsequent to the V3 Principles approval, HL7 undertook the development of V3 specifications. The strategy that was followed was to seek "parallel" development and balloting of the V3 standards. Under this strategy, each of the involved Work Groups would develop and seek approval for its own "domain" content, and the combined effort to become the "V3 Messaging Specifications". Subsequent to the first approvals of V3 specifications, the Modeling and Methodology Work Group evolved the notion of publication packages known as editions. In essence, each ballot would constitute an "implementation edition", and an annual "Normative Edition" would be produced as the formal specification which implementers were expected to use.

Since 2005, there have been seven "Normative Editions" published by HL7. Each has been self consistent. The expressed intent from Modeling and Methodology has been that implementations should be made from content drawn solely from a single Normative Edition. Thus, implicitly, the transition from one Normative Edition to a subsequent Edition might be "non--compatible". In effect, the earlier Normative Edition contains the "obsolescent" messages, and the later Normative Edition is the "newly defined" set of messages.

To be clear, this was never expressed relative to the obsolescence principles cited above.

Implementation of "Deprecation" in the Reference Information Model

In order to support the concepts embodied in the V3 Principles, including the notion of "obsolescence", the Harmonization process, which manages the HL7 Reference Information Model and its associated vocabulary, adopted the notion of "Deprecation" to support the gradual adoption of "breaking" changes in the RIM.

Whenever a "breaking" change is proposed, such as the removal of an attribute, Harmonization adopted with the following steps and expectations:

1. the element to be removed would be listed as "Deprecated" as of a particular date and release of the RIM

2. the deprecation notations would contain a statement as to how to represent the "semantics" of the deprecated element in future models. (This might include identification of a new, replacement element, or of an alternate design structure that would express the same thing.)

3. Effective with the date of deprecation, the selected element was not to be used in the development of any new models. (To be clear, the meaning of "new models" was never fully explored. For example if a model was developed prior to deprecation, and was subsequently (after deprecation) modified within the context of a second ballot, did this constitute a new model?)

4. Subsequent to deprecation, the element was to remain part of the RIM for at least two Normative Edition. Subsequent to that time, the deprecated element could be removed (deleted) from the RIM.

Semantic Compatibility in Normative Editions

As each Normative Edition is published, the content within that edition is updated to the "current" RIM and Data Types, so long as the changes are semantically equivalent.

The latter distinction was only important in the conversion from data types release one to data types release two( for Normative Edition 2012). In that circumstance, the intention to make these changes was formally announced to the cochairs meeting had several Working Group Meetings, and formally expressed in published announcements to the cochairs. Indeed the actual conversion occurred 20 months after the first announcement, and almost 4 years after the beginning of the development of Data Types Release 2.

One consequence of the management of Normative Edition content has been that several Normative Editions include "deprecated" elements (both attributes and vocabulary terms) that, in effect, form and "obsolescent" message structure.

Implications of ANSI Version Expectations

Recently, HL7 has begun to stumble over the ANSI expectations for versioning. Specifically, ANSI expects each specification to be reviewed at the end of five years, at which time it should either be reaffirmed for five more years or updated. At the end of 10 years, the specification must be updated.

While several Work Groups have followed this pattern, a number of others have not. As a consequence, most of the "common" message content in V3 (with the exception of CMETs) has passed the time at which could be reaffirmed. This includes: messaging wrappers, query wrappers, registry wrappers, control act wrappers, and common messages.

One solution to this problem would be to re--ballot the original release one specifications, but with subsequently deprecated content replaced by its semantic equivalent. This material could become the "obsolescent" specification, while the Infrastructure and Messaging Work Group undertakes to develop the release two content.

Recasting "Draft motion" per Prior Task Force Discussions

Draft Motion (Hugh):

• HL7 has not and will not guarantee wire format backward compatibility of V3 payloads as the standard evolves. However HL7 recognises that in some circumstances breaking wire format backward compatibility will cause significant difficulty to users and that this in turn will reflect badly on HL7 and have a substantial negative impact on the adoption of V3. In such circumstances, and where a resolution can be found, HL7 will publish multiple versions of affected schemas so that both new versions and old wire format backward compatible versions are available to the user community that has been affected.

• Discussion points:
  • The reason for trying to provide a backward compatible version is HL7 self-interest – we lose credibility if we don’t. We are not committing to satisfy all backward compatibility requirements everywhere.
  • We are only committing to provide a solution where one can be found.
  • We are not committing to providing the solution to everyone – but only to the group that has the issue.

Motion for Consideration

Whereas:

• HL7 Version 3 is, by design a standard that must be able to evolve as new requirements and new technical solutions arise. Therefore the "V3 Principles", adopted when HL7 undertook Version 3, provide for semantic backward compatibility, but can not guarantee wire format backward compatibility of V3 payloads if the standard is to evolve. At the same time, however, HL7 recognizes that in some circumstances the breaking of "wire format backward compatibility" will cause significant difficulty to some communities of users; that this, in turn, will reflect badly on HL7; and therefore will have a substantial negative impact on the adoption of these standards.

• In such circumstances, when a viable alternate solution can be found, this Task Force believes HL7 has an obligation to provide a reasonable "migration path" for the affected communities that have implemented the prior standards. Such a migration path should be formally adopted as an alternate standard(s) whose scope of intended use is defined, and whose intended period of expected use is predetermined in order to allow the using community to complete the migration to the "universal" standard(s).

• The Task Force recognizes that the formal adoption of HL7 Data Types Release 2, created such a breaking change for a community of Version 3 Messaging users that had adopted specifications predicated upon backwards compatibility of the XML schemas for the messages. Principle among these are those specifications in the domain of Regulated Products centering around the HL7 Version 3 Standard: Structured Product Labeling specifications. These groups have determined that a pair of in-ballot specifications -- HL7 Version 3 Standard: XML Implementation Technology Specification - V3 Structures for Wire Format Compatible Release 1 Data Types, Release 1 and HL7 Version 3 Standard: XML Implementation Technology Specification - Wire Format Compatible Release 1 Data Types, Release 1-- (commonly referred to as "Data Types R2B"), which while not fully backwards compatible with Data Types R1, do afford the degree of wire-format backwards compatibility needed for their endeavors.

Therefore: This task force recommends to the TSC that the TSC:

1. Facilitate the prompt adoption of the Data Types R2B specifications as Normative specifications for use as the intended "migration path" for identified communities;

2. The initial identified community is those users in Regulated Products that have implemented specifications dependent upon consistent XML schema representations of Structured Product Label;

3. Set an expected "sunset" for this specification at five-years after registration as an ANSI specification;

4. Instruct the Publishing Work Group to work with the affected community to devise a publication suite that reflects the "migration" content and distinguishes it from the more general "universal" content.

5. The TSC might approve further communities of use, if the case therefore is presented, and if a constrained scope can be established;

6. Should similar concerns arise from future changes to the Version 3 standards, the TSC should carefully consider:
   • the potential for a formally defined migration path;

   • the ability to limit the scope and time-frame for the solution;

   • The inability of the affected community to freeze their adoption at a particular Normative Edition; and

   • burden on the HL7 Working Group to sustain dual paths.

Furthermore:

• The Task Force, while considering backwards compatibility, whether wire-format or semantic, recognizes that DSTUs, by their nature as Draft standards, are intended to allow change when initial trials reveal weaknesses. Therefore DSTUs should never be held to a backward compatibility restriction.
  • As a corollary, if the community of implementers intends to seek backwards compatibility (based on backwards compatibility rules for that product line) to the first formally adopted version of a specification, then that first formally adopted release must be balloted as Normative, not DSTU.

The Task Force recommends that the TSC formally adopt this position, include these principles as a reminder on the Project Scope Statement and Notice of Intent to Ballot, add as a pro-forma notice on DSTU ballots and publication requests, and ensure that the HL7 membership understands these intents.