• Print

Proposed Interoperability Standards Measurement Framework Public Comments


Proposed Interoperability Standards Measurement Framework Public Comments

The comment period for this document ended on 5 p.m. ET on Monday, July 31, 2017. Comments for this document will be posted shortly.

The document solicits feedback on a proposed industry-wide measurement framework for assessing the implementation and use of health care interoperability standards. Currently, stakeholders’ capabilities to measure and report on the use of standards vary significantly across the health IT ecosystem. This framework aims to help health IT developers, health information exchange organizations, and health care providers move towards a set of uniform measures to assess interoperability progress.

Please note that comments and recommendations submitted as part of this process will be made public by ONC.

 

Tony Mallia
Wednesday, July 12, 2017 - 10:07
We all talk about interoperability – how interoperable systems are beneficial and how networks can be assembled from interoperating systems but in many cases we do not know when we have reached interoperability. That is partly because there appears to be no criteria against which we can judge achievement.
It has become apparent that there can be impediments to achieving interoperability in the specifications or standards themselves. Just because a specification is a standard does not guarantee interoperability. It is left to the implementers after the specification has been released to find the interoperability issues with the specification. The measurement framework is addressing the collection of this experience. This is late in the standard development process and may result in an inadequate specification being adopted for implementation by a community. We need a preventative approach to avoiding the deployment of non-interoperable specifications.
Another measurement in the framework should be the assessment of the standard against interoperability criteria developed in a Theory of Interoperability. This can be used as a quality mechanism against a draft standard to assess whether it will result in interoperable systems.
Initial work on the Theory is intended to be understandable by specification developers and does not cover the math or science of interoperability.

rhys giron-Delahay
Tuesday, July 11, 2017 - 17:26
Any framework and standard guidelines are important
Karen Green
Tuesday, July 11, 2017 - 15:28
My organization represents Post Acute, and we are struggling with interoperability with transitions in care from acute. We are able to receive CCD's, but the acute care side is not sending them very timely or with the content we need. Meaningful use has established some good format and content standards for communicating patient information, and this is measured by quantity of transactions. However, but there is no measure of quality of the content or timeliness in applying the standard. I think lack of timeliness is the biggest barrier to adoption of standards. APIs and FHIR help to facilitate timelier exchange of information, but the triggers for these mechanisms have not been clearly defined so that the information flow is more timely. Post Acute who is at the tail end of the interoperability food chain, does not get electronic information soon enough to determine if there is medical necessity. We have to access EHR's or visit the patient face to face to get the information we need. We then assess whether the patient should go IRF, SNF, OP, or homecare. This takes several people and processes to accomplish. Interoperability should focus on improving transition in care, and allow for digital rounds to do a clinical assessment and support a clinical decision as to where the patient needs to go. The efficacy of standards being applied can be measured by the time it takes to transition a patient, reduce LOS, or communicate the information that it takes to navigate the patient to the right care setting. The number of days it takes to receive a document under the CCD architecture, by HISP or HIE, should be -1 date from discharge date. I would like to see # of days as a metric that would help foster adoption if a receiving organization could trust the standard CCD was timely and useful. Getting a CCD 2-3 days after discharge renders the information "meaningful useless', especially when we are trying to identify high risk patients that need extra attention as they come to our care settings.

Ryan Joyce
Friday, June 30, 2017 - 12:30
TBD

Wednesday, June 28, 2017 - 14:15

Page 6 b, “particular versions of their products or services that are not already derivable from the ONC Certified Health IT Product List (CHPL)4.”
Why would there be a gap between standard in system and listed in CHPL, is this to track a certified release version / date list versus what may have been added during service updates over time within a specific release?

Page 6 c, “Product version with standard implemented deployed to end users: Health IT developers should publicly report how many end users have deployed a product version or subscribed to a service with a standard implemented.”
Please clarify Client IDs will not be available in the data collection, this information could be miss used in a competitive market and reduce quality of data retrieved.

Page 6 Objective 2, “could result from a number of factors, including, but not limited to, insufficient education on how to use the functionality enabled by the standard, difficulty finding or using the functionality, or lack of other users with whom to exchange.”
Other factors, purposeful consumer administrative choice to delay, expense to consumer in updating, and training of system users. We frequently see clients upgrading but upgrading Like for Like and slowly implementing components over time. Providers have established workflows and behavior, changing system behavior or adding new requirements is disruptive. From the perspective of the consumer it would be valuable to determine what they have implemented, what they have not (and why). What is their implementation timeline for standards not in use. Understanding the context of use/non-use is critical. Do consumers see functional, clinical value in features/standards beyond the checking a box?

Page 7 a, “Standard used by end user: Health IT developers and exchange networks should publicly report what percentage of end users (who have access to a product version with a standard implemented) have actually used a particular standard.”
I contend IT Vendors have devoted considerable resources in meeting standards. There is valuable in knowing consumer’s perspective on workflow changes that have come to pass based on standards. Consumers belief in how they have been impacted, positively and negatively. What features they have sought that have been delayed/not delivered r/t meeting standards. What standards have added value to the provision of care and what standards are seen as no or negative impact to provision of care.


Page 7 a, “This will enable stakeholders to identify standards that have been deployed widely but not used by end users.”
End users and administration. We have seen Hospital IT and Administration control what information is available about features. E.g. Data Export. When introducing development underway, the best response from a client provider, "Why would you do that?" Not all features are deemed valuable and information about features is withheld from the end user.

Page 7 c, “Level of conformance/customization of interoperability standards: Stakeholders have limited experience to date in measuring this area. As a result, additional foundational work is essential to identify the best approach(es) to track the conformance and customization of standards after implementation in the field. ONC requests stakeholder feedback on the best methods to measure this area.”
Consumer list of implemented features, identify those that they have requested vendor enhancements (Provided or not). Bonus would be knowing what the enhancements were. List un-implemented features, identify if unimplemented r/t need for required functional enhancements. From a provider’s view, there is only one right way to do X. My Way. An overstatement but it is frequently true.

Page 8, table1 Obj 2, measurement a, “Standard used by end users in deployed systems”
What is the level of acceptance of required standard functionality is seen as useful/valuable?



• Is a voluntary, industry-based measure reporting system the best means to implement this framework? What barriers might exist to a voluntary, industry-based measure reporting system, and what mechanisms or approaches could be considered to maximize this system’s value to stakeholders?
o Voluntary system will not net a true picture. The response will be low from Vendors with poor implementation and likely low from consumers who have poor implementation. If Mandatory reporting compliance from IT vendors and consumers were required what would it be tied to, to insure compliance? What benefit would the system consumer receive for what would be one more reporting requirement that given the nature of the information could be cumbersome to collect (end user provider information).
• Does the proposed measurement framework include the correct set of objectives, goals, and measurement areas to inform progress on whether the technical requirements are in place to support interoperability?
o Implementation compliance may be linked to the perspective that standards requires workflow/data capture as having low value in clinical context of patient care. Collect the providers view or standards impact to care.
• Would health IT developers, exchange networks, or other organizations who are data holders be able to monitor the implementation and use of measures outlined in the report? If not, what challenges might they face in developing and reporting on these measures?
o Concern r/t system monitoring, does monitored information impact PHI that could become available to monitoring. Does monitoring become invasive to the consumer? Instrumentation of software for monitoring can have a system impact.

• Ideally, the implementation and use of interoperability standards could be reported on an annual basis in order to inform the Interoperability Standards Advisory (ISA), which publishes a reference edition annually. Is reporting on the implementation and/or use of interoperability standards on an annual basis feasible? If not, what potential challenges exist to reporting annually? What would be a more viable frequency of measurement given these considerations?
o Consider the industry impact to vendors and consumers head count required for data collection/management. What value will either party derive from the exercise?
• Given that it will likely not be possible to apply the measurement framework to all available standards, what processes should be put in place to determine the standards that should be monitored?
o Standards required for MU certification. Of those which standards are critical to the objectives of interoperability and which have a positive impact on patient care and outcome.

Sheree Speakman
Tuesday, June 27, 2017 - 14:32
Haven't read it yet.
Rose Miller
Tuesday, June 27, 2017 - 13:16
Having a highly technical communication tool is immeasurably valuable- and useful in thousands of existing benchmark quantitative statements and scenarios. However the brutal reality of transferring the costs of initiating and maintaining EHR, subsequent interoperability and then having an inaccessibility experience with personal medical record retrieval is shocking! The technology implementation cost has just contributed to enormous ##ACA $$, is now considered basic essential as working class with > 75,000 on duty dedicated hours employment seeking basic quality care, the costs of care are beyond daunting even insurmountable!

Steven Waldren
Friday, June 23, 2017 - 10:22
Introduction:
Interoperability is critical to successfully achieving the Quadruple AIM and ensuring the success of value-based payment models. We appreciate the work of ONC to drive interoperability and we urge ONC to continue to prioritize this work. We are highly supportive of measuring interoperability throughout the health care ecosystem as we cannot improve what we do not measure. We also support a far greater level of transparency in the market place especially among health IT developers and health care systems. At the same time, we have ample concern that physicians and their practices will have a disproportionate share of the burden in measuring interoperability. We strongly urge ONC to not place more administrative burdens on physicians, as we already have significant burdens due to the lack of interoperability.

We appreciate the opportunity to provide ONC with our thoughts on some of the key questions laid out in the proposed interoperability standards measurement framework.

1) Is a voluntary, industry-based measure reporting system the best means to implement this framework? What barriers might exist to a voluntary, industry-based measure reporting system, and what mechanisms or approaches could be considered to maximize this system’s value to stakeholders?
A voluntary reporting system would ensure that there is alignment with the value to those reporting, as if there is not value, they will not report. This of course means that the reporting system must provide more value than the effort required to report, which may limit reporting. We believe that an incentive based (voluntary) reporting is more likely to get us to the end desired state albeit at maybe a slower rate than a mandatory reporting system. Physicians are already over burdened with administrative requirements and lack of interoperability; let us not add additional mandated reporting burdens.
2) What other alternative mechanisms to reporting on the measurement framework should be considered (for example, ONC partnering with industry on an annual survey)?
A multi-faceted approach is most likely needed given the current state of interoperability and the industry’s nascent ability to measure it. An annual survey would be a useful facet to gauge the general level of interoperability. We also believe that complaint or end-user issue reporting can also be useful to categorize the types of issues that are being encountered. This would help inform the construction of the annual survey. In addition, the complaint/issue reporting could also help surface industry entities that may be information blocking or having a specific issue with interoperability. Finally, an ability to audit implementations as such issues surface would be warranted to determine appropriate actions. This can be augmented with measures of transaction volume, both in volume of transactions and variety (message types and sender/receiver type) of transactions.
3) Does the proposed measurement framework include the correct set of objectives, goals, and measurement areas to inform progress on whether the technical requirements are in place to support interoperability?
If one uses the flow of information use as (1) data are made available for exchange, (2) they are exchanged, (3) they are received, (4) they are interpretable, and finally (5) they are usable, then the framework is incomplete to measure wide-scale interoperability. We have heard from many family physicians that are able to receive data into their practice, and the data are interpretable from the standpoint of showing the exchanged data on the screen (i.e. display a C-CDA with an XSL) but the data within the C-CDA is not usable from the standpoint of their EHR. The physician and/or staff are required to manually integrate the new data into the patient record. We would argue that if the exchanged information is not usable by the receiving health IT system to support integration into a patient’s record, then we have yet to achieve interoperability; we have only achieve health information exchange.
4) What, if any gaps, exist in the proposed measurement framework?
As stated in the answer to question three, interoperability is only achieved when the exchanged information is usable by the receiving system and ultimately the user. We have concerns that a gap exists in the current proposed framework, as the usability of the exchanged information is not addressed. Additionally, there is a gap in the measurement of the impact of the standard and its implementation on usability.

We would urge ONC to consider adding a component to the framework that expressly deals with the usability of standards-based exchange of information. ONC should consider a methodology including focus groups of clinicians (and other end-users as appropriate) to qualitatively understand the usability issues around standards-based information exchange. ONC could then use those findings to establish a more quantitative survey instrument or other measurement tool to further measure usability and the impact of standards on usability. The findings from these qualitative and quantitative measures could inform the work of the Interoperability Standard Advisory Taskforce. They also should be used to help direct the work of ONC on driving usable interoperability.

6) Would health IT developers, exchange networks, or other organizations who are data holders be able to monitor the implementation and use of measures outlined in the report? If not, what challenges might they face in developing and reporting on these measures?
If such monitoring cannot be done, we have concern that like interoperability itself the burden will be passed down to the physician and their staff. ONC should be extremely cognizant of the potential administrative burdens placed on physicians if they are the entities that must monitor and measure interoperability.

It is high time vendors are required to report on something, be held accountable for accuracy of reporting, and be held accountable for creation of highly usable products that support natural clinician workflows – or else face the associated repercussions that may now unfold tied to a product which was not developed with user-centered design principles that results in low usability of functionality that has been developed with bare bones efforts strictly to meet certification requirements. We caution, however, that should reporting be implemented in a mandatory manner rather than via a voluntary or incentivized voluntary basis, ONC should be cognizant of the fact that onerous reporting requirements that exceedingly increase costs to HIT vendors and health information exchange organizations will result in increased costs being passed on to physicians.

For measure category a) standard used by end user, this absolutely should be monitored and reported by health IT vendors (not clinicians). There should be a means of ensuring integrity and accuracy within this reporting – in a similar way to how ONC has required QCDRs to have data validation plans to monitor and ensure accuracy of data, upon which ONC bases assessments which impact incentives earned within the EHR incentive program. We would strongly recommend that ONC consider conflicts of interest of entities monitoring and measuring their own interoperability. Therefore, ONC should consider a process to verify the measurement, such as random, routine audits of developers and networks.

7) Ideally, the implementation and use of interoperability standards could be reported on an annual basis in order to inform the Interoperability Standards Advisory (ISA), which publishes a reference edition annually. Is reporting on the implementation and/or use of interoperability standards on an annual basis feasible? If not, what potential challenges exist to reporting annually? What would be a more viable frequency of measurement given these considerations?
We would believe that an annual reporting would be sufficient and feasible.

8) Given that it will likely not be possible to apply the measurement framework to all available standards, what processes should be put in place to determine the standards that should be monitored?
We would recommend using the standards that support the key data that has shown to impact the quadruple AIM either in a negative way when not available or in a positive way when available. If such empirical evidence were insufficient to complete the prioritization, then we would recommend that a group of clinicians be convened to complete the prioritization process.
10) What measures should be used to track the level of “conformance” with or customization of standards after implementation in the field?
We have some serious concern about the use of the phrase “customization of standards after implementation.” Customization of a standard should only be done by its standards development organization (SDO). As for constraining optionality within a standard, the industry has used the process of creating implementation guides, which can take a standard designed to support numerous use cases and provide guidance on how the optionality in the standard should be used to support a specific (or few) use case(s). We would encourage ONC to continue to take advantage of the implementation guide creation process. As for conformance, measures should focus both on conformance to the standard and to specific implementation guides for the standard.


Steven Waldren
Friday, June 23, 2017 - 10:22
Interoperability is critical to successfully achieving the Quadruple AIM and ensuring the success of value-based payment models. We appreciate the work of ONC to drive interoperability and we urge ONC to continue to prioritize this work. We are highly supportive of measuring interoperability throughout the health care ecosystem as we cannot improve what we do not measure. We also support a far greater level of transparency in the market place especially among health IT developers and health care systems. At the same time, we have ample concern that physicians and their practices will have a disproportionate share of the burden in measuring interoperability. We strongly urge ONC to not place more administrative burdens on physicians, as we already have significant burdens due to the lack of interoperability.

We appreciate the opportunity to provide ONC with our thoughts on some of the key questions laid out in the proposed interoperability standards measurement framework.

Louise Edmonds
Tuesday, June 13, 2017 - 21:44
At this stage I am interested in reading the document so I don't currently have feedback to offer you.

Pages