Printer Friendly, PDF & Email Printer Friendly, PDF & Email

§170.315(g)(3) Safety-enhanced design

Version 1.1 Updated on 07-25-2016
Revision History
Version # Description of Change Version Date
1.0

Final Test Procedure

01-20-2016
1.1

Added reference to the SED CHPL reference document at the beginning.

07-25-2016
Regulation Text
Regulation Text

§170.315 (g)(3) Safety-enhanced design—

  1. User-centered design processes must be applied to each capability technology includes that is specified in the following certification criteria: paragraphs (a)(1) through (9) and (14), (b)(2) and (3) of this section.
  2. Number of test participants. A minimum of 10 test participants must be used for the testing of each capability identified in paragraph (g)(3)(i) of this section.
  3. One of the following must be submitted on the user-centered design processed used:
    1. Name, description and citation (URL and/or publication citation) for an industry or federal government standard.
    2. Name the process(es), provide an outline of the process(es), a short description of the process(es), and an explanation of the reason(s) why use of any of the existing user-centered design standards was impractical.
  4. The following information/sections from NISTIR 7742 must be submitted for each capability to which user-centered design processes were applied:
    1. Name and product version; date and location of the test; test environment; description of the intended users; and total number of participants;
    2. Description of participants, including: sex; age; education; occupation/role; professional experience; computer experience; and product experience;
    3. Description of the user tasks that were tested and association of each task to corresponding certification criteria;
    4. The specific metrics captured during the testing of each user task performed in (g)(3)(iv)(C) of this section, which must include: task success (%); task failures (%); task standard deviations (%); task performance time; and user satisfaction rating (based on a scale with 1 as very difficult and 5 as very easy) or an alternative acceptable user satisfaction measure;
    5. Test results for each task using the metrics identified above in paragraph (g)(3)(iv)(D) of this section; and
    6. Results and data analysis narrative, including: major test finding; effectiveness; efficiency; satisfaction; and areas for improvement.
  5. Submit test scenarios used in summative usability testing.
Standard(s) Referenced

Please consult the Final Rule entitled: 2015 Edition Health Information Technology (Health IT) Certification Criteria, 2015 Edition Base Electronic Health Record (EHR) Definition, and ONC Health IT Certification Program Modifications for a detailed description of the certification criterion with which these testing steps are associated. We also encourage developers to consult the Certification Companion Guide in tandem with the test procedure as they provide clarifications that may be useful for product development and testing.

Note: The order in which the test steps are listed reflects the sequence of the certification criterion and does not necessarily prescribe the order in which the test should take place.
 

Testing components

No GAP Icon Documentation Icon Visual Inspection Icon No Test Tool Icon No ONC Supplied Test Data Icon

 

The ONC Health IT Certification Program requires that a limited set of quantitative data elements related to the safety-enhanced design (SED) testing be reported and displayed to the public via the new Certified Health Product List. Reporting instructions to the CHPL, including data elements and their definitions, formats, and allowable values, may be found at CHPL SED Guide.

 

Paragraph (g)(3)(i)

System Under Test Test Lab Verification

The health IT developer submits documentation demonstrating that user-centered design (UCD) process(es) were applied to all of the safety-enhanced design referenced certification criterion for which these corresponding capabilities are being presented for certification.

The tester verifies that the submitted documentation outlines user-centered design (UCD) process(es) to each capability for any of the following safety-enhanced design criterion submitted for certification:

§ 170.315 (a)(1)   Computerized Provider Order Entry (CPOE) – medications
§ 170.315 (a)(2)   CPOE – laboratory
§ 170.315 (a)(3)   CPOE – diagnostic imaging
§ 170.315 (a)(4)   Drug-drug, Drug-allergy Interaction Checks for CPOE
§ 170.315 (a)(5)   Demographics
§ 170.315 (a)(6)   Problem List
§ 170.315 (a)(7)   Medication List
§ 170.315 (a)(8)   Medication Allergy List
§ 170.315 (a)(9)   Clinical Decision Support
§ 170.315 (a)(14) Implantable Device List
§ 170.315 (b)(2)   Clinical Information Reconciliation and Incorporation
§ 170.315 (b)(3)   Electronic Prescribing


Paragraph (g)(3)(ii)

System Under Test Test Lab Verification

The Heath IT Module must have conducted summative usability testing with a minimum of 10 test participants for each safety-enhanced design criterion capabilities for which certification is being sought.

The tester verifies that at least 10 test participants, representative of the intended user population, participated in summative usability testing for each safety-enhanced design criterion and the associated capabilities.


Paragraph (g)(3)(iii)

System Under Test Test Lab Verification

The health IT developer submits documentation outlining the user-centered design (UCD) process used for each of the safety-enhanced design criteria (specified in (g)(3)(i)) submitted for testing that includes one of the following:

  1. Where one of the industry or federal government standards is used:
    • Name;
    • Description;
    • Citation (URL and/or publication citation).

OR
 

  1. Where an industry or federal government standard is NOT used:
    • Name & citation (URL and/or publication citation) of the industry standard Process(es) that formed basis of the “custom” process;
    • Outline of the process;
    • Short description of the process(es);
    • Explanation of the reason(s) why use of any of the existing UCD standards was impractical.

The tester verifies user-centered design (UCD) process(es) have been documented for each of the safety-enhanced design either by:

  1. The name, description, and citation (URL and/or publication citation) reference of the UCD industry standard (e.g ISO 9241-210, ISO 13407, ISO 16982, ISO/IEC 62366, and NISTIR 7741);

OR
 

  1. The name and citation (URL and/or publication citation) of the industry standard process(es) that formed the basis of the “custom” process, outline of the process, short description of the process(es) used, and explanation of the reason(s) why use of any of the existing UCD industry or federal government standards was impractical.

Paragraph (g)(3)(iv)

System Under Test Test Lab Verification

The health IT developer must have conducted summative usability testing and provides the required documentation for each of the safety-enhanced design criteria (specified in (g)(3)(i)) submitted for summative usability testing, using:

  • The NISTIR 7742 Customized Common Industry Format (CIF) Template for Electronic Health Record Usability Testing content report for usability test report(s) that address the application of the documented and referenced UCD process(es)

The health IT developer may provide the required information/sections from NISTIR 7742 in any format, provided that the required information is included.

The tester verifies the existence and adequacy of the summative usability test report(s) for each safety-enhanced design criterion (specified in (g)(3)(i)) capability being presented for testing/certification. The tester verifies that the report(s) conform(s) to the content and completion requirements of the required information/sections of NISTIR 7742. The tester verifies that the name and version of the product are the final version (release) of the product for which the health IT developer is seeking certification.


Paragraph (g)(3)(iv)(A)

System Under Test Test Lab Verification

The summative usability testing report includes NISTIR 7742 Customized Common Industry Format (CIF) Template for Electronic Health Record Usability Testing information/sections:

  • Date and location of the test
  • Test environment
  • Description of the intended users
  • Total number of participants

The tester verifies the existence and adequacy of the required NISTIR 7742 information/sections.


Paragraph (g)(3)(iv)(B)

System Under Test Test Lab Verification

The summative usability testing report includes NISTIR 7742 Customized Common Industry Format (CIF) Template for Electronic Health Record Usability Testing information/sections:

  • Description of participants (i.e., Sex, Age, Education, Occupation/role, Professional experience, Computer experience, Product experience)

The tester verifies that the demographic characteristics of the subject pool meet the specifications of the particular requirement (NISTIR 7742 3.1 “Participants”); where use conditions and population of the users are analyzed to ensure participant characteristics reflect the audience of current and future users.


Paragraph (g)(3)(iv)(C)

System Under Test Test Lab Verification
  1. The summative usability testing report includes NISTIR 7742 Customized Common Industry Format (CIF) Template for Electronic Health Record Usability Testing information/sections:
    • Description of the user tasks (task scenarios) that were tested and association of each task to corresponding certification criteria
  1. The tester verifies that the user tasks employed in the study are prioritized in accordance with the risk associated with user errors (NISTIR 7742 3.3 “Tasks”).
  2. The tester verifies that the test scenarios included in the NISTIR 7742 content report for each of the UCD Required Criteria are inclusive of the tasks or functionality the health IT developer provided for testing to the certification criterion.

Paragraph (g)(3)(iv)(D)

System Under Test Test Lab Verification

The summative usability testing report includes NISTIR 7742 Customized Common Industry Format (CIF) Template for Electronic Health Record Usability Testing information/sections:

  • The specific metrics captured during the summative testing of each user task performed in (g)(3)(iv)(C) of this section (Task Success (%), Task Failures (%), Task Standard Deviations of Task Performance Time (%), Task Performance Time, User Satisfaction Rating (Scale with 1 as very difficult and 5 as very easy) or an alternative acceptable user satisfaction measure.)

The tester verifies that the specified metrics are captured in the report.


Paragraph (g)(3)(iv)(E)

System Under Test Test Lab Verification

The summative usability testing report includes NISTIR 7742 Customized Common Industry Format (CIF) Template for Electronic Health Record Usability Testing information/sections:

  • Test results for each task using metrics listed above in (g)(3)(iv)(D).

The tester verifies that test results are provided for each task using the specified metrics in (g)(3)(iv)(D).


Paragraph (g)(3)(iv)(F)

System Under Test Test Lab Verification
  1. The summative usability testing report includes NISTIR 7742 Customized Common Industry Format (CIF) Template for Electronic Health Record Usability Testing information/sections:
    • Results and data analysis narrative (i.e., Major test finding, Effectiveness, Efficiency, Satisfaction, Areas for improvement). Measures of satisfaction may include task-based satisfaction measures, post-session satisfaction measures and other industry-standard or literature-recognized satisfaction measures (e.g., the Single Ease-of-use Question, System Usability Scale, Software Usability Measurement Inventory, etc.).
  1. The tester verifies that all major test findings and the identified area(s) of improvements are reported.
  2. The tester verifies how effectiveness and efficiency were evaluated (NISTIR 7742 3.9 “Usability Metrics”).
  3. The tester verifies that test results provided an analysis of the use, tested performance and error rates in order to identify risk prone errors -- with a potential likelihood of occurrence and adverse consequences (NISTIR 7742. results).
  4. The tester verifies that the following NISTIR 7742 measures of effectiveness, efficiency, and satisfaction were collected for each participant:
    • Number of tasks successfully completed within the allotted time without assistance;
    • Time to complete the tasks;
    • Number and types of errors;
    • Path deviations;
    • Participant’s verbalizations; and
    • Participant’s satisfaction ratings of the system.

Paragraph (g)(3)(v)

System Under Test Test Lab Verification
  1. The heath IT developer supplies the test scenarios used for the summative usability testing conducted on each of the safety-enhanced design criteria (specified in (g)(3)(i)) submitted for testing. The test scenarios used in the summative testing should reflect prioritized use cases based upon a risk analysis.
  1. The tester verifies the existence of the test scenarios used for the summative usability testing, containing at a minimum, test scenarios to cover all of the safety-enhanced design criteria and associated capabilities (specified in (g)(3)(i)) submitted for testing/certification.
  2. The tester shall verify that the name and version of the product are the final version (release) of the product for which certification is being sought.

Version 1.3 Updated on 06-04-2018
Revision History
Version # Description of Change Version Date
1.0

Initial Publication

10-26-2015
1.1
  • General clarification added to clarify how gap certification eligible certification criteria are treated under the SED criterion.
  • Clarification were removed from paragraphs (g)(3)(iii)(A) and (g)(3)(iii)(B) as it was deemed redundant with the regulatory requirement.
  • Clarification added to paragraph (g)(3)(iv) to distinguish it from paragraph (g)(3)(iii)’s requirements and to clarify applicable timing considerations.
03-30-2016
1.2

An additional resource has been added to paragraphs (g)(3)(i) and (g)(3)(ii)(A) that health IT developers may choose to review in order to select a UCD process specific for pediatric care settings.

02-01-2018
1.3

Added NISTIR 7742 and 7804 to the list of examples of resources that technology developers may choose to review in order to select a UCD process for clarification to paragraphs (g)(3)(i) and (g)(3)(iii)(A).

06-04-2018
Regulation Text
Regulation Text

§170.315 (g)(3) Safety-enhanced design—

  1. User-centered design processes must be applied to each capability technology includes that is specified in the following certification criteria: paragraphs (a)(1) through (9) and (14), (b)(2) and (3) of this section.
  2. Number of test participants. A minimum of 10 test participants must be used for the testing of each capability identified in paragraph (g)(3)(i) of this section.
  3. One of the following must be submitted on the user-centered design processed used:
    1. Name, description and citation (URL and/or publication citation) for an industry or federal government standard.
    2. Name the process(es), provide an outline of the process(es), a short description of the process(es), and an explanation of the reason(s) why use of any of the existing user-centered design standards was impractical.
  4. The following information/sections from NISTIR 7742 must be submitted for each capability to which user-centered design processes were applied:
    1. Name and product version; date and location of the test; test environment; description of the intended users; and total number of participants;
    2. Description of participants, including: sex; age; education; occupation/role; professional experience; computer experience; and product experience;
    3. Description of the user tasks that were tested and association of each task to corresponding certification criteria;
    4. The specific metrics captured during the testing of each user task performed in (g)(3)(iv)(C) of this section, which must include: task success (%); task failures (%); task standard deviations (%); task performance time; and user satisfaction rating (based on a scale with 1 as very difficult and 5 as very easy) or an alternative acceptable user satisfaction measure;
    5. Test results for each task using the metrics identified above in paragraph (g)(3)(iv)(D) of this section; and
    6. Results and data analysis narrative, including: major test finding; effectiveness; efficiency; satisfaction; and areas for improvement.
  5. Submit test scenarios used in summative usability testing.
Standard(s) Referenced

Certification Companion Guide: Safety-enhanced design

This Certification Companion Guide (CCG) is an informative document designed to assist with health IT product development. The CCG is not a substitute for the 2015 Edition final regulation. It extracts key portions of the rule’s preamble and includes subsequent clarifying interpretations. To access the full context of regulatory intent please consult the 2015 Edition final rule or other included regulatory reference. The CCG is for public use and should not be sold or redistributed.
 

 

Certification Requirements

This certification criterion was adopted at § 170.315(g)(3), and is required for all developers seeking certification to § 170.315(a)(1) through (9), (a)(14), (b)(2) or (b)(3). There are no associated required privacy and security criterion for this certification criterion.

Technical Explanations and Clarifications

 

Applies to entire criterion

Clarifications:

  • The application of user-centered design (UCD) during development and summative testing is limited to only those twelve 2015 Edition certification criteria specified in this certification criterion and only for which certification is sought, namely [80 FR 62670]:
    • § 170.315 (a)(1) Computerized provider order entry (CPOE) – medications
    • § 170.315 (a)(2) Computerized provider order entry (CPOE) – laboratory
    • § 170.315 (a)(3) Computerized provider order entry (CPOE) – diagnostic imaging
    • § 170.315 (a)(4) Drug-drug, drug-allergy interaction checks for CPOE 
    • § 170.315 (a)(5) Demographics
    • § 170.315 (a)(6) Problem list
    • § 170.315 (a)(7) Medication list
    • § 170.315 (a)(8) Medication allergy list
    • § 170.315 (a)(9) Clinical decision support (CDS)
    • § 170.315 (a)(14) Implantable device list
    • § 170.315 (b)(2) Clinical information reconciliation and incorporation
    • § 170.315 (b)(3) Electronic prescribing
  • As a “revised” certification criterion, the safety-enhanced design (SED) certification criterion is not “gap certification eligible.” [80 FR 62609-62610, 62670] Thus, despite the fact that some of the functionality-based certification criteria referenced by the SED criterion are “gap certification eligible” for their functionality, all of the certification criteria referenced by this SED criterion (as applicable to certification scope sought) must have UCD processes applied and new “2015 Edition” summative usability test results as the basis for 2015 Edition certification.
  • To demonstrate compliance with this certification criterion, UCD process(es) must have been applied to each capability of technology that is associated with the certification criteria named in this certification criterion. [77 FR 54188]
  • If technology is presented for certification and includes capabilities to which this certification criterion would apply, but for which certification is not sought, then those other capabilities for which certification is not sought would not have to have had UCD process(es) applied because they would be beyond the scope of certification. [77 FR 54188]
  • ONC-Authorized Certification Bodies (ONC-ACBs) should be notified when changes to user-interface aspects occur. ONC-ACBs are required to obtain a record of all updates to certified Health IT Modules affecting the capabilities in certification criteria to which this “safety-enhanced design” criterion applies on a calendar quarterly basis. [80 FR 62727]
  • The documentation required by this “safety-enhanced design” criterion will become a component of the publicly available testing results on which a certification is based. [77 FR 54187]
  • We do not expect health IT developers to include trade secrets or proprietary information in these reports. [77 FR 54188]
  • The ISO definition of usability is “[t]he extent to which a product can be used by specified users to achieve specified goals with effectiveness, efficiency, and satisfaction in a specified context of use." [see also 77 FR 54186]
  • Health IT developers who have already followed UCD in previous development efforts for the certification criteria identified in the SED criterion would be performing a retrospective analysis for the purposes of certification. [see also 77 FR 54188] We note that the discussion of retrospective analysis provided in the 2014 Edition Final Rule was in the context of health IT being certified for the first time to the new 2014 Edition “SED” certification criterion. As an illustration of retrospective analysis for certification to the 2015 Edition “SED” certification criterion, if a health IT developer had followed/applied a UCD process for any or all of the certification criteria referenced by the 2015 Edition SED certification criterion, a health IT developer would be permitted to cite that previously applied UCD process.

Paragraph (g)(3)(i)

Technical outcome – Developer must have applied UCD process(es) for any of the twelve 2015 Edition certification criteria specified in this criterion and for which certification is sought.

Clarifications:

  • The reported UCD process(es) must have been applied during the design and development of the capabilities/associated criteria.
  • If UCD had not been previously applied to capabilities associated with the certification criteria, the technology would ultimately need to have such UCD process(es) applied before it would be able to be certified. [77 FR 54187]
  • Examples of resources that technology developers may choose to review in order to select a UCD:
    • ISO 9241-11;
    • ISO 13407;
    • ISO 16982;
    • ISO/IEC 62366;
    • ISO 9241-210;
    • NISTIR 7741;
    • NISTIR 7742;
    • NISTIR 7804; and
    • NISTIR 7865 (“A Human Factors Guide to Enhance EHR Usability of Critical User Interactions when Supporting Pediatric Patient Care”).
  • Any UCD process selected by a health IT developer is appropriate, and it need not be listed in the examples we provided in order to be acceptable. [77 FR 54188]

Paragraph (g)(3)(ii)

Technical outcome – At a minimum, ten participants must have been included in the summative usability testing required for each required capability for which certification is sought.

Clarifications:

  • Although only 10 participants are required, health IT developers are strongly encouraged to exceed the mandatory minimum in an effort to identify and resolve more problems. [80 FR 62671]
  • The cohort of users who are selected as participants will vary with the product and its intended users and should not be limited to clinicians but instead consist of test participants with the occupation and experience that aligns with the capability under testing. [80 FR 62670]
  • We recommend that health IT developers follow NISTIR 7804 for human factors validation testing of the final product to be certified. [80 FR 62671]
  • Recommended resources to consider:
    • NISTIR 7804; and
    • NISTIR 7741.

Paragraph (g)(3)(iii)(A)

Technical outcome – The developer must submit the name, description and citation (URL and/or publication citation) for the industry or federal government standard used in UCD for the development of each required/applicable capability presented for certification.

Clarifications:

  • Examples of resources that technology developers may choose to review in order to select a UCD process include, but are not limited to:
    • ISO 9241-11;
    • ISO 13407;
    • ISO 16982;
    • ISO/IEC 62366;
    • ISO 9241-210;
    • NISTIR 7741;
    • NISTIR 7742;
    • NISTIR 7804; and
    • NISTIR 7865 (“A Human Factors Guide to Enhance EHR Usability of Critical User Interactions when Supporting Pediatric Patient Care”).

Paragraph (g)(3)(iii)(B)

Technical outcome – If a non-standard UCD process was used in development (i.e., § 170.315(g)(3)(iii)(A) was NOT met), the health IT developer must report the process(es), provide an outline of the process(es), a short description of the process(es), and an explanation of the reason(s) why use of any of the existing UCD standards was impractical for each required/applicable capability presented for certification.

Clarifications:

  • No additional clarifications available.

Paragraph (g)(3)(iv)

Technical outcome – The information specified in (g)(3)(iv)(A)-(F) must be submitted for each capability to which UCD processes were applied in product development and when summative usability testing was conducted.

Clarifications:

  • All of the data elements and sections specified must to be completed, including ‘‘major findings’’ and ‘‘areas for improvement.’’ [see also 80 FR 62670]
  • Health IT developers can perform many iterations of the usability testing, but the submission that is ultimately provided for summative usability testing and certification must be an expression of a final iteration. [see also 80 FR 62671]
  • Only lab based summative testing is necessary to be performed in order to demonstrate compliance with this certification criterion. Nothing precludes field testing and formative testing from also being performed and we encourage technology developers to do so. [see also 77 FR 54189]
  • Information must be submitted for each and every one of the criteria specified in the 2015 Edition “SED” criterion to become part of the test results publicly available on the Certified Health IT Product List. [see also 80 FR 62725]
  • To demonstrate compliance with this certification criterion this information would need to be available to an ONC–ACB for review, but the form and format for how the data would be presented for testing would not necessarily need to be NISTIR 7742 template. This documentation would become a component of the publicly available testing results on which a certification is based. [see also 77 FR 54187]
  • For the requirement in (g)(3)(iv)(D), it is permissible to submit an alternative acceptable user satisfaction measure to meet the requirements of this criterion. As such, a health IT developer could meet the proposed NISTIR 7742 based approach for user satisfaction or provide documentation of an alternative acceptable user satisfaction measure. [see also 80 FR 62671]
  • It is important to note a specific distinction between paragraphs (g)(3)(iii) and (g)(3)(iv). Both paragraphs are complementary, but require different information and have different applicable timing considerations.
    • Paragraph (g)(3)(iii) requires the disclosure of the UCD processes that were applied and followed in the development of the applicable capabilities named in the 2015 “SED” certification criterion and for which certification is sought.
      • If a health IT developer had previously developed a capability following a certain UCD process, we clarified that the health IT developer would not need to pick a new UCD standard as a result of our rulemaking and use it to reengineer the capability following/reapplying that UCD standard. In that respect, the timing for this paragraph (g)(3)(iii) allows for retrospective attribution. [see also 77 FR 54188]
      • We note that in the 2014 Edition final rule preamble we incorrectly referenced NISTIR 7742 on page 54188. NISTIR 7742 does not require the reporting of the UCD standard/process used. Rather, in the preamble of the 2014 Edition final rule we stated that requirement. [77 FR 54188]
    • Paragraph (g)(3)(iv) requires that information/sections from NISTIR 7742 must be submitted for each capability to which user-centered design processes were applied.
      • In so doing, a health IT developer cannot use summative usability test results to meet this paragraph that were not generated for the specific edition of certification criteria for which certification is sought.
      • In other words, if a health IT developer had followed a UCD process in 2010 to design its CPOE functionality, performed and provided summative usability test results in 2013 for the purposes of 2014 Edition certification, the health IT developer must produce a “fresh” set of summative usability test results for 2015 Edition testing and certification. See also the clarification note about gap certification eligibility in the general clarification section above.
    • Once certified to the 2015 Edition SED certification criterion, a health IT developer can request inherited certified status for a newer version of its 2015 Edition certified product consistent with 45 CFR 170.550(k). In so doing, the health IT developer may attribute (“carry forward”) the summative usability test results created for the previously certified 2015 Edition product version so long as the scope of referenced SED remains the same. In instances where an additional SED referenced certification criterion/criteria is/are added the summative usability test results applicable to those addition in scope would be needed.

Paragraph (g)(3)(v)

Technical outcome – The test scenarios participants used during the summative usability testing must be submitted as part of the test results report.

Clarifications:

  • In accordance with NISTIR 7804 Technical Evaluation, Testing, and Validation of the Usability of Electronic Health Records (EUP) (page 8), we recommended that the test scenarios be based upon an analysis of critical use risks for patient safety, which can be mitigated or eliminated by improvements to the user interface design. [80 FR 62670]
  • NIST recently developed an additional recommended resource for test scenarios: NISTIR 7804-1: Technical Evaluation, Testing, and Validation of the Usability of Electronic Health Records: Empirically Based Use Cases for Validating Safety-Enhanced Usability and Guidelines for Standardization.

Content last reviewed on September 24, 2018
Was this page helpful?