Patient Demographic Data Quality Framework

Data Profiling

Purpose

Develop and apply discovery techniques to physical data sets to develop an understanding of the content, quality, and rules of a specified set of data under management.

Introductory Notes

Data profiling is a cornerstone of an effective data quality improvement effort and is an important first step for many information technology initiatives. It is a discovery task conducted through automated (tool supported or custom queries) and/or manual analysis of physical records. For a selected data set, it reveals what is stored in databases and how physical values may differ from expected, allowed, or required values listed in data store documentation or described in metadata repositories. Definite errors are referred to as “defects.” Suspected errors are referred to as “anomalies.”

Data profiling differs from Data Quality Assessment in that profiling activities result in a series of conclusions about physical data sets, whereas the assessment process evaluates how well the data meets specific business quality requirements. Data profiling is typically the first step in conducting data quality assessments.

There are several levels of tests a data profiler can apply to a data set. At the most basic level, vendor data quality tools contain out-of-the-box tests that examine nulls, lengths, ranges, values, and formats. As a hypothetical example, if a profiling effort were conducted on 1,000 patient demographic records, the basic test (knows as “syntax” checks) results might yield defects and anomalies similar to these: 13 birth dates were entered in a MMDDYY format instead of the intended MMDDYYYY format; 27 street addresses contained less than 7 characters; 34 records did not contain a phone number; 15 records did not contain a ZIP code; 193 records did not contain a middle name; 3 records contained a patient gender value of ‘O’.

Another level of profiling includes tests on values that affect each other within a given record according to business rules (aka, “semantic” checks). For example, checking if a record with a patient status of “Deceased” also contains a date value in the date of death column. More complex exploration can also be applied between different records (aka, “context” tests). For example, determining that two patients are co-located (aka, “householding”) based on the occurrence of identical addresses, or identifying duplicate patient records. This level of profiling typically requires custom queries.

Profiling results are a primary input to resolution of anomalies, typically through manual analysis and data cleansing activities. In the example above of unusually short street addresses, a manual review may find that the values are valid, e.g., Zip ST or Fen ST. In the gender value example, the reviewer may need to research the gender of the three patients. In the case of missing ZIP codes or phone numbers, the reviewer might refer the lookup to billing. Profiling results also feed into data cleansing to correct errors. For example, standardization of the MMDDYYYY format can be implemented through an automated script.

An organization that implements effective profiling practices can realize the following benefits:

• Deepen understanding of patient demographic data among all stakeholders and fosters a quality culture;
• Increase reliability of data for research, population health and other quality reporting analytics, accuracy of Meaningful Use (MU) reporting, and interoperability;
• Identify and correct critical defects;
• Reliable discovery of anomalies and duplicates;
• Ability to update and adjust profile as the patient population changes over time and as new standards are enacted;
• Increase savings in effort and cost through reuse of quality rules; and
• Discovery of enhancements to business processes, technologies, and procedures to prevent errors.

It is recommended that an organization which hosts at least one data store containing patient demographic data first conduct a comprehensive baseline profiling effort. This will clearly reveal the current state of the data, allow formulation of meaningful metrics, and become the foundation for tracking improvements going forward.

Once the data has undergone baseline cleansing and anomaly resolution, the organization should determine the frequency at which it will conduct periodic profiling of the patient demographic data, reusing the same tests and enhancing with new tests if additional quality rules are adopted through data quality assessments, changes in standards, or additions to demographic data. For example, adopting and following a standard practice of basic profiling and cleansing before running advanced rule sets (i.e. algorithms) for identifying matching patient records will help to increase the accuracy of matching algorithms. This may initially increase the number of duplicates discovered (e.g., false negatives), but should lower the instance of duplicate records over time.

Data profiling is an important activity in evaluating data quality for a specific purpose and can be event-driven. For example, if a healthcare organization acquires another organization, profiling the new source would be recommended prior to migrating and integrating records into the destination system. Similarly, organizations are advised to profile data prior to: finalizing the design of a master data store; performing data conversion or consolidation; implementing a new EHR, patient registration, billing, or similar system or migration; and connecting to a health information exchange.

In addition to defect fixes and resolution of anomalies, organizations may determine that results indicate that: metadata descriptions should be updated, loading scripts should be enhanced with refined or new quality rules, or existing data structures should be redesigned. What is learned through data profiling often also serves as a key input to development of metrics, content standards, business process redesign, and technology refreshes, to further improve the quality of the organization's patient data.

It is recommended that an organization implement a standard process for conducting profiling efforts, which includes the reporting and publication of results to relevant stakeholders. Prioritization of candidate data stores and data sets for profiling should be based on business needs and objectives expressed in the data quality plan.