Evaluation Framework for
Public Health Information Systems

Purpose
This project developed a framework for assessing the value of linking newborn screening LIMS with child health information systems.

Project summary
The integration of public health newborn screening laboratory information management systems (LIMS) with child health program information systems promises to protect the health of children by assuring comprehensive follow-up and notification of screening results to the medical home. No comprehensive framework exists, however, to assess the value of electronic linkage between public health LIMS and other health information systems. Applying a comprehensive evaluation framework to the interface between newborn screening laboratory information systems and newborn screening program (e.g., case management, surveillance, etc.) information systems offers an opportunity to advance our understanding of where interoperability delivers significant impacts on health outcomes, as well as the processes of lab testing and programmatic case management and follow-up.

In an important step toward developing metrics for evaluating public health information systems, researchers at the Public Health Informatics Institute developed “A Framework for Assessing the Value of Integrating Newborn Screening Laboratory Information Management Systems with Child Health Information Systems.” In 2004-2005, they drafted a nine-dimension framework through review of published literature and evaluation of 16 state public health agency proposals to establish or expand integrated child health information systems. The resulting framework incorporates the major parameters of information quality, system quality, and service quality, as well as individual, organizational and economic impacts, and health and health services impacts.

A logic model for the framework, which relates characteristics of the system to health outcomes, depicts the logic of translating the technical and programmatic inputs of the integrated information system; how these inputs produce quality through better information, better service to stakeholders; how these quality improvements relate to the impact individuals make in doing their jobs or the overall impact a public health agency has in fulfilling its mission; and how all of these impacts taken together result in improvements health outcomes.

The framework was vetted by using it to evaluate the Rhode Island integrated child information system, KIDSNET, which encompasses an interface with the New England Newborn Screening laboratory’s LIMS. A case study of the integration of the New England Newborn Screening Laboratory information system with KIDSNET provided a rich description of how the nine dimensions of the framework and the attributes of those dimensions can be used to examine where and how an integrated child health information system creates valuable efficiencies and program effectiveness.

While acknowledging the limitations of a study based on application of the evaluation framework to one instance of integration of a public health LIMS with child health information systems for a limited period of time, the report draws several conclusions and makes recommendations for next steps, including broader application and refinement of the framework, and the translation of the concepts of the framework into quantifiable metrics.

The project was supported by a contract with the Association of Public Health Laboratories (APHL). Project partners included APHL, its Informatics Committee and Newborn Screening Committee, and the Rhode Island Department of Health.

Related Documents
Towards Measuring Value: A evaluation framework for public health information systems