Introduction

The COVID-19 pandemic forced a rapid transformation in healthcare delivery, with telehealth emerging not only as a convenience but also as a critical tool. Telehealth has proven to be valuable in areas such as chronic disease management, patient education, monitoring, and continuity of care.1 It could be particularly promising in respiratory therapy; however, evidence of its effectiveness remains inconsistent. Some studies report improvements in patient outcomes, satisfaction, and reduced hospital admissions,1,2 while others show minimal or no significant outcomes.3–5 The conflicting findings may reflect variations in patient demographics, disease severity, and the technology platforms used in these studies. A notable gap in telehealth services is the lack of structured program evaluations. Despite widespread implementation, few programs utilize program evaluations that systematically measure outcomes or guide continuous improvement. Without rigorous evaluation, even well-designed telehealth programs may fail to achieve their intended outcomes. Program evaluations can systematically assess the performance of telehealth programs and improve their design, implementation, and impact within respiratory care. To ensure telehealth achieves its intended impact within respiratory care, they must adapt a structured, evidence-based approach to program evaluation. This will support accountability and quality improvement while strengthening the ability to integrate telehealth into standard respiratory therapy practice.

THE THREE-PHASE EVALUATION FRAMEWORK

A three-phase program evaluation framework comprising the structure, process, and outcome phases is essential for the success of any telehealth program in pulmonary medicine. Each phase addresses distinct yet interconnected questions, ensuring that telehealth services are well-designed, effectively implemented, and capable of producing impactful clinical outcomes.

The structure phase lays the foundation and defines the primary context for evaluation, while the process phase determines to what extent a telehealth program has been implemented as planned. The outcome phase evaluates the goals and outcomes to determine whether the telehealth program has achieved its goals in pulmonary medicine. In each phase, several questions are answered to determine if the program is effective and produces enough benefits to justify its investment. This article provides a step-by-step approach to program evaluation for a successful telehealth program in respiratory care (Table 1).

Table 1.A three-phase program evaluation framework, including the description, questions, and strategies used in each phase. SMART= Specific Measurable, Achievable, Relevant, and Timebound.
Phase Description Questions Strategies
Structure Phase Lays the foundation and defines the primary context for evaluation.
  • Who should be involved in program evaluation?
  • When should program evaluation occur?
  • Engage all stakeholders early to align priorities, include diverse perspectives, and address potential conflicts upfront.
  • Implement ongoing evaluation and stakeholder feedback enable timely adjustments to achieve program goals.
Process Phase Determines to what extent a telehealth program has been implemented as planned
  • What information is needed for program evaluation?
  • Which method should be used?
  • Determine whether the program is reaching its target population, being delivered as intended, and using resources efficiently.
  • Use a mixed-methods approach to match the strengths of each method.
Outcome Phase Evaluates the goals and outcomes to determine whether the telehealth program has achieved its goals
  • What are the “good outcomes” of telehealth programs?
  • How should the evaluation findings be communicated?
  • Set SMART outcomes for immediate, intermediate, and long-term.
  • Interpret findings to highlight successes, limitations, and actionable recommendations.
  • Use historical or comparative examples to show the impact of the telehealth program.
  • Share the report widely to promote transparency and best practices.

STRUCTURE PHASE: LAYING THE FOUNDATION

Step 1. Who should be involved in program evaluation?

An effective program should begin with inclusive stakeholder engagement. Because effectiveness, quality, and success are subjective concepts that vary among administrators, funders, clinicians, and patients, engaging all stakeholders early on ensures that diverse perspectives are considered, priorities are aligned, and potential sources of conflict are addressed early. This collaborative approach lays the groundwork for a shared understanding of priorities, services, intended outcomes, and success indicators for telehealth programs within respiratory care.

Step 2. When should program evaluation occur?

The timing of the evaluation determines the success of the telehealth program. At the initiation of each program, attention is directed toward the intended outcomes rather than the program’s operation. Many factors, apart from the program itself, may cause the failure of intended outcomes if the evaluation is conducted after implementation. Waiting until the end can lead to misdiagnosing or missed opportunities for mid-course corrections. Continuous evaluation and ongoing feedback from stakeholders allow for immediate adjustments, ensuring the program’s intended outcomes are met.

PROCESS PHASE: MONITORING IMPLEMENTATION

Step 3. What information is needed for program evaluation?

Three questions should be answered in this phase:

  1. Is the program reaching the target population?

  2. Is the program delivered as intended?

  3. Are resources used efficiently?

Language barriers, technical limitations, lack of service standardization, patients’ disconcert with technology, and patient/caregiver resistance to using telehealth can limit participation and effectiveness.3–6 Program evaluations can identify and address these barriers, which enables targeted improvements in telehealth services.

Therefore, in addition to clinical and economic outcomes, incorporating the patient-experience lens is essential for evaluating telehealth programs in respiratory care. While language and technology barriers are frequently noted, it is equally important to ensure that evaluation tools themselves are culturally inclusive. One way to achieve this is to use validated patient satisfaction questionnaires that are available in multiple languages, ensuring that diverse populations can participate meaningfully in feedback processes. Tools should also account for varying levels of health literacy and digital literacy by using plain language. Also, it is essential to include questions that assess patients’ comfort, understanding, and trust in the telehealth process. Such questions will help identify gaps in access and engagement, particularly among historically underserved communities. Embedding these practices into program evaluation improves patient-centered care and strengthens the overall quality of telehealth services in respiratory therapy.

The success of telehealth depends on the recruitment and training of competent, motivated and well-trained personnel. However, motivating clinicians to teach patients effectively can be challenging with increased workloads and competing demands.3 Therefore, alternative strategies and ongoing support are vital to improve the clinical pathway of patients with pulmonary diseases treated at home. Continuous monitoring of program delivery helps ensure alignment between what is intended to be done and actual telehealth delivery.

The effective operation of telehealth programs is linked to funds available for such services, with well-defined objectives and anticipated outcomes. Program integrity can be an issue when the program goals, content, and evaluation strategies are not determined in the structure phase. Actions should be taken to set standards and ensure the program is well-operated. Program evaluation can determine if the goals and intended outcomes of telehealth programs are well articulated, delivered effectively, and if the required resources are adequate to ensure success.

Step 4. Which method should be used?

No single evaluation method suffices. Mixed methods, such as surveys, interviews, observational methods, clinical trials, and cost-benefit analyses, should be selected based on the expertise, resources, and time available. Using a mixed-methods approach aligns with the strengths of each, providing a more comprehensive understanding of program performance.

OUTCOME PHASE: MEASURING IMPACT

Step 5. What are the “good outcomes” of telehealth programs?

Setting specific, measurable, achievable, relevant and time-bound (SMART) outcomes across three levels—immediate, intermediate, and long-term—is essential for the success of effective telehealth programs. If the immediate outcome is to increase patient education and engagement, which improves knowledge and motivation for disease management, the intermediate outcome could focus on patient adherence to treatment plans. Long-term outcomes may target decreased mortality and morbidity. Clearly defining these outcomes upfront allows effective assessment and continuous improvement.

An example of how the SMART criteria can guide program evaluation in telehealth is the assessment of a telehealth intervention for patients with COPD by tracking the 30-day readmission rate following hospital discharge. This metric is specific, as it focuses on readmission among patients with COPD. It is measurable, using data readily available from electronic health records. The goal is achievable, as it can be benchmarked against historical or baseline readmission rates. Also, it is relevant, aligning with the overarching goal of reducing hospital utilization, and it is time-bound, with the outcome assessed within a defined 30 days of discharge.

Similarly, the percentage of rural patients who successfully complete remote spirometry assessment can serve as a useful indicator for assessing access and usability of remote monitoring technologies in respiratory care. This measure is specific as it focuses on spirometry completion within a defined rural population. It is measurable, calculated by comparing the number of completed spirometry with the number of tests prescribed. The goal can be achieved depending on the availability of adequate infrastructure support and technologies. It is relevant, aligning with broader healthcare access goals. It is also time-bound when evaluated over a defined monitoring period, such as within 60 days of the initial prescription.

Additionally, incorporating program evaluation into telehealth for respiratory care services necessitates careful consideration of cost-effectiveness, as administrators often rely on financial metrics to inform their decisions. Costs such as equipment and clinician training hours impact their overall budget. However, these investments can be offset by long-term savings from reduced hospital readmissions, more efficient use of clinician time, and improved patient self-management. For instance, remote monitoring may reduce the need for in-person visits, which lowers facility costs. Evaluating telehealth programs through a return on investment lens ensures that resources are not only used efficiently but also contribute to sustainable and scalable respiratory care services. Anchoring these evaluations in daily respiratory care practice, such as remote coaching, data interpretation, and therapy adjustments, bridges the gap between abstract evaluation concepts and clinical realities.

Step 6. How should the evaluation findings be communicated?

An integral part of the program evaluation is to communicate the evaluation results clearly and accessibly to all stakeholders. Reports should include recommendations that will improve program expenditures, operations, and outcomes. The evaluation report should interpret the findings, identify successes and limitations, and offer actionable recommendations in a manner that helps the intended audience understand the program’s performance. The report should also explain what would happen if the telehealth program did not exist, using historical or comparative analyses, such as other communities or countries without telehealth programs in pulmonary medicine, to highlight the impact. The reports should be distributed as widely as possible to foster transparency and encourage the adoption of best practices.

When well-designed and rigorously evaluated, telehealth has the potential to reduce unnecessary primary care and emergency department visits, improve disease management, strengthen patient-clinician relationships, and improve access to care in rural and underserved communities.4,6 Effective utilization of this step-by-step program evaluation framework can help improve patient outcomes, optimize resources, and transform care in respiratory care for now and in the future.

KEY MESSAGES

  • A structured, step-by-step program evaluation framework is essential for the success of telehealth in respiratory care, ensuring that services are designed, implemented, and assessed systematically for maximum impact.

  • Engaging all stakeholders and conducting continuous evaluation from program inception helps align goals, identify barriers early, and support ongoing improvements in telehealth delivery.

  • Using multiple evaluation methods and clearly defining desired outcomes allows for a comprehensive understanding of program effectiveness, resource use, and areas needing adjustment.

  • Setting SMART outcomes across three levels—immediate, intermediate, and long-term—is essential for the success ofeffective telehealth programs.

  • Transparent reporting and broad dissemination of evaluation results foster accountability, inform decision-making, and promote the adoption of best practices in telehealth for both current and future needs.

Contributors

Dr. Arzu Ari conceived the idea of this paper and was responsible for manuscript’s design All authors were involved in drafting and commenting on the paper and have approved the final version.

Funding

This study did not receive any specific grant from funding agencies in the public, commercial, or not-for-profit sectors.

Competing interests

All authors have completed the ICMJE uniform disclosure form and declare no conflict of interest. AA is an Associate Editor at CJRT but was blinded to the decision-making process.

Ethical approval

Not required for this article type.

AI Statement

The authors confirm no generative AI or AI-assisted technology was used to generate content.