TOOLS: Sample PI Monitoring and Evaluation Plan

A monitoring and evaluation plan will be appended to the PI project proposal once the Performance Needs Assessment is finalized and interventions are selected and sequenced. The following is a sample monitoring and evaluation plan for a PI project aimed at improving provider performance through selection, training and supervision.

I. Key Evaluation Questions

Baseline Context: What is the contraceptive discontinuation rate in the area (as a proxy to insufficient quality of services provided)? What is the community context in which the PI project will operate?

Needs: What is the difference between desired and actual provider performance? What changes in provider and organizational performance are needed to increase service access and quality?

Inputs: Have correct, sufficient and appropriate PI interventions been designed? Are there sufficient human and financial resources and leadership support to design and implement them?

Processes/Implementation: How well is the PI project proceeding: are all inputs, processes and outputs on track and are milestones being met? What is the quality of training and supervision processes being implemented? To what extent are key stakeholders providing support and involvement needed to achieve results?

Outputs: Have service providers acquired new knowledge and skills as a result of training? Have supervisory visits and work environment improvements taken place?

Effects: To what extent did the indicators showing the difference between desired and actual performance change as a result of the interventions we implemented? To what extent has worker performance improved? Do service providers apply skills and knowledge to solve problems and improve service delivery as a result of project interventions? To what extent do selection, training and supervisory systems operate as desired?

Impacts: To what extent have desired strategic results been achieved? How satisfied are clients with performance improvements? Are there changes in the contraceptive continuation rates? How sustainable is the PI process?

II. Evaluation Workplan

Focus/Objective Indicators (illustrative) Methodology/ Instruments When Person(s) responsible/involved
Baseline Context
To assess the community context and FP/RH behavior in the project area and how it supports or hinders delivery of community-level FP/RH services
 Yearly contraceptive discontinuation rates
 Existence of trained TBAs and Community Health Workers (CHWs)
 Community leaders’ opinions about desired quality and responsiveness of health services and providers

DHS/special studies

Assessment

Interviews with members—leaders during PNA

Before project interventions are selected and sequenced PNA team with local service providers
Needs
To identify desired and actual service provider performance related to FP/RH service delivery; gaps between desired and actual performance; and causes of these performance gaps.

Provider performance baseline:

 Percent of eligible clients counseled
 FP/RH counseling and referral according to standards set by MOH (e.g., the provider establishes rapport with client; delivers appropriate and sufficient information; listens and treats client with respect; etc.)
 Number of FP/RH clients who indicate satisfaction with FP/RH services

Interviews with providers during PNA

Observation of providers

Client exit interviews

Before project interventions are selected and sequenced PNA team
To identify desired and actual organizational performance related to RH service support; gaps between desired and actual performance; and causes of these performance gaps.

Organizational performance:

 Selection of trainees according to service coverage needs, work specifications and inputs from the community
 Off-site FP/RH training of no more than five days duration
 Monthly on-the-job training and supportive supervision offered
Program records Before project interventions are selected and sequenced PNA team
Inputs
To assess the extent to which resources are adequate to design, implement and evaluate selection, training and supervision/job aids systems for the two years of the pilot project

General: 30 percent funding from the MOH, 20 percent funding from WB and 50 percent funding from the mission

 

 

 

Program records

  Project coordinator (with assistance from PI leader and other members of the implementation team)

Selection system:

 List of service delivery sites
 Job descriptions
 Selection criteria developed in consultation with community
MOU  

Training system:

 Number of trainers/preceptors
 Number and locale of training and practicum sites
 Types/number of training supplies
 Number and appropriateness of RH service curricula and materials
Interviews  

Supervision system:

 User-friendly supervisory protocols
 Number of supervision visits per quarter
 Number job aids required
 Mopeds for supervisor work-site visits

Expert/user review

Interviews

Before curricula—materials are finalized and copied
Processes
To assess the quality of training
 Trainees selected according to selection criteria
 Perceived trainer effectiveness, breadth and experience of trainers
 Duration, frequency of practicum sessions, logistics and organization of workshop, use of adult teaching methodologies, appropriateness of materials
 Adequacy of venue (e.g., sufficient space for practica)

Document reviews

Participant reaction forms

At the time of training Training team (with PI Leader and project coordinator)
To assess the quality of supervision
 Number of supervisory visits in a given time period
 Perceived effectiveness of supervision
Project records and interviews with providers Six to eight months after training Project coordinator and PI team
To assess project implementation (timing or quality of deployment/use of inputs)
 Training activities completed according to implementation plan
 Supervision protocols distributed to supervisors one month before training begins
 Supervision activities occur according to schedule (i.e., on a monthly basis)
 Job aids are pre-tested and distributed to greater than 70 percent of providers
 Leadership (stakeholders’) support and involvement

Document review (curricula, protocols, project implementation plans)

Interviews during project review

Prior to implementation target dates and/or on a yearly basis Project coordinator (with assistance from PI leader and other members of the implementation team)
Outputs
To assess immediate outcomes of training and non-training activities
 Number of professional and nonprofessional providers who achieve cut-off score for FP/RH knowledge and skills
 Number and percent of supervisory visits per supervisor in relation to plan
 Percent of distributed job aids that are used by providers
 Percent increase in providers’ communication skills

Comparison of scores to cut-off points

Comparison of actual with planned visits

On-site assessments

Observations

End of training and six to eight months after providers return from training Training team and project coordinator
Effects

Provider Performance

To evaluate the extent to which provider performance has improved and performance gap has been closed as a result of project interventions

 Percent of eligible clients counseled
 Provider compliance with CPI norms according to job aids and standards set by MOH
 Number of obstetrical emergencies referred/attended
 Number of FP/RH clients who indicate satisfaction with FP/RH services

Interviews with providers

Observation

Record review

Exit interviews

Follow up six to eight months after providers have received training and supervision Local consultant or supervisors (with selected members of PI project implementation team)

Organizational Performance

Evaluate the extent to which organization uses training and supportive system to deploy “the right person for the right job” and ensure learning and conducive job environment

 Number of persons trained who met selection criteria (education, years at work, profession/occupation, work site, etc.)
 Number of service delivery points with at least two trained/refreshed providers
 Percent of providers that have received greater than two supportive supervision visits in last year
 Percent of SDPs with job descriptions, job aids and performance feedback for their personnel

Program records

Interviews

Biodata forms

At start and end of project (comparative) PI implementation team (with trainers and supervisors)
Impact
To evaluate increase in service utilization and expansion
 Number of new and continuing users of FP by method
Service statistics and special studies At start and semi-annually Project coordinator
To evaluate increased sustainability of performance support systems (selection, training and supervision)
 Clients and community leaders’ feedback is built into the peformance system
 Institution’s long-term strategic plan incorporates performance support systems
 Yearly budgets are assigned to the systems

In-depth interviews to leaders

Institutional reviews

Travel logs/records

At start and end of project (comparative) Project coordinator or hired consultant