Everyone agrees that poor countries need stronger health systems. Everyone agrees also that evaluation and monitoring is a crucial part of the policy cycle. Yet, little is written about what core investments need to be made in terms of data. And, even less is written about how to organize the human analytic resources to take best advantage of the data to provide the evidence needed to set priorities and to monitor solutions to gauge whether they are working as intended. My experience is that absolutely no in-country organizations have been established to do the routine health system evaluation and monitoring work, and certainly no strategic plans have been written about needed data investments. Basically, this all seems to be supplied and considered by donors. Indeed the only viable data push that has been evident is for NHA. But this is not generally seen as an important and useful in-country data resource, but as something that donor analysts want so that cross country comparisons can be made, and something the donors must pay for or it wont happen.

Data , and the ability to draw conclusions from it about what is working ( and not) is generally a threat to Ministry officials. I ran a focus group some years back in Egypt of young people who had returned to the Health sector there, after going abroad to earn MPH, Masters in economics, and other graduate degrees. Ever single one of them was disappointed in their ability to use their skills in data analysis and policy development in the Ministry. Every single one was trying to leave to the private sector, to donor funded projects, to the Gulf, and so on. Egypt is only one place, but the situation certainly rings true that data is hopelessly spotty and generally not available, and there is no organization anywhere in that country doing health system performance appraisal.

Other LMICs are not much different. Yes, there are statistics agencies, and some countries have Public Health analytic units. And, there are a handful of embedded NHA units. But, nowhere (outside of Donor groups and Observatories) is nobody doing in-country professional assessments of health system perfromance, and assessing (with data) whether it is getting better or worse. That is all left for the spin people, free from limits that might be binding from facts that might be introduced. With few exceptions, this is the world of the LMICs.

 

Data Investments

 Typically data on the health sector is

• Excessive and inaccurate about the government providers
• Unavailable for private and NGO providers
• Unavailable from households
• Sometimes disease reporting networks are available
• Vital events data are available (births, deaths, IMR, etc.)

But, in terms of access, health, out of pocket spending and other household measures, much is left to the periodic DHS and the NHA household survey,

Virtually no provider data exists from non government organizations. And much provider data from government providers, in my experience, is inaccurate, and inadequate to do research on budget equity (budgets data are never linked to service use), or usage of particular services (few categories of usage are used, and never is the provider data linked in any way to population characteristics of the catchment area). There are lots of numbers, but no evidence that a policy researcher has had a hand in specifying, assembling, and linking relevant data.

Country health data systems need to be composed on certain things in order to routinely assess performance of the health system. Sure, special data is needed for special topics. It would be silly to try to imagine all the data one could ever want. But the basic building blocks would seem to be:

1. A tri-annual household sample survey. Sample size would need to be adequate to make reasonable estimates for major demographic groups (men, women; elderly, children, adults; rural, urban; major ethnic groups; three income categories). The survey would capture health system performance metrics like utilization, regular source of care, out of pocket spending, self reported health, satisfaction, barriers to access, chronic diseases).
2.  Every 2 years–  Hospital statistics. Every hospital should have a unique ID a location, including NGOs and Private. Routine data could be things like ownership, services, size, usage by major types of patients (IP, OP, Births, ED, LTC/Rehab), special programs, spending, staffing, financing sources (OOP, insurance, budget), and affiliated free standing clinics, and affiliated programs of community outreach  (circuit visiting programs and embedded CHWs). Periodically, special studies could piggyback on this survey.

From such data two kinds of analytic files can be constructed (along with demographic, census, and vital event data) to support analysis of health system performance. Those primary analytic files could be

Area Aggregate File. This file would link together Census and various demographic data to provider data, and Household survey estimates into records for each of the major regains/cities of the country. In Egypt, there are 144 districts, which might be used. In the USA we have 307 hospital referral areas, and so forth. So for each record we would be able to see the number of providers of each type, the beds per capita, visit rates and days of care per 1000, facility spending per capita, etc. We would also be able to know the age mix of the population, average educational attainment, average income levels, employment rates, etc. Vital event data would also be summarized by region and included here.

 This file would be a primary resource to study the extent of geographic variation in the use of the health system, and would permit study of drivers of those geographic differences.

National sample person level file. This file would link patient survey data to provider characteristics and to population characteristics (from the above file). This would permit using the household survey data to study questions of access barriers, utilization rates, health status, and the patterns in these things as they relate to availability of care (in the area) .

To construct such analytic files would also require routine ways to code and crosswalk geographic regions on both the household data and the provider data.

In addition to these data required to examine health system performance, are other important data resources.

 3.   NHA flows of health financing. Periodically (every 3-4 years) This data collection is aimed at understanding the sources and uses of health spending according to a pre-described methodology. This makes the data more or less comparable to other countries. The NHA household survey items should be incorporated in the larger household survey, mentioned earlier as # 1.

Three other data sources are often available now. They serve the top needs of public health in the country but, with a few exception for mortality rates, are not a core data need for health system strengthening M & E work.

4.   Communicable disease reporting (electronic/manual)

5.   Vital events and Population (births/deaths)

6.   Disease/tumor registry system

 

Manual and Electronic information systems have been a health system strengthening investment in some countries. Manual systems of recording discharge abstracts for hospitalized patients, and then computerizing them, have been used. To a much lesser extent, similar manual-to-digital systems have been installed in LMICs for ambulatory care settings.

Developing an integrated national EMR is much more ambitious and expensive. The value of these systems, and the first generation manual-to-digital systems, is to improve clinical performance of the system. To prompt providers, to allow exact and quick order entry by providers, to eliminate untoward variations in behavior across providers, These are important. And they are expensive investments, in both technologic costs and in gaining physician compliance.    They may be investments donors want to make, but they don’t seem as urgent as some of the other data sources. These investments are

7. Manual-digital hospital discharge abstracts

8.  Simple EHR for primary care (interprovider comparisons and motivation)

9. National EHR system

What about point of service transactions data (eg administrative data, as it was always called here in the U.S.). Insurance operations, where it occurs, does produce very useful data for evaluating the health system. Utilization of services is well tracked by patient, and can be combined with the insurance eligibility data and provider data to create high value data sets on large numbers of people. (capable of detecting small impacts and changes).

LMICs usually have substantial numbers of health system encounters that do not involve an intermediary, like an insurance company. This role of someone paying a bill “on behalf of the patient” creates the need for an administrative record of service and the amount due. LMICs often have many transactions where out of pocket payments occur (no insurer) or occur in government facilities (where an eligibility card may need to be shown, but no “bill” is generated. So, we do not list this kind of record here.

 

A Data Analytic Organization

Data resources need to be complemented by the skills and motivation to use it effectively to evaluate the performance of the health system, and whether health policy is working as intended. Experience shows that skills in project design, analysis of data and report writing are probably far more prevalent in LMICs than the demand for such information in the MOH, and the availability of organizations charged with getting answers to such questions.

So, what is needed. There needs to be created a special unit within (or outside) government, that has the responsibility for producing an annual report that is a professional, and data driven, that assesses the performance of the country’s health system, and the effectiveness of the major policies taken by government to improve that performance. This could be called “an economic evaluation or analysis unit” or a “health economics unit” , or a “health system M&E unit”, or whatever. It could be inside the government, or in a school of public health. In the U.S. the unit is called MedPAC, and it reports directly to the Congress, rather than the Administration. Without a specially designated analytic unit, the necessary skills cannot easily and promptly be assembled, the data resources cant be negotiated and pulled together, and the primary objective becomes always too much, too fast, too expensive and it just never happens. An organization with mission needs to be established, given resources, and politically insulated so that it can do the job.

What would such an organization need to do:

• Produce an annual report on the performance of the health system and the performance of the major health policies of government

• Hire and manage all health economics analysis capability
• obtain ready access to all necessary data sets (including NHA) through collaborative agreements between stakeholders ‘Sign memoranda of understanding to share data across key stakeholders including MOH units, MOF, statistics agency, academic organizations

• Be able to integrate data sets to best analytic advantage using a staff of SAS/STATA-capable data managers and programmers

• Conduct special health economic support studies for the Government or Donors or others
• Maintain professional production discipline for producing reports

• Create stakeholder advisory group or some other mechanism to create data sharing incentives and Conduct workshops and training programs for stakeholder organizations and other officials on the methods and findings

The critical success factors in doing this would seem to be

• Independence from political forces and interests
• Excellent Leader with Technical & Project Management Skills
• Staff mix:   health economics training, experience with statistical computing,  and with report production
• Governance:   Multi Stakeholder & Transparent
• Regular Reporting Schedule
• seat at the table for decisions about new investment decisions about health data

The MedPAC agency in the U.S. has been limited to 25 analysts (and about 35 employees) for a long time. It is a highly technical and authoritative evaluator of health system performance.