Skip to content
An official website of the OECD. Find out more
Created by the Public Governance Directorate

This website was created by the OECD Observatory of Public Sector Innovation (OPSI), part of the OECD Public Governance Directorate (GOV).

How to validate authenticity

Validation that this is an official OECD website can be found on the Innovative Government page of the corporate OECD website.

Hearing the People: Computer-Assisted Telephone Interviewing to Evaluate Public Services

Hearing the people's demands and points of view, through reliable sources, is a central challenge to governance in the 21st Century. With no additional costs, we use the government's call center, representative scientific samples of the population, and up-to-date computer-assisted telephone interviewing to uncover popular evaluations of the quality of public services in Brasilia, Brazil.

Innovation Summary

Innovation Overview

After all, representation and responsiveness are in the essence of democratic governance. Hence, what the citizen thinks, wants, and how he evaluates the quality of public services and public policies is key to the success of a government and the stability of a political regime. In Latin America’s young democracies, facing intense episodes of crisis in the past 30 years, this is especially true. With the purpose of creating a communication channel between the population and government, Brasilia’s Planning Company (CODEPLAN) developed a strategy and the technology to bridge the gap between decision-makers and citizens, focusing on hearing what citizens have to say about policy, institutions, and public agents.

Using the government’s Call Center, with no additional contracts or increase in expenditures, we rely upon up-to-date Computer Assisted Telephone Interviewing of representative samples of the Federal District’s population to evaluate and monitor the impact of public policies, the quality of public services, and citizens’ views on policy proposals. Since its implementation in October of 2015, the project has conducted 16 surveys with samples of the population ranging between 2000 and 4000 respondents. We conducted over 40,000 interviews based on scientific samples and questionnaires designed based on principles of cognitive science.

We are putting social sciences to improve the government’s openness to society. The primary beneficiaries are other governmental agencies interested in learning about how citizens evaluated services delivery or seeking to understand popular perceptions about policy proposals. We studied diverse themes, including:

a) Quality of public services on transportation, health, education, and public safety

b) Views on land tenure (a major issue in Brasilia)

c) Evaluations of the Urban Cleaning Service (SLU) and of the Land Fiscalization Agency (AGEFIS)

d) Perspectives on the adoption of paid parking lots

e) The quality of buses and the use of distinct modes of transportation

f) Evaluations of the quality of life in specific neighborhoods

g) Victimization panel

Partners in the government, our direct beneficiaries, include the Secretary of Public Safety, Secretary of Urban Mobility, Secretary of Territory Governance, Secretary of Cities, SLU, AGEFIS, Urban Transportation Agency, and The Governor’s Special Projects Office. We have four more projects enlisted for the last months of 2017, including one on water consumption for the Water Regulatory Agency (ADASA), another wave of interviews on the victimization panel study for the Secretary of Public Safety, evaluations of local administrators, a request of the Office of the Governor, and a round of evaluation of public buses.

Partners have reacted positively to the initiative, indicating the usefulness of surveys to reassess their intervention strategies, reevaluate their policy programs and proposals, and rearrange their priorities. Ultimately, our most important beneficiary is society. Most of our research reports are made public in CODEPLAN’s website, providing more information to society, empowering citizens through knowledge, and increasing their capacity to hold politicians and bureaucrats accountable for their actions in office. Hence, our telephone-interviewing project is a form of active transparency, informing citizens about the quality of policies and services.

This innovative tool uses the government’s Call Center to actively seek citizen’s opinions and evaluations. Calls are made during moments of inactivity of the Call Center, with no interruption of other services. Our telephone catalog covers all regions in the Federal District, including over 4 million numbers. We use Probability Proportional to Size samples of the population in all 31 Administrative Regions in the Federal District. Hence, we attain significant variation in income and educational levels, age, gender, and region of residence. We validate the socioeconomic and demographic data of our telephone surveys with external and institutional data sources.

When necessary, weights are attached to correct datasets for biases. Finally, our reports include both descriptive as well as inferential statistical analysis, using data visualization techniques to facilitate comprehension of the findings. Hence, we follow a strict scientific protocol in collecting and analyzing data. Finally, it is fundamental to understand that this project does not incur in any additional costs to the government. We maximize and optimize existing contracts that support the call center to active seek citizens’ opinions and evaluations. Hence, the project increases the efficiency of the Government’s Call Center, providing a new service that was absent in the past. It does so increasing the government's levels of information about what citizens think.

Innovation Description

What Makes Your Project Innovative?

CODEPLAN’s project of hearing the people using government installed Call Center infrastructure and scientific survey research techniques is the first of its kind in Brazil and Latin America. Its major difference from prior uses of Government Call Centers and Ombudsman services is that we actively seek the populations’ opinions about specific topics, instead of passively waiting for their criticism and demands.

Traditionally, Ombudsman services receive criticism, requests, and suggestions and distribute these to the responsible government agencies. This is, no doubt, a very important use of government Call Centers. Still, such services have biases towards over-representing those more inclined to contact the government. Certainly, higher educational levels and income, on one hand, or populations from deprived areas on another, could probably be more inclined to contact Ombudsman services. These groups are not necessarily representative of the entire population in a city. Our project improves the use of Government Call Centers by actively seeking the opinions and evaluations of representative samples of the population, covering the entire territory and avoiding selection bias.

Given the overwhelming increase in the coverage of mobile and landlines in the Federal District and that 97% of the population lives in urban areas, we are able to reach a much broader audience, avoiding hearing only subsets of the population with specific inclinations. In addition, we utilize questionnaire design techniques that further limit biases on question wording or order, aiming at capturing interviewees’ true and honest opinions. We follow research ethics procedures, assuring anonymity and voluntary participation in the surveys, which also increase data reliability. Hence, the quality of data, its reliability, validity, and representativeness are trademarks of our project, advancing upon other sources of data collection in the government, especially about how citizen’s think.

What is the current status of your innovation?

The project “Hearing the People” aims at increasing the government’s levels of information about popular evaluations of public policies and services. The main problem is to reduce the distance between representatives/bureaucrats and citizens through the collection of reliable, valid, and representative data on citizen’s opinions and views about the quality of service provision. Through this mechanism, we aim at assisting other governmental institutions in improving the quality of public services and policies, accelerating the feedback loop in the policy cycle.

The problem of low-quality services and policies is rampant in the developing world, due, in part, to governments’ inability to consult the population about their demands and perspectives. The disconnect between decision-makers and population has become even more flagrant in Brazil after the massive public demonstrations in 2013, in the wake of the Confederations Cup. Bridging this gap is imperative to improve the quality, efficiency, effectiveness, and efficacy of public policies and services. The traditional mechanisms to hear the people, such as direct forms of democracy including public hearings, committees, referendums, and plebiscites are expensive, time-consuming, and rare. Not to mention that they may suffer from serious selection bias: those who chose to participate are not representative of the entire population.

A way around this is to contract private sector polling firms to conduct household, face-to-face or telephone surveys. Still, this requires additional expenditures and contracts, making the process costly and intermittent. To overcome the limitations of public consultation, we identified the existence of the Telephone Call Center infrastructure in the Federal District Government and, using our knowledge on sampling and questionnaire design developed the innovative idea of employing this installed equipment to hear the people efficiently, frequently, and scientifically. We can conduct around 4000 interviews in a period of 10 working days and produce a report in under 30 days, from the first meeting with our partner to the publication of the final report.

We produce expedite, reliable and valid informational feedback to governmental agencies. None of the existing forms of direct, participatory democracy possesses the same capability. We, therefore, maximized the utility of the Call Center, optimizing its use to evaluate public policies and services from the people’s perspective. The Government’s Call Center contains 178 operating posts used to operationalize several distinct services for the population. We use it in the off-hours to conduct computer-assisted telephone interviewing of the population. Hence, other services are maintained. We put the existing contract to its maximum use. Our business model is simple, with little bureaucracy.

Our partners show interest in conducting the policy or service evaluation through a formal letter. We organize meetings to define the content of the questionnaire, with intensive participation of the partners. In parallel, we select the sample using scientific calculations. Subsequently, telemarketing professionals at the call center receive training to conduct interviews with the selected sample. Finally, we elaborate and present the final report based on the data collected. This is the traditional model of survey data collection and analysis used by polling firms and academia. We employ these well-known techniques to answer governmental agencies’ questions about evaluations of public services.

We have evaluated the performance of the project by requesting our partners to assess the quality of the data collected and its impact on their service provision. Feedback indicates that the data was useful to reevaluate the offering of some services, improve agencies' websites and communication strategies, rearrange organizations' lists of priorities in attending clients’ requests, and assisting in reconsidering the policy proposals of government institutions. In spite of the diverse uses, all concur that the quality of the data and the rigorousness of collection methods are undeniable.

We are now at the moment of diffusing lessons about our innovation. One such way is participating in dissemination activities such as enlisting our project in initiatives such as the Edge of Government. We have been classified as a finalist in Brazil’s National Public Administration School (ENAP) Innovation Award, a great opportunity to showcase our project nationally. We want to make our project known. In a second phase of the diffusion process, we hope to establish collaborations with other local, state, and national governments in scaling up our project. The potential for diffusion is considerable, as long as the call center technological solutions are available. We offer technical assistance to capacitate human capital to conduct scientifically valid telephone surveys of the population.

Innovation Development

Collaborations & Partnerships

This innovation gained significant contributions from partners. To start with, we collaborate with partners to design questionnaires. The specialists on each specific theme are the governmental agencies we cooperate with. They are the ones who know which questions to ask. We assist in asking the questions in an unbiased format, avoiding distortions that may influence the responses we receive from citizens. Hence, we apply existing knowledge about questionnaire design, based on sociology and cognitive sciences, to improve the questions we ask. In doing so, we have learned tremendously about distinct issues in government, which is vital for the production of the final reports and recommendations we make on each specific topic. In addition, we work very closely with the subcontracted call center firm, especially in training interviewers and inputting questionnaires into their computational system.

Users, Stakeholders & Beneficiaries

An important contributor to this project is the subcontracted call center firm. They have been helpful in defining interview procedures, assisting in creating data entry interfaces, and pretesting questionnaires. Furthermore, the firm has assisted in cleaning the telephone number catalog by automatically verifying which numbers are valid and in use. This allows for increases in response rates. Furthermore, the firm hires highly experienced and capable telemarketing specialists trained specifically to conduct the interviews, improving data quality.

The partnership with and commitment of the private firm subcontracted to operate the government’s call center is fundamental for the project’s success. Finally, governmental agencies’ interest and courage in being evaluated and in understanding the efficiency of this modality of evaluation is what makes the project viable.

Innovation Reflections

Results, Outcomes & Impacts

Our main product is the provision of qualified information about citizens’ perspectives on policy and service quality. Through information, we believe governments can improve their performance and better respond to society’s demands. The informational goods we provide are unique and exclusive in that they are not available from any other source. The public opinion data we collect in an expedite way is also scientifically valid and reliable. Finally, we obtain data with no additional costs to the government.

The main results of the innovation are the production of datasets and reports based on Computer-Assisted Telephone Interviews with citizens of the Federal District in Brasilia, Brazil. The focus is on popular perceptions and evaluations of the government’s performance in the provision of public services and the implementation of public policies. We produced 16 reports since the project was launched in 2015, assisting over 10 governmental agencies in improving the quality of their services and policy programs as well as the representativeness of their policy proposals.

These reports are available online for the Brazilian society to follow and learn about the performance of the Federal District’s Government. An additional feature of our initiative is its contribution to empowering citizens through knowledge of governmental performance. Our partners in the government have adapted, rescaled, and reevaluated their portfolio of services based on the public opinion data we provide. Furthermore, some use our findings to design new policy proposals and services. We hope to use the innovation in the future to continue unraveling the citizens’ views on new, emerging topics and trends, assisting the government to learn about complex social and economic issues through new, updated, and unique information.

Challenges and Failures

The greatest challenge of the project is the construction of the telephone catalog with large coverage of the Federal District Territory. We went through several iterations using distinct registries of telephone numbers to compile the omnibus file we currently employ. The catalog in use is based on the list of possible telephone numbers according to rules provided by the National Telecommunication Agency (ANATEL) given local and area codes. In addition, we validate each number through automated dialing machines, constructing a final list of active numbers adding to over 4 million entries. In this way, we assure full coverage of the Federal District’s possible land and mobile lines. Therefore, the solution for the creation of the telephone catalog is one that can be implemented in all national territory, increasing the potential for scaling-up the project.

Conditions for Success

This initiative requires supporting infrastructure and services, in the form of a governmental call center, accompanied by contractual rules that allow for the conduction of active phone calls to citizens. It requires well-specified contracts that provide for sufficient numbers of attendants to conduct interviews and foresees the possibility of not just receiving phone calls, but also placing them. In addition, the presence of human resources specialized in social research, in particular, survey research, sampling, policy evaluation, and questionnaire design is necessary for the success of the project. However, technical assistance from Codeplan can easily qualify human resources to implement similar initiatives elsewhere. In fact, this is a goal of our diffusion strategy.

Replication

The problem the initiative confronts, the informational limitation of governments on policies and services evaluations, is a widespread public challenge. Governments everywhere want to understand how citizens evaluate the quality of public services and public policies as a means of improving their performance and the offer of outputs to society. Collecting public opinion data expediently, with low costs, and frequently is how our initiative innovates and advances upon existing alternatives.

The solution we propose is easily replicable once the required physical and legal infrastructure is available. However, it is common for governments in localities with significant populations and complex problems to hold call centers. Capacitating local human capital, another necessary step, is possible through the transfer of knowledge from the experience developed in Brasilia.

Lessons Learned

The major lesson from our experience is to use available resources – human, infrastructure and budget – to their fullest. In moments of scarcity, it is vital to do more with what is available or even with less. The Federal District today faces one of its most serious fiscal and economic crises, with severe cutbacks on governmental expenditures.

Efficiency - achieving desired results by optimizing costs – is key to the continuation of any policy. Our initiative moves the State in such direction, innovating by using technology to open a new channel of communication and information between State and society. A possible setback of the project is that hearing the people may cause discomfort to politicians and bureaucrats when evaluations are negative. It is important to bear in mind that the opposition and the media may use the information against the government. A solid communication strategy for data dissemination should accompany similar efforts elsewhere.

Anything Else?

This innovation, like any other, is a consequence of the dedication, commitment, and vision of public servants concerned with increasing policy efficiency and improving the quality of public services. It is the fruit of individual motivation. Stirring innovation, however, is also a consequence of the environment one is in. The current Federal District Government is fully committed to innovation in the public sector: it is part of the government's deliverables to society. Therefore, this innovation, ultimately, is only possible because of the combination of these two factors.

Status:

  • Diffusing Lessons - using what was learnt to inform other projects and understanding how the innovation can be applied in other ways

Innovation provided by:

Date Published:

24 May 2017

Join our community:

It only takes a few minutes to complete the form and share your project.