Posted on

How to gauge the impact of empathy-building initiatives

RabttSession.jpg
(Image courtesy of Rabtt)

By Haseeb Sher Bajwa

Pakistan and prejudice

Prejudice towards women and minorities is deeply entrenched in Pakistani society. Education plays a vital role in perpetuating these prejudices through one-sided narratives and a failure to inculcate adequate critical thinking skills for students to objectively analyze the information they receive. This results in a society comprised of adults heavily prejudiced towards women and minorities that lacks the analytical skills to change those prejudices.

Several social enterprise organizations, such as Rabtt, The History Project and Ravvish, have emerged in the last five to six years to incorporate empathy and critical thinking into education. Where some use a holistic approach to engender these characteristics, such as teaching courses in various subjects that range from world history to public speaking, others use a more targeted approach of teaching special courses that offer multiple perspectives on history. As these firms iteratively improve how they teach empathy, it is equally important to develop adequate measurement techniques to gauge empathy levels in children. Without them, they cannot ensure whether children are becoming more empathetic critical thinkers. Benchmarking students’ performance using tools designed specifically for gauging empathy would also legitimatize their efforts in front of donor organizations.

For six months, I worked with Asad Liaqat, a PhD candidate at Harvard University, to develop surveys measuring empathy in children for The History Project, a history education enterprise. We hope that other enterprises can use these tools to develop their own surveys that gauge empathy in children.

Using the hammer: Interpersonal Reactivity Index

The Interpersonal Reactivity Index (IRI) is an empathy-measuring tool consisting of 28 questions that are divided into four subscales: perspective taking (PT), fantasy scale (FS), empathic concern (EC) and personal distress (PD). Each subscale consists of six to seven questions that ask students how much a statement describes them as a person. Student scores are then calculated for each subscale both before and after the workshop. Previously, IRI has been conducted with various groups of people including adolescents as well as adults.[1] And in all cases, IRI seems to show consistent internal validity, making it less prone to error.

Our application of IRI in low and high income schools in Punjab followed a similar pattern but excluded the PD scale, since it is the only scale in IRI that does not give consistent results. We conducted a pre-post intervention, in which students took the IRI survey before and after the workshop. We carried out the survey with roughly 650 students, which was sufficient for our purposes. We did not see any significant improvement in children’s empathy levels.

While IRI is currently being used by firms like Rabtt to evaluate empathy levels in children, they are running into sampling problems and are not yielding significant results. Their student sample is too small to give statistically significant results even if their workshop increases empathy in children. In addition, they have not excluded the PD scale from IRI which may also distort the consistency of their results.

Using the scalpel: Implicit Association Test

A problem with IRI is that respondents often answer questions for a desired outcome, rather than provide an honest, accurate reflection of their biases. The Implicit Association Test (IAT) tries to solve this problem by measuring the implicit bias of people towards various groups in society.

IAT runs like a computer game in which negative and positive words appear on the left and right hand sides of the screen. Respondents allocate words pertinent to separate ethnic groups to either the positive or negative side and the computer records the respondent’s time for making each word association. The bias is then calculated by the difference in average time it takes for respondents to associate bad words with their own ethnicity and the average time it takes for them to associate bad words with other ethnicities. If they are much faster at making negative associations with other ethnicities than their own, then bias towards other ethnicities exists. Multiple studies using IAT show its internal validity as an effective tool for measuring implicit prejudice.[2] IAT can be used by social enterprises in Pakistan to gauge the baseline prejudice of students against minorities.

We used IAT in high income schools in Lahore to gauge students’ biases towards Indians. The game included a list of 10 Pakistani and 10 Indian words that students had to assign to either the positive word category or negative word category. As expected, we found a significant bias towards Indians, which did not decrease significantly after the workshop. However, due to the small number of students who took IAT, we cannot be conclusive about these results.

Conclusion

In 2013, the National Commission of Justice and Peace issued a report saying that:

“Over the past 33 years (1970 – 2013), at least eleven different governments [in Pakistan] have come up with their national educational policies, education sector reform action plans, policy review teams, and a whole host of white papers. Somehow, statements of prejudice, descriptions of biases, bulletins of violence, and cannonballs of hate have remained in every educational policy, every education sector reform action plan, every policy review team and every white paper.”[3]

Prejudice towards minorities is deeply entrenched in Pakistani society and efforts to change these attitudes needs to start in schools. Fortunately, quite a few social enterprises in Pakistan have realized this issue and have started developing curricula to reduce existing prejudices. However, it is crucial for these organizations to develop tools that measure the empathy they are trying to build in students. Otherwise they could fall into the same trap as most government organizations in Pakistan that are not only unclear about the exact nature of their service delivery, but also do not have any tool to measure the performance of their services.

Haseeb Sher Bajwa graduated from Swarthmore College with a double major in Economics and Political Science. He is currently working on a project with the Institute of Development and Economic Alternatives. He can reached at haseeb.sher.bajwa@gmail.com. 

[1] See Katherine Péloquin, Marie-France Lafontaine (2010), Ana Maria Fernández, Michele Dufey, and Uwe Kramp (2011).

[2] Baron & Banaji, 2006; Craeynest et al., 2005; Dunham, Baron, & Banaji, 2006; Skowronski & Lawrence, 2001

[3] National Commission for Justice and Peace, “Education vs Fanatic Literacy: A Study on the Hate Content in Textbooks in Punjab and Sindh Provinces”, March 2013.

Posted on

The six biggest challenges facing Pakistan’s urban future

karachi_pakistan_2010-01-08_lrgHina Shaikh and Ijaz Nabi

Pakistan is among the most urbanized countries of South Asia. As challenges mount, urban planning is gradually finding space in the policy discourse. This is the first of three blog posts on Pakistan’s rapid urbanization. It discusses the pace of urbanization and the major problems associated with it. This will be followed by posts on how the government is responding to the challenges and how and whether the research community is engaged in seeking solutions.
 Continue reading “The six biggest challenges facing Pakistan’s urban future”

Posted on

Understanding and responding to disadvantage in education

8401422071_6e59af4c92_o.jpg
(Image: DFID – UK Department for International Development)

Rabea Malik

Recent policy reform efforts in Pakistan have given immediate priority to the enrollment of out-of-school children, retention of those at high risk of dropping out and ensuring learning gains for those in school. Data are required for understanding who is benefiting, who is excluded, links between teaching, learning and disadvantage and ways in which reforms impact classrooms, schools and communities.

Children from poor households – particularly girls from poor households – and children with disabilities are much more likely to be at risk of dropping out or never entering school at all. Children from poor backgrounds are likely to face considerable learning challenges while they remain in school.

Also high on the policy agenda is teacher effectiveness, which plays a crucial part in increasing student enrollment and ensuring classrooms are inclusive of all backgrounds. Policy research over the past couple of decades has made it clear that experience and qualifications alone cannot explain why some teachers are more effective than others. Pedagogical expertise (classroom practice), teacher attitudes and motivation matter most for student learning and engagement. These are also areas with the most significant data gaps.

Current sources of data and gaps in data

A number of survey-based and administrative data sources are available and being used for monitoring teaching and learning, and in some cases for policy responses. A mapping of existing data sources, undertaken as part of the background work for Teaching Effectively All Children (TEACh), reveals a long list of sources with information on relevant variables:

Surveys Gender Income/wealth Disability Learning Teacher experience /qualifications Teacher attitudes /practices
Pakistan Social &Living Standards Measurement (PSLM) ✔
Household Integrated Economic Survey (HIES) ✔ ✔
Neilsen survey (McKinsey) ✔ ✔
Punjab Examination Commission (PEC) ✔ ✔
Directorate of Staff Development (DSD) ✔ ✔
Education Management Information System (EMIS) ✔ ✔
Pakistan Poverty Alleviation Fund (PPAF) ✔ **
Learning & Educational Achievement in Punjab Schools (LEAPS) ✔ ✔ ✔ ✔
Annual Status of Education Report (ASER) ✔ ✔ ** ✔
 Multiple Indicator Cluster Survey (MICS) ✔ ✔
Early Grade Reading Assessment (EGRA) ✔ ✔ ✔
Teaching Effectively All Children (TEACh) ✔ ✔ ✔ ✔ ✔ ✔

Note: ** Surveys piloted tools for identifying disability

The mapping also reveals some gaps. Sources that collect information on individual/household socio-economic characteristics (such as Pakistan Standards of Living Measurement Survey (PSLM) or Household Integrated Economic Survey (HIES)) do not collect information on learning levels. Administrative data sources that provide information from large scale standardized learning assessments do not provide student background information (such as Punjab Examination Commission results for 3rd, 5th, 8th and 10th grade, and the more recent Sindh Assessment Tests, as well as the monthly assessments undertaken by the Directorate for Staff Development in Punjab). Those that collect information on both learning and disadvantage (such as LEAPS, ASER), do not collect data on teacher attitudes and pedagogical practices.

The biggest gaps emerge for data on disability (inside and outside classrooms), information on teachers’ attitudes (regarding children with disabilities, those from poor backgrounds and slow learners), their level of preparedness for identifying and managing diversity in classrooms and the practices they undertake. Policy research on themes of learning, teaching and disadvantage can potentially bridge these gaps: proving missing data, and instruments for collecting data that can improve the effectiveness of policy responses.

Using data for policy impact

The silver lining is that policy makers today see the benefits of having information available to them for planning and policy design – a preference reiterated during a policy dialogue held recently in Lahore. They’re keen to – and in some ways they have already – put in place mechanisms that make available data on enrollment, teacher attendance, student attendance, school expenditures, etc. Digital dashboards highlighting key indicators and road maps tracking progress across regions are examples of such mechanisms.

However, steps can be taken to improve the effectiveness of these foundational measures. Chief among them are: 1) ensuring the right indicators are tracked, 2) data are being utilized by empowered local agents, rather than being retained centrally, and c) data help form a feedback loop between policy and practice, rather than being used exclusively for high-stakes assessments (when teachers’ promotions and appraisals are linked to trends in enrollments and assessments).

The systemic reforms needed for effective intervention

Currently, learning assessments are undertaken every month in government schools to inform teacher appraisals and school rankings by comparing students across schools, communities and regions. These comparisons do not consider student background or information on other challenges the child might be facing. Local education departments currently do not track children at risk of dropping out or those who need special attention in schools. A mechanism for systematic identification of learning challenges and other disabilities at scale is currently not in place.

Assessments can be restructured to track improvements in the same children over time. Information on teacher attitudes and practices, as well as the challenges they face, can inform the design and focus of support mechanisms (such as in-service training and school resource decisions). Internationally developed and validated survey modules that help identify the nature and severity of disability for all children in and out of school can provide accurate numbers for planning.

The ultimate goal should be to use the information to inform change in practice. An effective policy-practice feedback loop requires making information and authority to act on information available to local decision makers. In other words, it is important to make sure the data are used in a way that empowers teachers and school leaders, and enables them to change their practice willingly rather than being utilized for high-stakes accountability.

Rabea Malik is a research fellow at the Institute of Development and Economic Alternatives (IDEAS).