Measuring the impact of research: Mining data to build partnerships and enhance student learning

“There's the physical space where you can measure items with all sorts of interesting instruments, but when it comes to what people are thinking and what they're learning, it's hidden,” said Dr. Saad Chahine, Associate Professor of Measurement and Assessment at the Faculty of Education, Queen’s University. “You can't open somebody's mind and weigh how much competence they have in a certain area. The most we can do is infer, and we draw these inferences based on a bunch of indicators we can develop. Trying to reveal what's hidden is fascinating to me.”

Revealing what’s hidden often begins with a deep dive into data.

Early on in his academic career, Saad joined a research team conducting a baseline evaluation of education in Trinidad and Tobago.

“That first trip really opened my eyes to how valuable the work I do could be. And that's what inspired me - making connections with people within the communities where change is happening,” said Saad. “Developing countries are looking for ways to change and improve. There's a real thirst for using data to make meaningful change.”

Nearly a decade later, he’s heading back to Trinidad and Tobago for a collaborative research initiative with the University of West Indies to develop a data-informed education prosperity index spanning early childhood to adulthood. Saad’s three weeks of on-the-ground research is supported by the Government of Canada’s Emerging Leaders in the Americas Program.

In time, the index might lead to policy or practice changes within the country, ultimately helping to build local capacity to enhance learning processes and outcomes for students.

“The starting point is mapping out what data exists and where the gaps are, then identifying what the strengths and weaknesses are, and then supporting educators to have really focused, classroom-based interventions,” said Saad.

His research will lead to a published paper and recommendations for Trinidad and Tobago’s Ministry of Education on what other data might be needed to develop a more holistic picture of the state of education in the country, as well as what interventions could be used to support student success and long-term wellbeing.

Saad explains a lot of countries and organizations have access to massive amounts of data that are not being looked at in novel ways. Through using different mathematical models and techniques, it’s possible to model data to draw new and meaningful interpretations.

“Those interpretations sometimes do lead to policy changes,” he said. “They often lead to educational changes.”

Framing the index and forming partnerships

In 2018, the Organisation for Economic Co-operation and Development’s Programme for International Student Assessment (PISA) redesigned their survey with an educational prosperity framework to measure both the factors that contribute to prosperity and what prosperity looks like when it comes to academic outcomes.

Through his connection with the International Congress of School Effectiveness and Improvement, Saad reached out to his colleague Dr. Jerome DeLisle at the University of West Indies. With the PISA framework as inspiration, Saad invited Jerome to consider collaborating on creating an education prosperity index using existing data within Trinidad and Tobago, birthing a new partnership between the education faculties at Queen’s and the University of West Indies.

By working together on developing the education prosperity index, the researchers hope to expand the memorandum of understanding that currently exists between Queen’s and the University of West Indies to include education. In time, the vision is for graduate students from the two institutions to work together and grow their understanding of different educational contexts.

“The way education works in one place doesn't necessarily work in another,” explains Saad. “Yet, it fascinates me that you can use the same measurement models to really understand very, very different contexts.”

The index will draw on data sources such as school attendance, academic tests, household surveys, and demographic information.

“Often, it's the stitching together of these larger datasets that we get a comprehensive picture of what's going on,” said Saad. “Each one of these datasets might not be huge on its own, but when you bring them all together, they form one massive data source.”

Saad said international development education research is a very consultative process involving local experts and governments.

“It's not like they send us the data and we tell them what's going on with their country,” he said. “Rather, we walk through it together and go back and forth to really understand what's happening. The understanding is often derived from their interpretation of the data because they have a clear understanding of their own context and history.”

Saad points to a prior project in Belize where he led the design of a data-informed language acquisition policy. There are five official languages in Belize, and test data analysis revealed stark gaps amongst students studying in different languages. Gaps were also evident across different regions.

These findings highlighted that student scores and performance are closely tied not only to their spoken language but the regional differences that exist, including economic differences. In Belize, the results of Saad’s research were adopted by the local ministry to support the development of their language policy, which spurred on other initiatives including a national curriculum redesign.

“When you're thinking about designing interventions and supports, they have to be multifaceted,” said Saad. “It's not only putting in place things within a region, but they also have to be contextually relevant to support both the language and the economic differences that could exist there.”

Support projects like this by contributing to our Community Initiatives Fund.