Measuring the un-measureable

As part of the organizing team of the Inaugural CIES Symposium in Scottsdale, AZ this past November, we were thrilled to continue the debates about Global Learning Metrics (GLMs) at the recent CIES 2017 Conference in Atlanta, GA. CIES 2017 included a number of Presidential Highlighted Sessions. Among these was “Measuring the un-measurable in Global Learning Metrics” featuring  four scholars: Karen Mundy (University of Toronto & Global Partnership for Education), Jill Koyama (University of Arizona), Aaron Benavot (University at Albany & UNESCO), and Gustavo Fischman (Arizona State University). Moderated by Iveta Silova (Arizona State University), the participants in this session discussed the merits, possibilities, tensions, and obstacles of GLMs and their influence on education, pedagogical practices, and policy.

Questions such as “What is usually dismissed as too difficult to measure?” and “Should global learning metrics reflect learning outcomes beyond basic numeracy and literacy?” guided the beginning of the debate. Later, Silova challenged the panelists to explain how GLMs could be transparent and explicit in making their underlying epistemologies visible. Silova also asked the panelists to explain their views on whether or not GLMs could originate from the Global South.

Video: Watch the full Presidential Highlighted Session.

In this Presidential Highlighted session, there was neither a consensus about the meaningfulness of current globally linked measures of learning outcomes nor was there a clear vision of how they will evolve in the future. Additionally, it was obvious from both the symposium and the CIES 2017 Presidential Highlighted Session that educational stakeholders disagreed on the purpose of a GLM–for some, it was to be used as a simple tool, helpful for gaining data on student literacy performance, while for others, it had been utilized as an overarching instrument that had been abused to rank countries in ways that did not account for contextual differences.

While the Presidential Highlighted Session at CIES 2017 in Atlanta unsurprisingly concluded without a consensus about GLMs, the four scholars brought attention to ideas to consider when examining the merits and disadvantages of creating and relying upon GLMs. Silova explained in her introduction to the debate that GLMs will continue to expand, and Fischman later expressed concern that GLMs, despite their decontextualized nature and limited capacity to measure what is happening in classrooms, have already driven educational policy and reform efforts. In light of the growth of these metrics and to understand their effects, addressing issues of feasibility is essential.

One of the salient themes in the debate focused on who is involved in the creation of GLMs and how GLMs are and will be applied. While Mundy argued for reinforcing national ownership of metrics, Koyama claimed that linking nations by a single global learning metric was not something that the policy and development communities should be working toward. Perhaps there was some overlap between the two sides, however, when Koyama suggested that because metrics do exist, there should not just be a single one. Rather, diverse and contextualized measures are essential so that the local needs are taken into account, and, for Koyama, these metrics should be applied from the bottom up. Koyama cautioned, however, that the data generated by any metric is not the solution. She also raised the ethical issue of authority when she asked about who gives education stakeholders the power to measure learning outcomes. Koyama problematized the use of nuanced metrics to “rank and diminish some [countries] while elevating others.”

Benavot agreed with Koyama’s argument that metrics should not follow a top-down approach. He concurred with Fischman that metrics should be meaningful and advance pedagogy. However, Benavot observed that the current metrics are not used to inform quality instruction at the classroom level. Instead they are used for policy-making and the distribution of resources. While Mundy argued that GLMs should be used for such distributive justice, Benavot disagreed that a single, cross-nationally linked metric was the correct avenue. He argued that focusing on GLMs distracts from what is happening in classrooms. According to Benavot, if metrics are to be linked, they should be emergent and develop from the local levels, then to the regional and national levels, rather than imposing them from the top down. Despite their dispute about the goal of GLMs, Mundy agreed that metrics should be grounded in national capacity building.

Where does this leave us in the GLM debate? What has been accomplished by holding a symposium dedicated to this topic in November and a follow-up highlighted session four months later? For one, it opened up a dialogue, providing educational stakeholders opportunities to define what they mean when using the term “global learning metric.” It also helped bridge a gap between academia and development to move the conversation in new directions. At the end of the Presidential Highlighted Session, an audience member and Benavot both raised an interesting question that might offer a different angle from which to approach the debate: Is there a universal learning progression that can be defined and measured?

The Possibility and Desirability of Global Learning Metrics

In November 2016, the Center for Advanced Studies in Global Education (CASGE) and edXchange at Arizona State University hosted the Inaugural Symposium of the Comparative and International Education Society (CIES) in Scottsdale, Arizona. The Symposium addressed the theme “The Possibility and Desirability of Global Learning Metrics: Comparative Perspectives on Education Research, Policy and Practice.” The Symposium brought together academics, practitioners, policymakers, educators, and social activists for an alternating series of keynote plenary debates and parallel sessions about the desirability and feasibility of global learning metrics. The participants came from 61 institutions, 17 different countries, and 17 states within the United States.

We selected global learning metrics as the focus of the Symposium because it is a timely and increasingly challenging educational and political issue at the center of multiple global debates about the future of education. Learning outcomes have recently been enshrined as central policy objectives in the new international education and development agenda. Unlike goals that seek to universalize access for education, for which consensus is strong, debates around learning are considerably more contested. Proponents argue that more robust global learning metrics have the potential to reduce academic disparities and improve learning outcomes for children across different contexts. Critics note that such universal measures typically focus on a narrow assessment of basic skills, while overlooking the importance of a more holistic approach to education, including human rights, aesthetics, morality, religion, or spirituality. Others call attention to the dangers associated with the emergence of the data-fixated punitive accountability regimes, privatization and marketization of public education, and a growing disconnect between systems, actors, and larger pedagogic changes. Some critics warn that global learning metrics can contribute to enacting hegemonic neocolonial globalization.  More broadly, the debate about the global learning metrics reveals an underlying tension in our field – a tension between the desire to replicate and scale up “best practices” (and an assumption that there is a global consensus on what constitutes “good” education), on the one hand, and the awareness about the importance of context, and deeply culturally contextualized education practice, on the other hand. Bringing a comparative perspective to the disjuncture between replicability and contextuality is one way our field can contribute to education research and practice broadly.

This raises the central questions, which guided the organization of this Symposium: Are global learning metric desirable and are they feasible? How can learning among children be measured and compared across diverse contexts and systems? Which learning domains should be assessed and why? How is learning revised or reframed for those who have less power or less “value” in the society in which they reside? How, if at all, are learning assessments actually used by governments, nongovernmental entities, teachers, curriculum developers, and other stakeholders? The Symposium brought together a group of researchers, policymakers, practitioners, and activists for a focused intellectual and policy engagement around these questions. While not designed to forge consensus or alignment, the Symposium was a step towards linking together academic research and policy debates in order to enable critical reflection, innovation, and proactive action in the area of developing global learning metrics.

The Symposium featured 4 plenary keynote debates, which addressed issues ranging from the desirability and feasibility of global learning metrics to their potential to be pedagogically innovative and culturally responsive. In addition, we offered 3 workshops and 65 parallel presentations a range of topics related to Global Learning Metrics. The plenary keynote debates were recorded and are available for viewing below.

Plenary keynote debate #1: Are global learning metrics desirable?

Moderated by: Iveta Silova

Panelists: David Edwards, Education International; Eric Hanushek, Stanford University; Silvia Montoya, UNESCO Institute for Statistics; Karen Mundy, Global Partnership for Education

Learning metrics of any sort are necessarily politicized, as they raise issues with clear philosophical, technical and policy dimensions. The first keynote session of the symposium set the stage for the debate by focusing on the different actors and rationales behind the development of global learning metrics. The guiding questions included: Are global learning metrics desirable and why? What are the end goals of global learning metrics from the panelists’ particular disciplinary and institutional perspectives? Why do we need GLMs at this particular moment? What are the main political challenges and opportunities? Who should be in charge of the development of global learning metrics and who should pay for it? What role should nation-states, international agencies, NGOs, teacher unions, academics, and other actors play in coordinating efforts to develop global learning metrics?

Video: Watch Eric Hanushek and David Edwards debate about why we need GLMs now.

Plenary keynote debate #2: Are global learning metrics feasible?

Moderated by: Gustavo E. Fischman

Panelists: Monisha Bajaj, University of San Francisco; Aaron Benavot, UNESCO Global Monitoring Report; David C. Berliner, Arizona State University

Developing learning metrics is a complex and contested enterprise. It is one of the biggest political, pedagogical and technical challenges of contemporary educational systems. The second plenary debate focused on the feasibility of the development of global learning metrics, addressing the following questions: How can we measure and compare educational achievement and outcomes across diverse contexts and educational systems? Can GLMs capture educational outcomes beyond the basic numeracy and literacy skills? What balance can be sought between the assessment of basic numeracy and literacy skills and the measurement of learning related to informational technologies, citizenship, human rights, sustainability, aesthetics, morality, religion and/or spirituality? In other words, how can we measure what it often pronounced as “too difficult to measure” though it is at the core of teaching and learning?

Plenary keynote debate #3: Can global learning metrics be pedagogically innovative?

Moderated by: Sherman Dorn

Panelists: Chris Higgins, University of Illinois at Urbana-Champaign; Radhika Gorur, Deakin University; Pasi Sahlberg, University of Helsinki

In the discussion of global learning metrics, the global data banks and the International Large-Scale Assessments (ILSAs) – so-called “big data” – have played a central role. While producing new information and important insights about educational systems and learning outcomes, the big data movement has serious limitations.The third keynote debate was structured around the following guiding questions: How well is big data suited to help us make decisions about improving teaching and learning in schools and classrooms? Do global learning metrics actually allow for pedagogical innovation or do they narrow pedagogical practices? What are alternative assessment and measurement tools that could complement global efforts of increasing educational access and outcomes, as well as improving teaching and learning?

Plenary keynote debate #4: Can global learning metrics be culturally responsive?

Moderated by: Gustavo E. Fischman

Panelists: Supriya Baily, George Mason University; Stafford Hood, University of Illinois at Urbana-Champaign; Hugh McLean, Open Society Foundations; J. Douglas Willms, University of New Brunswick

The concept of global learning metrics is based on the assumption that there is an agreement about what constitutes “good” and “quality” education worldwide. However, efforts to develop global learning metrics have often neglected the diversity of cultural contexts and educational systems. The symposium concluded with a debate about whether global learning metrics are culturally responsive. The panelists were asked to address the questions: Is there a global core of fundamental knowledge, skills and competencies that are relevant across different countries? How can GLMs capture the dynamics of race, ethnicity, class, gender, religion, and other factors that contribute to students’ cultural identities? More broadly, how can GLMs be more culturally responsive and relevant in the context of uneven power dynamics globally?

These four debates highlight some of the perspectives about the desirability and feasibility of GLMs, but there are many ways to approach the debate, and these diverse opinions will be featured in forthcoming blog posts.