Last December PISA released the results of its 2018 international assessments.  PISA is the acronym for the Programme for International Student Assessment, which is developed by the Organisation for Economic Co-operation and Development (OECD) and administered to 15-year-old students once every three years.  In 2018 it was administered in 78 countries.  In the United States it was administered to 4,811 public and private school students from 162 schools.  Statistical techniques are used to extrapolate the scores of that sample to all of the more than 4 million 15-year-olds in the U. S.  Countries are ranked on the basis of standardized scores in the areas of science, math, and reading.  In 2018, the United States ranked 37^th in math, 18^th in science, and 13^th in reading.  Some have questioned the validity of these techniques and argue that the “true” ranking of a country can vary significantly from that reported by PISA.

The rankings for the U.S. haven’t changed much since PISA was first administered in 2000.  And every three years, the scores trigger a fresh round of handwringing and angst over the sad state of American education and the failure to improve.  This time is no different.  The headline in the New York Times read, “It Just Isn’t Working: PISA Test Scores Cast Doubt on U.S. Education Efforts.”  In the Boston Globe: “School Reforms Fail to Lift US on Global Test.”  And the LA Times chimed in with, “Why do U.S. schoolchildren underperform academically compared with students in other countries?”  Based on prior experience, the U.S. reaction to the PISA scores would probably have been more intense if the release of the scores had not coincided with the impeachment of the president and the daily diet of the latest news on Trump, Ukraine, Giuliani, Biden, etc.

And the U. S. is not alone.  For example, the Sidney Morning Herald wonders, “Should Australia Be in PISA Shock?”  And the Guardian reports, “UK school reforms to come under scrutiny as world rankings released.” 

Such is the demeaning power of PISA.  It has the ability to throw entire nations into doubt over the state of their education systems.  But is this power justified?  In a word, no.  Here’s why.

First, and most critically, any comparison requires the populations to be compared to be similar, but that’s not the case with PISA.  For example, the scores for China are based on only four provinces (Beijing, Shanghai, Jiangsu and Zhejiang), which are the most affluent in the country.  The remaining, lower-performing provinces are excluded.  The OECD permits this, even though it admits that affluence is positively correlated with test scores.

Similarly, Macau, which ranks third on PISA, is expected to become the richest country in the world (as measured by gross domestic product per capita) this year.  Singapore, which ranks second on PISA, is the third richest country in the world based on the same GDP per capita measure.  Singapore’s scores also reflect an education system that is based on rote memorization and test preparation beginning in preschool.  Most students in Singapore also receive several hours per week of private tutoring in addition to attending public schools.  Test scores are used to place students in one of four tracks in high school

In contrast, the U.S. has proportionately more students in poverty than most other post-industrial countries.  This brings the U. S. scores down, even though low-income students in the U. S. actually outperform their peers in other post-industrial countries.  They simply account for a higher percentage of the U. S. test taking population.  To make matters worse, there is some evidence that the U. S. oversampled low-income students in the past.

Participating countries also treat special needs students in different ways.  PISA has non-binding “guidelines” that govern the exclusion of students from the testing sample.  Students may be excluded if they have “functional” (i.e., physical) disabilities, “intellectual” (i.e., mental or emotional) disabilities, or “insufficient language experience.”  This further reduces comparability of testing samples, because counties decide for themselves which (if any) students to exclude and what criteria to use.  The U.S. includes special needs students in its sample.  These students comprise the lowest scoring subgroup on standardized assessments.

Valerie Strauss of the Washington Post (who I believe is the best education reporter in America) notes that in 2014 more than 100 academics from around the world called for a moratorium on PISA.  She writes that the reasons include: “PISA feeds into an overreliance on standardized tests and an emphasis on learning that can be easily measured and, some experts say, has major flaws with how the tests are administered, how samples of students are determined, and how some of the test questions are constructed.”

Strauss’s article (available here) includes a post by Yong Zhao, a Foundation Distinguished Professor in the School of Education at the University of Kansas, which summarizes common criticisms of PISA.  These include:

• The claim that PISA measures knowledge and skills essential for the modern society or the future world is not based on any empirical evidence. 

• PISA’s assumption that there is a set of measurable skills and knowledge that are universally valuable in all societies, regardless of their history and future is problematic. 

• PISA treats economic growth and competitiveness as the sole purpose of education.

This article is well worth reading and is a good start to understanding the problems with PISA.