The PISA test, administered every three years to 15-year olds in 72 economies, invites comparison of U.S. students and schools to those in other countries. PISA scores are "used by educators, politicians and policy analysts to provide broad prognostics on what is wrong or right in the U.S. educational system and elsewhere," Levin writes. But, he adds, "if past versions of PISA are any indication, much of the conclusions and comparisons will be misguided.”

Historically, U.S. results have been about average among developing countries and well below Asian countries such as South Korea, Japan, Singapore and the city of Shanghai, as well as below Canada, Finland and the Netherlands. The release of PISA scores typically has led to "hand-wringing" in the U.S. on the causes and consequences of these disparities, as well as suggestions or mandates for improvements, Levin writes.

"PISA studies are careful and scientific, requiring representative samples of students," writes Levin, who has served as an educational consultant to the Organization of Economic Cooperation and Development (OECD), which develops the test. But while the PISA scores often provide "insights and detailed data" that can be helpful, "there are dangers in believing that PISA scores can be raised by simply adopting the practices of the high-performing countries." This is because "the statistical approach of PISA cannot provide causal links between school differences and student performance, only descriptive information ...  on education." PISA is "not a statistical study that isolates cause and effect."

He lists other reasons why PISA falls short in diagnosing the reasons why performance varies from country to country:

• It does not measure government assistance for poor families that accounts for student success.

• School data is collected only on a student's present school and does not measure the characteristics or results of previous schools attended.

• Korea's exemplary results may not reflect schools as much as outside assistance such as after-school programs.

• It does not measure the soft skills acquired in school, such as collaboration and persistence, which have been identified recently as contributing as much to employment and earnings as test scores, nor does it assess the skills that enable one to be effective as citizens – a major goal of schools.

The quality of an educational system can be based partially on its achievement scores, but comprehensive comparisons require a much deeper evaluation of causes and effects as well as performance on broader purposes of education," concludes Levin, who will be writing periodically for the "Best Countries" website of US News & World Report.

LINK: PISA 2015: A Misleading Test?

Published Thursday, Dec 1, 2016