Skip to main content

Advertisement

SpringerLink
Log in

Two decades of havoc: A synthesis of criticism against PISA

  • Published:
Journal of Educational Change Aims and scope Submit manuscript

Abstract

The Programme for International Students Assessment (PISA) has become one of the most influential forces in global education. The growing influence has been accompanied by growing criticism. For nearly two decades since the first round of PISA was conducted in 2000, the global assessment program has been roundly scrutinized and criticized by education researchers all over the world. But the mounting criticism seems to have had little impact on PISA’s influence as evidenced by its growing power in global education policy and practice. The lack of impact of criticism does not mean the criticism is not valid or PISA has improved. It simply means that the criticism has been largely ignored. The lack of impact is no reason to give up exposing PISA as a flawed business that has great power to misguide education. The expanding influence of the PISA enterprise makes it more even more important to be critical of this juggernaut today. It is also important to consider more effective and more straightforward approaches to present the criticism. The purpose of this article is to present a summary of criticisms that reveal the most fundamental flaws of PISA in non-technical language in one place. Specifically, the article focuses on criticisms of PISA’s three fundamental deficiencies: its underlying view of education, its implementation, and its interpretation and impact on education globally.

This is a preview of subscription content, log in via an institution to check access.

Access this article

Log in via an institution

Price excludes VAT (USA)
Tax calculation will be finalised during checkout.

Instant access to the full article PDF.

Institutional subscriptions

Similar content being viewed by others

References

  • Arzarello, F., Garuti, R., & Ricci, R. (2015). The impact of PISA studies on the Italian national assessment system. In K. Stacey & R. Turner (Eds.), Assessing mathematical literacy (pp. 249–260). Berlin: Springer.

    Google Scholar 

  • Baird, J.-A., Johnson, S., Hopfenbeck, T. N., Isaacs, T., Sprague, T., Stobart, G., et al. (2016). On the supranational spell of PISA in policy. Educational Research,58(2), 121–138.

    Google Scholar 

  • Barber, M., & Mourshed, M. (2007). How the world’s best-performing school systems come out on top. Retrieved August 3, 2010 from New York https://www.mckinsey.com/industries/social-sector/our-insights/how-the-worlds-best-performing-school-systems-come-out-on-top.

  • Baroutsis, A., & Lingard, B. (2017). Counting and comparing school performance: An analysis of media coverage of PISA in Australia, 2000–2014. Journal of Education Policy,32(4), 432–449.

    Google Scholar 

  • Berliner, D. C. (2011). The context for interpreting PISA results in the USA: Negativism, chauvinism, misunderstanding, and the potential to distort the educational systems of nations. In M. A. Pereyra, H.-G. Kotthoff, & R. Cowen (Eds.), Pisa under examination (pp. 77–96). New York: Springer.

    Google Scholar 

  • Bieber, T., & Martens, K. (2011). The OECD PISA study as a soft power in education? Lessons from Switzerland and the US. European Journal of Education,46(1), 101–116. https://doi.org/10.1111/j.1465-3435.2010.01462.x.

    Article  Google Scholar 

  • Breakspear, S. (2012). The policy impact of Pisa: An exploration of the normative effects of international benchmarking in school system performance. Retrieved July 3, 2016 from Paris http://search.oecd.org/officialdocuments/displaydocumentpdf/?cote=EDU/WKP(2012)8&docLanguage=En.

  • Bybee, R., & McCrae, B. (2011). Scientific literacy and student attitudes: Perspectives from PISA 2006 science. International Journal of Science Education,33(1), 7–26.

    Google Scholar 

  • Campbell, M. (2013). West vs Asia education rankings are misleading: Western schoolchildren are routinely outperformed by their Asian peers, but worrying about it is pointless. New Scientist (2898). Retrieved January 10, 2015 from https://www.newscientist.com/article/mg21728985-800-west-vs-asia-education-rankings-are-misleading/.

  • Carney, S., Rappleye, J., & Silova, I. (2012). Between faith and science: World culture theory and comparative education. Comparative Education Review,56(3), 366–393.

    Google Scholar 

  • Domínguez, M., Vieira, M.-J., & Vidal, J. (2012). The impact of the programme for international student assessment on academic journals. Assessment in Education: Principles, Policy & Practice,19(4), 393–409.

    Google Scholar 

  • Eivers, E. (2010). PISA: Issues in implementation and interpretation. The Irish Journal of Education/Iris Eireannach an Oideachais,38, 94–118.

    Google Scholar 

  • Feniger, Y., & Lefstein, A. (2014). How not to reason with PISA data: An ironic investigation. Journal of Education Policy,29(6), 845–855. https://doi.org/10.1080/02680939.2014.892156.

    Article  Google Scholar 

  • Figazzolo, L. (2009). Impact of PISA 2006 on the education policy debate. Retrieved September 9, 2015 from http://download.ei-ie.org/docs/IRISDocuments/ResearchWebsiteDocuments/2009-00036-01-E.pdf.

  • Friedman, T. L. (2007). The world is flat: A brief history of the twenty-first century. New York: Farrar, Straus and Giroux.

    Google Scholar 

  • Gewertz, C. (2018). PISA working toward exam to gauge creative thinking—Built by ACT. Education Week,38(6), 4.

    Google Scholar 

  • Goldstein, H., Bonnet, G., & Rocher, T. (2007). Multilevel structural equation models for the analysis of comparative data on educational performance. Journal of Educational and Behavioral Statistics,32(3), 252–286. https://doi.org/10.3102/1076998606298042

    Article  Google Scholar 

  • Gould, S. J. (1996). The mismeasure of man. New York: Northon.

    Google Scholar 

  • Green, A., & Mostafa, T. (2013). The dynamics of education systems: Convergent and divergent trends, 1990–2010. In J. Janmaat, M. Duru-Bellat, P. Méhaut, & A. Green (Eds.), The dynamics and social outcomes of education systems (pp. 15–45). Berlin: Springer.

    Google Scholar 

  • Grek, S. (2009). Governing by numbers: The PISA ‘effect’ in Europe. Journal of Education Policy,24(1), 23–37.

    Google Scholar 

  • Grisay, A., De Jong, J., Gebhardt, E., Berezner, A., & Halleux-Monseur, B. (2007). Translation equivalence across PISA countries. Journal of Applied Measurement,8(3), 249.

    Google Scholar 

  • Grisay, A., & Gonzalez, E. (2009). Equivalence of item difficulties across national versions of the PIRLs and PISA reading assessments. IERI Monograph Series: Issues and Methodologies in Large-Scale Assessments,2, 63–83.

    Google Scholar 

  • Grönqvist, E., & Vlachos, J. (2016). One size fits all? The effects of teachers’ cognitive and social abilities on student achievement. Labour Economics,42, 138–150.

    Google Scholar 

  • Gruber, K. H. (2006). The German ‘PISA-Shock’: Some aspects of the extraordinary impact of the OECD’s PISA study on the German education system. Paper presented at the Cross-national Attraction in Education: Accounts from England and Germany. Oxford: Symposium Books.

  • Hanushek, E. A. (2013). Economic growth in developing countries: The role of human capital. Economics of Education Review,37, 204–212.

    Google Scholar 

  • Hanushek, E. A., & Woessmann, L. (2008). The role of cognitive skills in economic development. Journal of Economic Literature,46, 607–668.

    Google Scholar 

  • Hanushek, E. A., & Woessmann, L. (2010). The high cost of low educational performance: The long-run economic impact of improving PISA Outcomes. Retrieved March 20, 2011 from Paris http://books.google.com/books?id=k7AGPo0NvfYC&pg=PA33&lpg=PA33&dq=hanushek+pisa+gdp&source=bl&ots=2gCfzF-f1_&sig=wwe0XLL5EblVWK9e7RJfb5MyhIU&hl=en&sa=X&ei=MLPCUqaOD8-JogS6v4C4Bw&ved=0CGcQ6AEwBjgK-v=onepage&q=hanushek%20pisa%20gdp&f=false.

  • Hanushek, E. A., & Woessmann, L. (2012). Do better schools lead to more growth? Cognitive skills, economic outcomes, and causation. Journal of Economic Growth,17(4), 267–321.

    Google Scholar 

  • Hargreaves, A., & Sahlberg, P. (2015). The tower of PISA is badly leaning. An argument for why it should be saved. Retrieved April 5, 2017 from https://www.washingtonpost.com/news/answer-sheet/wp/2015/03/24/the-tower-of-pisa-is-badly-leaning-an-argument-for-why-it-should-be-saved/?noredirect=on&utm_term=.69275346b5d3.

  • Harris, A., & Jones, M. S. (2015). Leading futures: Global perspectives on educational leadership. New Delhi: SAGE Publications India.

    Google Scholar 

  • Hatzinikita, V., Dimopoulos, K., & Christidou, V. (2008). PISA test items and school textbooks related to science: A textual comparison. Science Education,92(4), 664–687.

    Google Scholar 

  • Hopfenbeck, T. N., Lenkeit, J., El Masri, Y., Cantrell, K., Ryan, J., & Baird, J.-A. (2018). Lessons learned from PISA: A systematic review of peer-reviewed articles on the programme for international student assessment. Scandinavian Journal of Educational Research,62(3), 333–353. https://doi.org/10.1080/00313831.2016.1258726.

    Article  Google Scholar 

  • Hopmann, S. T. (2008). No child, no school, no state left behind: Schooling in the age of accountability. Journal of Curriculum Studies,40(4), 417–456.

    Google Scholar 

  • Hopmann, S. T., Brinek, Gertrude, & Retzl, M. (Eds.). (2007). PISA zufolge PISA—PISA according to PISA. Berlin: Lit Verlag.

    Google Scholar 

  • Jones, R. S. (2011). Education reform in Japan. Paris: OECD. https://doi.org/10.1787/5kg58z7g95np-en.

    Book  Google Scholar 

  • Kamens, D. H. (2015). A maturing global testing regime meets the world economy: Test scores and economic growth, 1960–2012. Comparative Education Review,59(3), 420–446.

    Google Scholar 

  • Kjærnsli, M., & Lie, S. (2011). Students’ preference for science careers: International comparisons based on PISA 2006. International Journal of Science Education,33(1), 121–144.

    Google Scholar 

  • Klees, S. J. (2016). Human capital and rates of return: Brilliant ideas or ideological dead ends? Comparative Education Review,60(4), 644–672.

    Google Scholar 

  • Komatsu, H., & Rappleye, J. (2017). A new global policy regime founded on invalid statistics? Hanushek, Woessmann, PISA, and economic growth. Comparative Education,53(2), 166–191.

    Google Scholar 

  • Kreiner, S., & Christensen, K. B. (2014). Analyses of model fit and robustness. A new look at the PISA scaling model underlying ranking of countries according to reading literacy. Psychometrika. https://doi.org/10.1007/s11336-013-9347-z.

    Article  Google Scholar 

  • Kuramoto, N., & Koizumi, R. (2016). Current issues in large-scale educational assessment in Japan: Focus on national assessment of academic ability and university entrance examinations. Assessment in Education: Principles, Policy & Practice. https://doi.org/10.1080/0969594X.2016.1225667.

    Article  Google Scholar 

  • Labaree, D. (1997). Public goods, private goods: The American struggle over educational goals. American Educational Research Journal,34(1), 39–81.

    Google Scholar 

  • Labaree, D. F. (2014). Let’s measure what no one teaches: PISA, NCLB, and the shrinking aims of education. Teachers College Record,116(9), 1–14.

    Google Scholar 

  • Lewis, S. (2017). Policy, philanthropy and profit: The OECD’s PISA for schools and new modes of heterarchical educational governance. Comparative Education,53(4), 518–537. https://doi.org/10.1080/03050068.2017.1327246.

    Article  Google Scholar 

  • Lingard, B., Rezai-Rashti, G., Martino, W., & Sellar, S. (2015). Globalizing educational accountabilities. London: Routledge.

    Google Scholar 

  • Loveless, T. (2012). The 2012 brown center report on American education: How well are American students learning? Retrieved January 20, 2015 from Washington DC http://www.brookings.edu/~/media/Files/rc/reports/2012/0216_brown_education_loveless/0216_brown_education_loveless.pdf.

  • Loveless, T. (2014). PISA’s China problem continues: A response to Schleicher, Zhang, and Tucker. Retrieved January 20, 2015 from http://www.brookings.edu/research/papers/2014/01/08-shanghai-pisa-loveless.

  • Lundgren, U. P. (2011). PISA as a political instrument: One history behind the formulating of the PISA programme. In M. A. Pereyra, H.-G. Kotthoff, & R. Cowen (Eds.), Pisa under examination (pp. 17–30). New York: Springer.

    Google Scholar 

  • Meyer, H.-D., et. al. (2014). OECD and Pisa tests are damaging education worldwide—academics. Retrieved Februrary 3, 2015 from https://www.theguardian.com/education/2014/may/06/oecd-pisa-tests-damaging-education-academics.

  • Meyer, H.-D., & Benavot, A. (2013). PISA, power, and policy: The emergence of global educational governance. Oxford: Oxford University Press.

    Google Scholar 

  • Morrison, H. (2013). Pisa 2012 major flaw exposed. Retrieved Februrary 3, 2015 from https://paceni.wordpress.com/2013/12/01/pisa-2012-major-flaw-exposed/.

  • Ninomiya, S. (2016). The impact of PISA and the interrelation and development of assessment policy and assessment theory in Japan. Assessment in Education: Principles, Policy & Practice,26, 1–20.

    Google Scholar 

  • Nyborg, H. (2007). Do recent large-scale cross-national student assessment studies neglect general intelligence g for political reasons? European Journal of Personality,21, 739–741.

    Google Scholar 

  • OECD. (1999). Measuring student knowledge and skills: A new framework for assessment. Retrieved March 15, 2019 from Paris: http://www.oecd.org/education/school/programmeforinternationalstudentassessmentpisa/33693997.pdf.

  • OECD. (2010). OECD programme for international student assessment. Retrieved March 15, 2019 from http://www.pisa.oecd.org/pages/0,2987,en_32252351_32235731_1_1_1_1_1,00.html.

  • OECD. (2013). Ready to learn: Students engagement, drive, and self-beliefs. Retrieved March 15, 2019 from Paris: http://www.oecd.org/pisa/keyfindings/pisa-2012-results-volume-III.pdf.

  • OECD. (2016a). PISA 2015 results (volume I): Excellence and equity in education. Retrieved March 15, 2019 from Paris: http://dx.doi.org/10.1787/9789264266490-en.

  • OECD. (2016b). PISA 2015: Results in focus. PISA. Retrieved March 15, 2019 from https://www.oecd.org/pisa/pisa-2015-results-in-focus.pdf.

  • OECD. (2017). PISA 2015 results: Students’ well-being. Retrieved March 15, 2019 from Paris http://www.keepeek.com/Digital-Asset-Management/oecd/education/pisa-2015-results-volume-iii_9789264273856-en-.Wk1WGrQ-fOQ#page1.

  • OECD. (2018a). About PIAAC. Retrieved March 15, 2019 from http://www.oecd.org/skills/piaac/about/.

  • OECD. (2018b). PISA for development. Retrieved March 15, 2019 from https://www.oecd.org/pisa/pisa-for-development/.

  • OECD. (2018c). PISA for schools. Retrieved March 15, 2019 from http://www.oecd.org/PISA/pisa-for-schools/.

  • OECD. (2018d). PISA: About. Retrieved March 15, 2019 from http://www.oecd.org/pisa/aboutpisa/.

  • OECD. (nd). The international early learning and child well-being study—The study. Retrieved March 15, 2019 from http://www.oecd.org/education/school/the-international-early-learning-and-child-well-being-study-the-study.htm.

  • OECD/UNESCO. (2003). Literacy skills for the world of tomorrow—Further results from Pisa 2000. PISA. Retrieved October 23, 2017 from http://www.oecd.org/education/school/2960581.pdf.

  • Pence, A. (2016). Baby PISA: Dangers that can arise when foundations shift. Journal of Childhood Studies,41(3), 54–58.

    Google Scholar 

  • Pereyra, M. A., Kotthoff, H.-G., & Cowen, R. (2011). PISA under examination. New York: Springer.

    Google Scholar 

  • Popkewitz, T. (2011). PISA: Numbers, standardizing conduct, and the alchemy of school subjects. In M. A. Pereyra, H.-G. Kotthoff, & R. Cowen (Eds.), Pisa under examination (pp. 31–46). New York: Springer.

    Google Scholar 

  • Prais, S. J. (2003). Cautions on OECD’s recent educational survey (PISA). Oxford Review of Education,29, 139–163.

    Google Scholar 

  • Rindermann, H. (2007). The g-factor of international cognitive ability comparisons: The homogeneity of results in PISA, TIMSS, PIRLS and IQ-tests across nations. European Journal of Personality,21, 667–706.

    Google Scholar 

  • Rutkowski, L., & Rutkowski, D. (2016). A call for a more measured approach to reporting and interpreting PISA results. Educational Researcher,45(4), 252–257.

    Google Scholar 

  • Sahlberg, P. (2011). Finnish lessons: What can the world learn from educational change in Finland?. New York: Teachers College Press.

    Google Scholar 

  • Sahlberg, P. (2012). How GERM is infecting schools around the world. Retrieved October 23, 2017 from http://www.washingtonpost.com/blogs/answer-sheet/post/how-germ-is-infecting-schools-around-the-world/2012/06/29/gJQAVELZAW_blog.html-comments.

  • Sahlberg, P. (2017). FinnishED leadership. Thousand Oaks: Corwin.

    Google Scholar 

  • Schleicher, A. (2013). Are the Chinese cheating in PISA or are we cheating ourselves? Retrieved Februrary 3, 2015 from http://oecdeducationtoday.blogspot.com/2013/12/are-chinese-cheating-in-pisa-or-are-we.html.

  • Schleicher, A. (2018). World class: How to build a 21st-century school system. Paris: OECD.

    Google Scholar 

  • Schuelka, M. J. (2013). Excluding students with disabilities from the culture of achievement: The case of the TIMSS, PIRLS, and PISA. Journal of Education Policy,28(2), 216–230.

    Google Scholar 

  • Sellar, S., & Lingard, B. (2014). The OECD and the expansion of PISA: New global modes of governance in education. British Educational Research Journal,40(6), 917–936.

    Google Scholar 

  • Shirley, D. (2017). The new imperatives of educational change: Achievement with integrity. New York: Routledge.

    Google Scholar 

  • Sjøberg, S. (2012). PISA: Politics, fundamental problems and intriguing results. Recherches en Education,14, 1–21.

    Google Scholar 

  • Sjøberg, S. (2015a). OECD, PISA, and globalization: The influence of the international assessment regime. In C. H. Tienken & C. A. Mullen (Eds.), Education policy perils (pp. 114–145). London: Routledge.

    Google Scholar 

  • Sjøberg, S. (2015b). PISA and global educational governance—A critique of the project, its uses and implications. Eurasia Journal of Mathematics, Science and Technology Education,11(1), 111–127.

    Google Scholar 

  • Solheim, O. J., & Lundetræ, K. (2018). Can test construction account for varying gender differences in international reading achievement tests of children, adolescents and young adults? A study based on Nordic results in PIRLS, PISA and PIAAC. Assessment in Education: Principles, Policy and Practice,25(1), 107–126.

    Google Scholar 

  • Stewart, W. (2013). Is Pisa fundamentally flawed? Retrieved Februrary 3, 2015 from http://www.tes.co.uk/article.aspx?storycode=6344672.

  • Stromquist, N. P. (2016). Using regression analysis to predict countries’ economic growth: Illusion and fact in education policy. Real-World Economics Review,76, 65–74.

    Google Scholar 

  • Tröhler, D. (2013). The OECD and cold war culture: Thinking historically about PISA. In PISA, power, and policy: The emergence of global educational governance (pp. 141–161).

  • Tucker, M. (Ed.). (2011). Surpassing Shanghai: An agenda for American education built on the world’s leading systems. Boston: Harvard Education Press.

    Google Scholar 

  • Tucker, M. (2014). Chinese lessons: Shanghai’s rise to the top of the PISA league tables. Retrieved October 23, 2017 from Washington DC. http://www.ncee.org/wp-content/uploads/2013/10/ChineseLessonsWeb.pdf.

  • Tucker, M. (2016). Asian countries take the U.S. to school. The Atlantic. Retrieved October 23, 2017 from https://www.theatlantic.com/education/archive/2016/02/us-asia-education-differences/471564/.

  • Uljens, M. (2007). The hidden curriculum of PISA: The promotion of neo-liberal policy by educational assessment. In S. T. Hopmann, G. Brinek, & M. Retzl (Eds.), PISA zufolge PISA—PISA According to PISA (pp. 295–303). Berlin: Lit Verlag.

    Google Scholar 

  • Urban, M. (2017). We need meaningful, systemic evaluation, not a preschool PISA. Global Education Review,4(2), 18–24.

    Google Scholar 

  • Wu, M. (2009). A comparison of PISA and TIMSS 2003 achievement results in mathematics. Prospects,39(1), 33.

    Google Scholar 

  • Zhao, Y. (2009). Catching up or leading the way: American education in the age of globalization. Alexandria: ASCD.

    Google Scholar 

  • Zhao, Y. (2012). World class learners: Educating creative and entrepreneurial students. Thousand Oaks: Corwin.

    Google Scholar 

  • Zhao, Y. (2014). Who’s afraid of the big bad dragon: Why China has the best (and Worst) education system in the world. San Francisco: Jossey-Bass.

    Google Scholar 

  • Zhao, Y. (2015). Lessons that matter: What we should learn from Asian school systems. Retrieved July 20, 2017 from Melbourne. http://www.mitchellinstitute.org.au/reports/lessons-that-matter-what-should-we-learn-from-asias-school-systems/.

  • Zhao, Y. (2016a). From deficiency to strength: Shifting the mindset about education inequality. Journal of Social Issues,72(4), 716–735.

    Google Scholar 

  • Zhao, Y. (2016b). Who’s afraid of PISA: The fallacy of international assessments of system performance. In A. Harris & M. S. Jones (Eds.), Leading futures (pp. 7–21). Thousand Oaks: Sage.

    Google Scholar 

  • Zhao, Y. (2017a). Fatal attraction: Why the west must stop copying China’s flawed education system. New Internationalist,505, 24–25.

    Google Scholar 

  • Zhao, Y. (2017b). What works can hurt: Side effects in education. Journal of Educational Change,18(1), 1–19.

    Google Scholar 

  • Zhao, Y. (2018a). The changing context of teaching and implications for teacher education. Peabody Journal of Education,93, 1–14.

    Google Scholar 

  • Zhao, Y. (2018b). Shifting the education paradigm: Why international borrowing is no longer sufficient for improving education in China. ECNU Review of Education,1(1), 76–106.

    Google Scholar 

  • Zhao, Y. (2018c). What works may hurt: Side effects in education. New York: Teachers College Press.

    Google Scholar 

  • Zhao, Y., & Gearin, B. (2016). Squeezed out. In D. Ambrose & R. J. Sternberg (Eds.), Creative intelligence in the 21st century (pp. 121–138). Berlin: Springer.

    Google Scholar 

  • Zhao, Y., & Wang, Y. (2018). Guarding the past or inventing the future: Education reforms in East Asia. In Y. Zhao & B. Gearin (Eds.), Imagining the future of global education: Dreams and nightmares (pp. 143–159). New York: Routledge.

    Google Scholar 

Download references

Author information

Authors and Affiliations

  1. School of Education, University of Kansas, 419 JRP, 1122 West Campus Rd., Lawrence, KS, 66045-3101, USA

    Yong Zhao

Authors
  1. Yong Zhao
    View author publications

    You can also search for this author in PubMed Google Scholar

Corresponding author

Correspondence to Yong Zhao.

Additional information

Publisher's Note

Springer Nature remains neutral with regard to jurisdictional claims in published maps and institutional affiliations.

Rights and permissions

Reprints and permissions

About this article

Check for updates. Verify currency and authenticity via CrossMark

Cite this article

Zhao, Y. Two decades of havoc: A synthesis of criticism against PISA. J Educ Change 21, 245–266 (2020). https://doi.org/10.1007/s10833-019-09367-x

Download citation

  • Published:

  • Issue Date:

  • DOI: https://doi.org/10.1007/s10833-019-09367-x

Keywords

Search

Navigation