Appendices

A

Appendices

Appendix 1: Definitions of 21st-century skills

  Skill Definition
Foundational literacies Literacy Ability to read, understand and use written language
Numeracy Ability to use numbers and other symbols to understand and express quantitative relationships
Scientific literacy Ability to use scientific knowledge and principles to understand one's environment and test hypotheses
ICT literacy Ability to use and create technology-based content, including finding and sharing information, answering questions, interacting with other people and computer programming
Financial literacy Ability to understand and apply conceptual and numerical aspects of finance in practice
Cultural and civic literacy Ability to understand, appreciate, analyse and apply knowledge of the humanities
Competencies Critical thinking/
problem-solving
Ability to identify, analyse and evaluate situations, ideas and information to formulate responses and solutions
Creativity Ability to imagine and devise new, innovative ways of addressing problems, answering questions or expressing meaning through the application, synthesis or repurposing of knowledge
Communication Ability to listen to, understand, convey and contextualize information through verbal, nonverbal, visual and written means
Collaboration Ability to work in a team towards a common goal, including the ability to prevent and manage conflict
Character qualities Curiosity Ability and desire to ask questions and to demonstrate open-mindedness and inquisitiveness
Initiative Ability and desire to proactively undertake a new task or goal
Persistence/ grit Ability to sustain interest and effort and to persevere to accomplish a task or goal
Adaptability Ability to change plans, methods, opinions or goals in light of new information
Leadership Ability to effectively direct, guide and inspire others to accomplish a common goal
Social and cultural awareness Ability to interact with other people in a socially, culturally and ethically appropriate way

Sources: ESCO Skills Hierarchy for Transversal Skills; Partnership for 21st Century Skills. "Framework for 21st Century Learning." NEXT: Washington DC, 2001; Burkhardt, Gina. "enGauge 21st Century Skills: Literacy in the Digital Age." North Central Regional Educational Laboratory and The Metiri Group, 2003.; Learning Metrics Taskforce. "Towards Universal Learning: What Every Child Should Learn." Center for Universal Education at the Brookings Institution and UNESCO Institute for Statistics: Washington, DC, 2013; The Economist Intelligence Unit. "The Learning Curve: Education and Skills for Life." Pearson: London, 2014. Other sources considered but not included: AT21CS, WorldSkills, Iowa Dept. of Education's 21st Century Skills, and Tony Wagner's Seven Survival Skills.

Appendix 2: The measurement challenge

Measuring 21st-century skills presents numerous obstacles. Researchers have access to only limited direct metrics to assess performance on the full range of skills. In addition, the coverage of these metrics is often confined to the developed world.

The majority of tests measuring 21st-century skills focus on foundational literacies. Beyond the indicators we used in our methodology – the Programme for International Student Assessment (PISA), the Southern and Eastern Africa Consortium for Monitoring Educational Quality (SACMEQ) and the Latin American Laboratory for Assessment of the Quality of Education (LLECE) – other tests that measure literacy, numeracy and scientific literacy include the Progress in International Reading Literacy Study (PIRLS), the Early Grade Reading Assessment (EGRA), the Program for the Analysis of Education Systems (PASEC) and the Trends in International Mathematics and Science Study (TIMSS).

The three other literacies – financial, ICT and cultural and civic – have not been part of the traditional focus of international assessments and therefore there is less data and fewer assessments available to draw on. The only test currently available for financial literacy is PISA, but that test covers only 16 countries. For civic and cultural literacy, we evaluated two direct measurements, the International Civic and Citizenship Education Study (ICCS) and the Civic Education Study (CivEd). We picked ICCS for its wider coverage. Finally, we used PISA’s digital literacy assessment, which is a valuable assessment but has limited global coverage.

We found large gaps in coverage in the measurement of many core skills. When we combined existing metrics for literacy and numeracy, for example, we were able to cover fewer than half of the countries in the world.

Measurement challenges are amplified when it comes to competencies and character qualities. PISA has pioneered the assessment of problem-solving, a key competency. This assessment still covers only approximately 44 countries. For creativity, communication and collaboration, no direct measurements exist to date. For creativity, we used a proxy from one of the sub-scores in PISA’s mathematics assessment.[15] We encountered difficulties finding metrics that measure character qualities, with the exception of curiosity. For that metric, we used PISA’s problem-solving subscale.[16]

Note that PISA is in the middle of promising work to extend its 2015 and 2018 assessments. It plans to add collaborative problem-solving and global competencies, measuring skills such as intercultural understanding, empathy and perspective taking.

It is critical that countries support and facilitate research to improve both the direct measurement of 21st-century skills as well as their global coverage. Only then will countries be able to create an accurate baseline from which to measure progress in the future.

Appendix 3: Indicators considered and used in the report

  Skill Indicator Source
Foundational literacies Literacy
  • Mean PISA 2012, 2009 reading score
  • Mean SACMEQ III reading score
  • Mean LLECE reading score
  • Mean PIRLS literacy score
  • Mean EGRA score
  • Mean PASEC reading score
  • Youth literacy rate
  • OECD
  • UNESCO (IIEP)
  • UNESCO
  • IEA
  • USAID (funded)
  • UNESCO (IIEP)
  • UNESCO
Numeracy
  • Mean PISA 2012, 2009 mathematics score
  • Mean SACMEQ III mathematics score
  • Mean LLECE mathematics score
  • Mean TIMSS mathematics score
  • Mean EGMA score
  • Mean PASEC mathematics score
  • Quality of science and mathematics education index
  • OECD
  • UNESCO
  • UNESCO
  • IEA
  • USAID (funded)
  • UNESCO (IIEP)
  • WEF
Scientific literacy
  • Mean PISA 2012, 2009 science score
  • Mean LLECE science score
  • Mean TIMSS science score
  • Quality of science and mathematics education index
  • OECD
  • IEA
  • UNESCO
  • WEF
ICT literacy
  • PISA 2012 Digital Reading Assessment
  • Students per computer in school
  • Students per web-enabled computer
  • Internet access in schools (Global Competitiveness Index)
  • Percentage of schools with shortage of internet connectivity (PISA 2012 School Questionnaire)
  • Percentage of schools with shortage of computers for instruction (PISA 2012 School Questionnaire)
  • OECD
  • UNESCO
  • UNESCO
  • WEF
  • OECD
  • OECD
Financial literacy
  • PISA 2012 Financial Literacy Index
  • OECD
Cultural and civic literacy
  • ICCS score
  • CivEd test score
  • World Governance Indicators: Rule of Law, Voice and Accountability and Government Effectiveness
  • Percentage of students communicating with parents about social and cultural issues
  • Percentage of students with possessions related to classical culture
  • Frequency of going to cinema, live performances, cultural sites or attending live sport events
  • IEA
  • IEA
  • World Bank
  • EU (Eurostat data)
Competencies Critical thinking/
problem-solving
  • PISA 2012 Problem-Solving score
  • OECD
Creativity
  • PISA 2012 mathematics formulate subscore
  • International Innovation Index
  • Global Innovation Index
  • Global Creativity Index
  • Creative Class Share
  • OECD
  • BCG, NAM
  • INSEAD, WIPO
  • MPI
  • MPI
Communication
  • No indicators available
  • N/A
Collaboration
  • Self-reported score on university-company research collaboration
  • Self-reported score on cooperation in labour employment relations
  • World Bank
Character qualities Curiosity
  • PISA 2012 Problem-Solving – Acquisition of Knowledge Subscale
  • OECD
Initiative
  • Total early stage entrepreneurial activity (% of working age population both about to start an entrepreneurial activity and that have started one from a maximum of three and a half years)
  • Established business ownership rate (% of 18-64 population who are currently owner-manager of an established business)
  • National Expert Survey Index (combination of indicators with significant impact on national entrepreneurship)
  • GEM
Persistence/ grit
  • Self-reported measure of perseverance (PISA Student Questionnaire)
  • Total Early Stage Entrepreneurial Activity (GEM)
  • OECD
Adaptability
  • No indicators available
  • N/A
Leadership
  • Quality of management schools (2010)
  • Reliance on professional management (2010)
  • World Bank
Social and cultural awareness
  • Ethnic fractionalization index (2003)
  • Cultural diversity index (2003)
  • James Fearon, Stanford University

Factors influencing indicator selection include broad country coverage, direct skill measure and independent assessment.

bold denotes indicator selected to measure skill performance


Indicators considered and used to estimate educational factors holding countries back:

Education factors Definition Indicators
Policy Enablers Standards that govern K-12 education
  • Number of years of mandatory education (UNESCO)
  • Satisfaction with education policy
Human Capital Teacher quality, training and expertise
  • Number of students per trained teacher, primary[17]
  • Number of students per trained teacher, secondary
  • Number of students per teacher
  • Teachers working in schools that use some kind of formal appraisal
  • Average years of working experience as a teacher in total
  • Teachers who report high need to develop ICT skills for teaching (%)
  • Teaching time per week (hours)

Ideal quality metric would have included:

  • % of teachers coming from top tier universities
  • salary data for teachers relative to other jobs with same qualifications
Financial Resources Importance of education in public budgets
  • Government expenditure per primary student (constant PPP$) (UNESCO)
  • Government expenditure on education as % of GDP (%)
  • Government expenditure per secondary student (constant PPP$)
Technological Infrastructure Access to new digital tools and content via the internet
  • Internet access in schools (World Economic Forum)
  • Mobile broadband penetration
  • Household broadband penetration
  • Population using the internet (%)

Sources: OECD, UNESCO, American Educational Research Association.

Other factors, such as socioeconomic status and conflict, also present significant challenges to educational attainment.

bold denotes indicator selected to measure skill performance

Appendix 4: Countries with available skill data included in the report

High-income OECD High-income non-OECD Upper-middle income Lower-middle income Low-income
Australia Shanghai-China[18] Albania El Salvador Kenya
Austria Croatia Argentina Georgia Malawi
Belgium Cyprus Azerbaijan Guatemala Mozambique
Canada Hong Kong Botswana Indonesia Tanzania
Chile Latvia Brazil Kyrgyzstan Uganda
Czech Republic Liechtenstein Bulgaria Lesotho Zimbabwe
Denmark Lithuania Colombia Moldova
Estonia Macao Costa Rica Nicaragua
Finland Malta Dominican Republic Paraguay
France Qatar Ecuador Swaziland
Germany Russia Hungary Vietnam
Greece Singapore Jordan Zambia
Iceland Trinidad and Tobago Kazakhstan
Ireland United Arab Emirates Malaysia
Israel Uruguay Mauritius
Italy Mexico
Japan Montenegro
Luxembourg Namibia
Netherlands Panama
New Zealand Peru
Norway Romania
Poland Serbia
Portugal Seychelles
Slovak Republic South Africa
Slovenia Thailand
South Korea Tunisia
Spain Turkey
Sweden
Switzerland
United Kingdom
United States

Sources: World Bank; project team analysis.

Appendix 5: A comparison of performance data across tests

Many countries use widely varying measures to assess similar skills, making comparisons between countries’ absolute scores on comparable tests difficult. To increase the number of countries with comparable data for literacy, numeracy and scientific literacy, we conducted a crosswalk analysis, which allows researchers to compare results on tests of comparable skills that use widely different scales.

For countries in Africa and Latin America that had taken only the SACMEQ or LLESE tests and not the PISA test, we devised a way to convert those region-specific test scores into the equivalent of PISA scores. We looked at the handful of countries in those areas that had taken both PISA and either of the SACMEQ or LLESE tests in 2009 or 2012, in order to calculate an average conversion factor from one of the two regional tests into PISA. We then applied that conversion factor to SACMEQ and LLESE scores to translate them into PISA scores. Since we did not have access to the raw data, we assumed that the statistical distribution of the converted scores corre- sponded to the distribution of the original scores. The methodology allowed only for a ranking-based comparison of the countries studied, not an absolute score assessment. We therefore have not provided the converted scores for comparison but rather used percentile ranks. This approach draws on the more advanced methodology demonstrated by Altinok and Murseli,[19] as well as Hanushek, Peterson and Ludger,[20] and it is intended to provide an indicative comparison among countries rather than a rigorous assessment of relative performance.

As a result of the analysis, we increased the sample size from 72 to 91 countries. In particular, coverage for the lower-middle-income cluster increased from two to 12 countries and coverage for the low-income cluster increased from zero to six countries.


  1. ^ PISA 2012 mathematics subscale: "For individuals to use their mathematical knowledge and skills to solve a problem, they often first need to translate the problem into a form that is amenable to mathematical treatment. The framework refers to this process as one of formulating situations mathematically. In the PISA assessment, students may need to recognize or introduce simplifying assumptions that would help make the given mathematics item amenable to analysis. They have to identify which aspects of the problem are relevant to the solution and which might safely be ignored. They must recognize words, images, relationships or other features of the problem that can be given a mathematical form; and they need to express the relevant information in an appropriate way, for example in the form of a numeric calculation or as an algebraic expression."
  2. ^ PISA 2012 Creative Problem Solving, acquisition of knowledge subscale: "In knowledge-acquisition tasks, the goal is for students to develop or refine their mental representation of the problem space. Students need to generate and manipulate the information in a mental representation. The movement is from concrete to abstract, from information to knowledge. In the context of the PISA assessment of problem solving, knowledge-acquisition tasks may be classified either as “exploring and understanding” tasks or as “representing and formulating” tasks."
  3. ^ Calculated using UIS data: Enrollment in primary education [number] / (teachers in primary education [number] x teachers in primary education who are trained [%]). Calculated using Teacher Quality Opportunity Gap and National Achievement data for countries without UIS data for primary teacher education rates.
  4. ^ Shanghai is grouped in high-income non-OECD due to its income level. (PISA reports China data for Shanghai only.)
  5. ^ Altinok, Nadir and Hatidje Murseli. “International Database on Human Capital Quality.” Economics Letters 96, no. 2. 2007; Altinok, Nadir. “A New International Database on the Distribution of Student Achievement.” 2011. United Nations Educational, Scientific and Cultural Organization (UNESCO).
  6. ^ Hanushek, Eric A., Paul E. Peterson and Ludger Woessman. “Achievement Growth: International and U.S. State Trends in Student Performance.” Harvard Kennedy School of Government. 2012.