Fighting for Equity in Education

The struggle is long but hope is longer

PISA Rankings Are Misleading Because of Differences in Student Coverage

Wednesday August 23, 2017

One of the standout performers in the results from PISA 2015 was Vietnam. It achieved a ranking of 8th in science with a score of 525, which was significantly above Australia’s score of 510. More remarkably, only 6% of its students were below the minimum PISA standard compared to 18% of students in Australia. Vietnam had the smallest proportion of students below the science standard of the 72 countries and economies participating in PISA 2015.

However, there seems to be more than meets the eye in these results because over half of Vietnam’s 15-year-old population was not covered by the PISA sample because they were not in school. The OECD’s PISA 2015 report shows that the sample of Vietnam students participating in PISA was taken from only 48.5% of Vietnam’s 15-year-olds. This is the lowest coverage of 15-year-olds by far of all the countries participating in PISA [see OECD, PISA 2015 Results (Volume 1): Excellence and Equity in Education, Table A2.1, column 15, p. 290]. The next lowest coverage is Mexico at 61.7%. Australia’s coverage was 90.6% and most OECD countries had over 90% coverage.

John Jerrim, an academic at the Institute of Education at the University College of London, has suggested that Vietnam’s PISA science scores are artificially inflated because those who were not covered by the sample (eg, left school before turning 15) are likely to be academically weaker than those who were tested. He assesses the impact of the low coverage of Vietnam’s 15-year-old students by comparing its science results with other PISA data where it is assumed that 15-year-olds who are not in school perform at or below the national median. He estimates that the “real” performance of Vietnam is probably 50 to 60 points below its mean science score. This would put Vietnam’s ranking in science at 35-40th, a far cry from 8th.

The evidence for this conclusion is based on a comparison of scores at the 75th percentile (50th percentile scores are not available for Vietnam). The PISA 2015 report shows that Vietnam’s score excluding students not in school was 576 [Table I.2.3, p. 323]. However, alternative results where the 15-year-olds not in school are included, assuming they score at or below the median for all students, show a score of 519 [Online Table I.2.4d].

This makes a huge difference in Vietnam’s ranking. It is ranked 19th on scores at the 75th percentile where 15-year-olds not in school are excluded compared to 47th for results where they are included. This suggests that Vietnam’s ranking of 8th on national mean scores is also likely to be substantially inflated by the low coverage of its 15-year-old population. Vietnam’s 15-year-olds that are in school are doing very well by international standards, but a very large proportion are not in school and this indicates Vietnam has a long way to go in improving overall education levels.

Inclusion of 15-year-olds not in school would probably put Vietnam’s mean score substantially below that of Australia’s, based on a comparison of the alternative results at the 75th percentile. Australia’s 75th percentile score excluding 15-year-olds not in school was 583, only 7 points higher than Vietnam’s of 576. However, Australia’s score including those not in school was 575 which was 56 points higher than Vietnam’s score of 519.

The low coverage of Vietnam’s 15-year-olds has similar effects on the comparison of mathematics scores. While Vietnam’s mean score in mathematics of 495 was similar to Australia’s score of 494, it is likely to be substantially inflated by the low coverage. For example, Australia’s 75th percentile score at 559 was only 8 points higher than Vietnam’s score of 551 [Table I.5.3, p. 389]. However, when 15-year-olds not in school are included, Australia’s 75th percentile score was slightly lower at 551, while Vietnam’s score of 489 was 62 points lower [Online Table I.5.4d]. Vietnam’s score here was also 62 points lower than its score when 15-year-olds not in school are excluded.

The high ranking (10th) of the four Chinese cities/provinces (Beijing, Shanghai, Jiangsu, Guangdong) that participated in PISA 2015 is also likely to be a result of low proportion of 15-year-olds in school. The coverage rate for these cities was only 63.9%, and was the 4th lowest of all the 72 countries/regions participating in PISA 2015.

There is a huge difference in the median score for the four Chinese cities/provinces when 15-year-olds not in school are excluded and when they are included. The median science score when they are excluded was 524, just 9 points above Australia’s median score of 515. However, when the 15-year-olds not in school are included (on the assumption that they perform below the median) the median score of the Chinese cities/provinces falls by a massive 92 points to only 432, which is 68 points below the corresponding score for Australia.

This makes a huge difference to the international ranking of these cities. They are ranked 9th on median scores where 15-year-olds not in school are excluded compared to 42nd when they are included. Given that the mean and median scores for these cities/provinces are similar (518 and 524), this suggests that their “real” performance is also about 90 points below their mean science score and their ranking would drop from 10th in the world to possibly around 50th, although the actual ranking depends on changes in the mean scores of other countries when their results are adjusted to include 15-year-olds not in school.

It should also be noted that the combined results of Beijing, Shanghai, Jiangsu and Guangdong are also likely to be misleading in that

A lower proportion of 15-year-olds in school appears to have also inflated the results for the UK. Its coverage rate was 84%. Its mean and median science scores were similar to Australia’s (see table below). However, its median score of 512 when 15-year-olds not in school are excluded falls by 27 points to 485 when they are included and its ranking on median scores drops from 14th to 23rd. In contrast, Germany, Netherlands and Switzerland improved their top 20 ranking.

Australia’s coverage of 15-year-old students at 90.6% is in the mid-range of OECD and top 20 country rankings by mean scores. Its ranking on median scores is slightly affected. Australia’s median score of 515 excluding 15-year-old students not in school drops to 500 when they are included. As a result, Australia’s international ranking on median scores falls slightly from 13th to 15th. However, Australia is doing much better than Vietnam, the four Chinese cities/provinces and the UK despite the perception given by the published PISA rankings. Australia has high test scores and a high proportion of 15-year-olds in schools, although there is room for improvement on both measures.

The analysis demonstrates that caution is needed in interpreting the PISA international league tables. Not only do scores and rankings change according to whether mean or median scores are used, but they can be significantly different if the coverage of 15-year-olds varies significantly between countries/cities/regions. Country and city test scores that include 15-year-olds not in school are lower than when they are excluded. In some cases, such as Vietnam and the four Chinese cities/provinces, there is a very large difference in scores that amounts up to two or more years of schooling (see chart below, where the four Chinese cities/provinces are a clear outlier). Vietnam, the Chinese cities/provinces and the UK all drop out of the top 20 PISA rankings when account is taken of the proportion of 15-year-olds not in school.

It is a stark reminder of how the proportion of 15-year-olds in schools can have a very large impact on country/city test scores and international rankings. A country, region or city with high test scores can hardly be considered a successful education performer if a large proportion of its 15-year-olds are not in education. The OECD should consider not including countries/regions/cities with a high proportion of 15-year-olds not in school in its international rankings because they can significantly distort the rankings and make them highly misleading.

Trevor Cobbold

Student Coverage Rates & Top 20 Country & City Rankings – PISA 2015.pdf

Median Science Scores Including & Excluding 15-Year-Olds Not in School – PISA 2015.pdf

  1. I taught in the UK for 18 years and do not believe the figure you suggest for 15 year olds not in school when referring to the UK. Are you sure the coverage figure means that those students are not in school? I suspect that it may be excluding either particular UK counties or perhaps a school sector. If 15% of UK 15 year olds were not in school it would be a national scandal and these students would be causing major problems on the streets.


    peter johnstone    Sep 3, 08:32 PM    #
  Textile Help

Previous

Search

Contact us

News feed from Save our Schools

SOS Twitter

Chris Bonnor's Education Media Watch

Sections

Links


Education news


Education blogs


Education information


Public education


Policy Briefs


Research

Speeches


Submissions