My School Comparisons are Biased in Favour of Private SchoolsFriday February 19, 2010
The ‘like school’ comparisons on the My School website purport to compare the test results of schools having similar socio-economic student populations. However, like is not consistently compared with like. My School’s measure of the socio-economic status (SES) of schools is systematically biased in a way that favours private schools in comparing their results with so-called ‘like’ government schools.
The bias works in two separate, but compounding ways. My School tends to under-estimate the SES of private schools that draw enrolments from high SES families living in lower SES areas. It also tends to over-estimate the SES of government schools because high SES families resident in their area tend to choose private schools.
The source of the bias is that the Index of Community Socio-Educational Advantage (ICSEA) used to measure the socio-economic status (SES) of schools is based on the average socio-economic characteristics of the areas in which students live and not on the actual SES of their families. Studies by the Australian Bureau of Statistics show that some high income families live in low SES areas and vice versa, so the actual SES of some students will be above the area-average and others below the area-average.
This is compounded by the false assumption that similar proportions of parents in different socio-economic circumstances choose private schools over government schools. My School assumes that the proportion of high and low SES students in private and government schools reflects the respective proportions of these families in the area from which schools draw students. However, this assumption is incorrect as there is a greater leakage of high SES students to private schools.
High and low SES families do not enrol in private and government schools in the same proportion. On average, 47% of high income families choose private schools compared to 24% of low income families. In the case of secondary schools, 55% of high income families choose private schools compared to 26% of low income families. Even middle income families are more likely to choose private schools than low income families – on average 32% across Australia.
The greater leakage of high SES students from each area into private schools causes the ICSEA rating of private schools to under-estimate their actual SES because these students are classified according to their (lower) area SES measure rather than by their (higher) family SES. The ICSEA rating of private schools thus mismeasures their actual SES.
The extent of the bias in particular instances will vary according to the SES composition of different Census collection districts and the proportion of families in these areas who choose private schools. In some cases, there may be little bias, but in others it is potentially large.
The bias is most apparent where high fee private boarding schools draw a significant proportion of their students from high income families in rural areas that are classified as relatively low SES areas. For example, half the student population at Geelong Grammar are boarders, many of whom are traditionally drawn from rural areas in which the very high income families who can afford tuition and boarding fees of nearly $50 000 a year are a small minority of the population. Yet, despite being the most expensive school in Australia, Geelong Grammar has an ICSEA rating of 1135 which puts it in the second tier of high SES school ratings.
On the other hand, the ICSEA rating of government schools over-estimates their actual SES because of the leakage of high SES students to private schools. Government schools take a greater proportion of low SES students, but these students are classified at the (higher) area SES rating rather than by the actual SES of their families. The lower SES students carry the higher area SES score influenced by high SES families in the area whose students do not attend government schools. Thus, the level of disadvantage in government schools is under-estimated by ICSEA.
This systematic bias in the measurement of the SES of government and private schools can be illustrated by a simple hypothetical example. A suburb or town made up of several low SES census collection district areas will have some high SES families who may all choose to send their children to a private school while the majority low SES families send their children to local government schools. The private school attended by the high SES families would be measured as a low SES school because the average SES of each collection district in the area it draws its students from is low. The government schools would be measured as low SES as well, but their actual SES would be even lower than the overall SES of the area because the high SES families which are taken into account in the area measure do not attend the government schools.
The bias is further illustrated by examples of so-called like school comparisons taken from My School.
Take the comparison of the prestigious and wealthy King’s School in Sydney with Gundaroo Public School, a small primary school in a semi-rural area of NSW near Canberra. The test results for the King’s School are excellent with lots of green colour codes for above average results while Gundaroo by comparison has a lot of red colour codes for below average results.
Both schools have an ICSEA rating of 1156, even though the King’s School has annual primary school fees of over $20 548 for Years 5 & 6 and boarding fees of $14 592. My School says that 78% of students at the King’s School are in the top quarter of the ICSEA compared to 91% of students at Gundaroo. However, in reality, the schools are very unalike.
The SES rating for the King’s School is likely under-estimated because it traditionally draws a large number of students from farming families. About 30% of its enrolments are boarding students, although not all are from rural Australia. Only the wealthiest of rural families can afford tuition and boarding fees of over $36 000 a year for primary students. Yet, because these students are resident in lower SES rural areas they carry a lower SES score than their actual family circumstances. The relatively large proportion of these students attending the King’s School therefore reduces its overall ICSEA rating.
On the other hand, it is likely that the SES rating for Gundaroo Public School is over-estimated because many high income families in the area send their children to schools in nearby Canberra. This may also lead to the SES rating of the receiving Canberra schools being under-estimated as well.
The 2006 Census data shows that the Gundaroo region had 358 family households who stated their full income and about 44 of these, or 12%, could be considered as relatively low income (less than $1000 a week) while about 55% were high income households (over $2000 a week). Some 32% of the population over 20 years of age did not finish Year 12, but 68% did. One quarter had certificate based non-school qualifications while 75% had higher education degrees or diplomas. About 30% were employed in lower skilled occupations while 54% were employed as managers or professionals.
Clearly then, Gundaroo contains a relatively large proportion of high income, well educated, highly skilled households, but it also has a significant proportion of lower SES families.
According to the Census, there were 314 children under 15 in Gundaroo in 2006. If these children were distributed proportionally across different ages within this group, about 150-160 would have been in the primary school age group. However, only about half of these are enrolled at Gundaroo Public School.
There appears to be a significant difference in the pattern of school enrolments between the higher and lower SES families in the region. Most of the high SES families send their children to schools in Canberra while the local school is mostly attended by children of the lower and middle SES families.
Given that the large part of differences between school results is explained by differences in the socio-economic background of families, different choices made by Gundaroo families likely explain why test results at the local school are so much lower than those of the King’s School. Far from being ‘like schools’, they are very unalike schools. It is the flawed measurement of the SES of schools which is at fault because it does not measure the actual SES of families enrolled at schools.
Another example is the initial comparison on My School between Brighton Grammar School in Melbourne with Dargo Primary School in the Dargo High Plains, a comparison that has since been removed from the website. However, it does serve as a stark illustration of the potential bias that may arise by using area-based measures of school SES.
Brighton Grammar has an ICSEA rating of 1169 and 93% of its students come from the top quarter of the index. Dargo Primary School had one student in 2008 and none in 2009.
According to the 2006 Census, there were 31 families in Dargo who stated their income. Twenty-three earned less than $1000 a week while three earned over $2500 a week. Only 18% of those over 20 years of age had completed Year 12; two-thirds had certificate level post-school qualifications and one-third had higher education degrees or diplomas and over half of employed persons worked in lower skill occupations.
If the one student at Dargo Primary School in 2008 was from one of the few high income, well-educated and higher skilled families in Dargo the comparison with Brighton Grammar may have been appropriate. However, if that student was from one of the majority low income, less educated and lower-skilled families the comparison was totally misleading and unfair.
These are not isolated examples. There are many others on My School.
Another source of bias in the ‘like school’ comparisons is the enrolment of international students by many high fee private schools. Examples include Ascham, Kambala, and the King’s School in Sydney; Melbourne Grammar School, Melbourne Girl’s Grammar School and Scotch College in Melbourne; Geelong Grammar; and Prince Alfred College in Adelaide.
My School does not indicate how international students are classified in school ICSEA ratings, but presumably they are excluded because it is not possible to geo-code their addresses to a Census collection district. Excluding these students also results in lowering the measured SES of relevant schools because it is only wealthy international families who can afford the high tuition and boarding fees and associated costs of sending their children from overseas to Australia. Admittedly, this bias may not be large because of the relatively small number of international students, but it does add to the bias inherent in ICSEA.
Thus, My School fails to compare like with like. It tends to compare higher SES private schools with lower SES government schools. The ICSEA ratings of schools suffer from what the Australian Bureau of Statistics terms the ‘ecological fallacy’ of area-based measures of SES.
This systematic bias in the measurement of school SES causes private schools to be shown in a more favourable light in comparison with so-called ‘like’ government schools because students from higher SES families tend to have higher average results than students from lower SES families.
Whatever the extent of the bias in particular comparisons, it is scandalous that My School uses a measure to make supposedly objective ‘like school’ comparisons but which is systematically biased against government schools. Not only is it unfair in the way it will affect the reputations of schools and the careers of teachers and principals, but it will also mislead parents in choosing schools and mislead policy makers in drawing conclusions from comparisons of so-called ‘like schools’. It is a travesty of ‘like school’ comparisons.
Commenting is closed for this article.