«Graham Hobbs Anna Vignoles May 2007 Published by Centre for the Economics of Education London School of Economics Houghton Street London WC2A 2AE © ...»
CEE DP 84
Is Free School Meal Status a Valid Proxy for SocioEconomic Status (in Schools Research)?
Centre for the Economics of Education
London School of Economics
London WC2A 2AE
© Graham Hobbs and Anna Vignoles, submitted March 2007
The Centre for the Economics of Education is an independent research centre funded by the
Department for Children, Schools & Families. The views expressed in this work are those of the author and do not reflect the views of the DCSF. All errors and omissions remain the authors.
Executive Summary Across the social sciences, family socio-economic status (SES) is seen as a potentially key determinant of children’s educational attainment. However, a lot educational research in the UK relies on administrative datasets that rarely contain measures of SES. Instead, they almost always include an indicator of pupils’ “Free School Meal (FSM) Eligibility”. FSM status is widely used as a proxy for SES in UK educational research, therefore. Future research is likely to rely even more heavily on administrative data, in particular, the National Pupil Database (NPD), and hence on FSM status as a proxy for SES. There is therefore a pressing need to evaluate the validity of the FSM measure in this context. This is our aim.
The “FSM eligibility” measure is, in fact, a measure of claiming FSM, rather than just eligibility. Nevertheless, our evaluation begins by examining FSM eligibility rules and nationally-representative data on those eligible for FSM. Children in families claiming Income Support (IS) and Income-Based Job Seekers Allowance (IB-JSA) were eligible for FSM in 2001/2 (the year relevant to our main analysis). Broadly to be eligible for these benefits in that year, a child must have been in a household without a member working more than 24 hours a week, with a low income (defined relative to needs) and limited capital assets.
National data suggests that the majority of children claiming free school meals will be dependents of IS claimants (rather than IB-JSA claimants). National data also suggests the majority will be in “one parent families”. Specifically, in May 2002, over 75% of primary school age dependents of IS claimants were in “one-parent families”. National data also suggests that a significant proportion of children who are eligible to receive free school meals will remain eligible for more than five years. For example, 34% of “lone parent” claimants received IS for five or more years. Furthermore, administrative data suggests that only 10% of the cohort of children entering reception year in 1997/98 in state schools in England changed FSM status during the period of interest for this study (between January 2002 and January 2004).
Thesecond part of our evaluation examines the relationship between FSM status and other key characteristics of the child’s family. We do this by using a data set that includes both FSM status and rich data on the child’s family background, namely merged Avon Longitudinal Study of Parents and Children (ALSPAC) - National Pupil Database (NPD) data. These merged Avon data include not only FSM status of the child but also data on his or her family income, mother’s and partner’s employment, family employment, one-parent family status, mother’s and partner’s education, and mother’s and partner’s social class. The Avon data do paint a somewhat different picture to that described above from the national data. In particular, in our Avon data, 43% of children claiming FSM are in families with at least one parent in full-time employment and only 28% are in one-parent families. One explanation of these differences is that some families misreport their employment and partnership status when claiming benefits. Another explanation highlights a limitation of the data used here: the Avon measures of socio-economic status (SES) are observed five or more years before the FSM measure (from NPD) and so may genuinely change over time.
With the above caveat in mind, we investigated the range of SES measures to determine which measures FSM status proxies “best”. We find that FSM status “best” proxies children in households with family incomes below £200 per week, i.e. the bottom quartile of the income distribution, rather than the bottom decile (even though less than 10% of children claim FSM in this sample). However, we do observe apparent anomalies. Firstly, 22% of FSM children have incomes above £200 per week and secondly, 20% of non-FSM children have incomes below £200 per week. Thus FSM status does not appear to identify all lowincome children in our data, although we acknowledge the problem above about the timing of the data we have.
For family employment, FSM status proxies “best” children in workless families and those with only one part-time worker. While only 8% of non-FSM children are in workless families and those with one part-time worker, 43% of FSM children are in families with one or more full-time workers, or two part-time workers.
FSM status is a far from perfect proxy of one parenthood. In particular, 72% of FSM children have two-parents. FSM status is also a far from perfect proxy of mothers’ and partners’ education and social class.
We then investigated the use of FSM as an imperfect proxy for true SES in two specific contexts. Firstly, when researchers are trying to investigate differences in educational achievement for low and high SES children, where FSM status is the variable of interest.
Secondly, when researchers want to simply take account of family background and SES in their model and therefore include FSM status as a control variable in an OLS regression. We find that the bias produced by using FSM instead of true SES is context-specific.
Firstly, we assessed the extent of imperfect proxy bias when estimating differences in Key Stage 2 attainment by family income, using FSM status to proxy incomes below £200 per week. FSM status is the variable of interest, here. In English, maths and science, the difference in the average Key Stage 2 attainment of children with family incomes below and above £200 per week is 0.5 standard deviation units. However, the difference in the average Key Stage 2 attainment of FSM and non-FSM children is 0.6-0.7 standard deviation units.
The difference in these differences is the imperfect proxy bias. The bias is around 30-40% of the “true” difference and is significant at the 1% level.
Finally, we assess the bias in a model of the effects of school type and Special Educational Needs status on a child’s Key Stage 2 attainment. FSM status is a control variable, here. We estimate three specifications: 1) a “true” regression including our SES measures, 2) an omitted variables regression just omitting the SES measures, and 3) a proxy variable regression omitting the SES measures but including FSM status. The omitted variables regression equation suffers from omitted variables bias because it does not allow for the effects of SES. The proxy variable regression suffers from imperfect proxy bias because it includes only a proxy rather than true measures of SES. The amount of this bias is the difference in the parameter in the proxy variable and “true” regressions.
FSM is status is statistically significant in the proxy variable regressions, so it does predict achievement in the absence of true SES measures. However, FSM status does not always do a good job as a proxy for the true SES of the child. In general using FSM status reduces the bias caused by omitting SES altogether by only 10-25%. Furthermore, using FSM status instead of true SES generates imperfect proxy bias, which in our model was substantial on the voluntary-aided and voluntary-controlled schools coefficients. The imperfect proxy bias on these coefficients is 90% and 75-80% of the omitted variables bias, respectively (the reference category is community schools). On Special Educational Needs, it is 75-80%. So the biases arising from using FSM status instead of true SES are significant and sometimes large. Moreover, using FSM status instead of true SES sometimes changes the key findings from the model. For instance, the statistical significance of the school type effects are sometimes different in the “true” and proxy variable regressions, influencing the basic interpretation of the results. For example, in maths and science, the effect of voluntary-aided schools is positive and significant in the proxy variable regression, but insignificant in the “true” regression.
Future research should evaluate the FSM measure in other contexts and datasets, and the use of various small area data matched to children’s home postcodes as proxies for measures of SES. In the meantime, researchers should be cautious in drawing inferences from research reliant on the FSM measure. When used as the variable of interest, FSM status is an imperfect proxy of low income or “workless” families, or one-parenthood. In the context of estimating differences in educational attainment by family income, the bias generated by using FSM as opposed to “true” SES is quite large. When used as a control variable in an OLS regression, FSM status reduces omitted variables bias to a moderate extent only. In other words, if omitted variables bias is a concern, then the inclusion of FSM status in the model should do little to diminish this concern.
Is Free School Meal Status a Valid Proxy for Socio-Economic Status (in Schools Research)?
The authors would like to acknowledge the helpful and constructive comments of those who attended the Bedford Group seminar and the PLUG seminar in Bristol, at which this paper was presented. Particular thanks go to Rebecca Allen and Nikos Tzavidis for their input into this paper.
Graham Hobbs is a Research Officer for the Centre for the Economics of Education at the Bedford Group for Lifecourse and Statistical Studies, Institute of Education. Anna Vignoles is a Reader in Economics of Education at the Bedford Group for Lifecourse and Statistical Studies, Institute of Education, a Research Associate at the Centre for Economic Performance, London School of Economics and Deputy Director of the Centre for the Economics of Education.
1. IntroductionAcross the social sciences, family socio-economic status (SES) is seen as a potentially key determinant of children’s educational attainment. 5 However, a lot educational research in the UK relies on administrative datasets which rarely contain measures of SES. Instead, they almost always include an indicator of pupils’ “Free School Meal (FSM) Eligibility”. FSM status is widely used as a proxy for SES in UK educational research, therefore. Future research is likely to rely even more heavily on administrative data, in particular, the National Pupil Database (NPD), and hence on FSM status as a proxy for SES. There is therefore a pressing need to evaluate the validity of the FSM measure in this context. This is our aim. 6 In some research, FSM status is the “variable of interest”. In this context, it is typically used to proxy low income. One example is research on differences in educational attainment and progress by FSM status (e.g., Sammons et al., 1997a; Strand 1999; DfES 2003; DfES 2005b).
Another example is research on the effects of school composition on educational attainment.
Here the proportion of FSM pupils proxies the proportion of low-income pupils at schoollevel (e.g., Sammons et al., 1997b; Strand 1997; DfES 2003; Hutchison 2003; Schagen and Schagen 2005). A third example is research on socially-segregated schooling. Here, FSM status proxies low income in measures of segregation (e.g., Gibson and Asthana 2000; Noden 2000; Gorard et al., 2002, 2003; Goldstein and Noden 2003; Allen and Vignoles 2006).
In other research, FSM status is a “control variable”. In particular, FSM status is included in models of educational attainment to eliminate or reduce the extent of omitted variables bias.
In this context, FSM status is used, sometimes implicitly, as a proxy not just for income, and not just low income, but also for other unobserved SES variables. Important examples have
Appendix 1 provides a brief review of evidence on the effects of SES variables on educational attainment in the UK.
There has been little formal evaluation of the FSM measure. Croxford (2000) is a notable exception.
1. Studies of “ethnicity gaps” in educational attainment (Strand 1999; DfES 2003);
2. Studies of the effects of pupil mobility (Strand 2002);
3. Evaluations of education policies, e.g., Excellence in Cities (Machin et al., 2004) and the Literacy Hour (Machin and McNally 2004); 7
4. Studies of selective versus comprehensive school systems (Atkinson et al., 2006);
5. Studies of the effects of class size (Blatchford et al., 2003);
6. Studies of the effects of school resources (Levacic et al., 2005); and
7. School effectiveness research (Strand 1997; Thomas et al., 1997).
The first part of our evaluation examines FSM eligibility rules and nationally-representative data on those eligible for FSM. The second part examines the joint distributions of FSM status and nine SES measures: family income, mother’s and partner’s employment, family employment, one-parent family status, mother’s and partner’s education, and mother’s and partner’s social class. We report the binary indicator of each SES measure which FSM status proxies “best”, along with the associated probabilities of “false positives” and “false negatives”.
The extent of imperfect proxy bias is context-specific. The third part of our evaluation examines two contexts, one when FSM status is the variable of interest, the other when FSM status is a control variable in an OLS regression. First, we assess the extent of bias when estimating differences in Key Stage 2 attainment by family income, using FSM status to proxy incomes below £200 per week. FSM status is the variable of interest, here. Second, we assess the extent of bias when estimating the effects of school type and Special Educational Needs on Key Stage 2 attainment, using FSM status to proxy eight SES measures. FSM status is a control variable, here.