«Grahame Cook PSM Director ISBN 978-1-76028-061-1 [PDF] 978-1-76028-062-8 [DOCX] With the exception of the Commonwealth Coat of Arms, the ...»
Review of My School
Final Report to the Australian Government
Department of Education
Grahame Cook PSM
With the exception of the Commonwealth Coat of Arms, the Department’s logo, any material
protected by a trade mark and where otherwise noted all material presented in this document is
provided under a Creative Commons Attribution 3.0 Australia
The details of the relevant licence conditions are available on the Creative Commons website (accessible using the links provided) as is the full legal code for the CC BY 3.0 AU licence (http://creativecommons.org/licenses/by/3.0/au/legalcode).
The document must be attributed as the Review of My School.
Grahame Cook Consulting Pty Ltd ABN 59 132 859 977 PO Box 4 Sunnybank Qld 4109 Disclaimer While Grahame Cook Consulting Pty Ltd endeavours to provide reliable analysis and believes the material it presents is accurate, it will not be liable for any claim by any party acting on such information.
This document takes into account the Terms of Reference and requirements of the Australian Government Department of Education.
Acknowledgments Grahame Cook Consulting would like to thank the Executive and staff of the Australian Curriculum Assessment and Reporting Authority for freely sharing information and data related to the My School website throughout the course of this review.
Grahame Cook Consulting would also like to thank all those with whom informal consultations were held as part of the My School review process. Their real world experience of the My School website provided valuable insights for the review.
Grahame Cook Consulting also wishes to thank and acknowledge the support provided throughout the review by officers of the Evidence Services Group of the Australian Department of Education The views expressed in this review and any errors and omissions remain the responsibility of the author.
Table of Contents Acknowledgments 3 Acronyms 7 Executive Summary 8 Background 8 Main issues 8 Key findings 9 Recommendations 11 Strategic issues and next steps 12 Website design and data release 13 Chapter 1 – The My School Initiative 14 The My School Website 14 Investment in My School Website 14 Stated purpose of My School 15 Key My School Stakeholders and Information Needs 16 Parents and
Acronyms Acronym Description ABS Australian Bureau of Statistics ACARA Australian Curriculum Assessment and Reporting Authority ATAR Australian Tertiary Admission Rank CAPTCHA Completely Automated Pubic Turing test to tell Computers and Humans Apart COAG Council of Australian Governments EdStats World Bank Education Statistics FAQ Frequently Asked Question FDWG ACARA Finance Data Working Group FTE Full time equivalent ICSEA Index of Community Socio-Educational Advantage IPS Independent Public Schools LBOTE Language Background other than English MCEECDYA Ministerial Council on Education, Early Childhood Development and Youth Affairs MCEETYA Ministerial Council on Education, Employment, Training and Youth Affairs NAPLAN National Assessment Plan - Literacy And Numeracy NYC New York City OECD Organisation for Economic Cooperation and Development PDF Portable Document Format PIAAC Progress in International Assessment of Adult Competencies PIRLS Progress in International Reading Literacy Study PISA Programme for International Student Assessment TIMMS Trends in International Mathematics and Science Study SCSEEC Standing Council on School Education and Early Childhood SWD Students with Disabilities TVASS Tennessee Value-Added Assessment System UK United Kingdom USA United States of America VET Vocational Education and Training WCAG 2.0 Web Content Accessibility Guidelines version 2
As part of its Policy for Schools: Students First platform for the 2013 Federal Election, the Government committed to review the My School website (www.MySchool.edu.au). The Australian Department of Education subsequently commissioned a review of My School to provide
• The success of the site in meeting the original purpose of My School;
• How information published on My School is currently used and displayed;
• The level to which the current suite of information available supports the Government’s reform agenda;
• The publication of additional indicators; and
• Opportunities for improvements.
The advice is to identify options for changes to the My School website and make recommendations for those changes. The full terms of reference for the review are at Appendix 1.
My School has not previously been formally reviewed although an evaluation of the website is being progressed by the Australian Curriculum Assessment and Reporting Authority (ACARA).
The My School website is inherently complex due to nature of the National Assessment Plan Literacy and Numeracy (NAPLAN) data covered and sensitivities about comparisons between schools and the interpretation of the data and information presented. This has been exacerbated by the status of ACARA as a multi-jurisdictional body and complex governance arrangements, which make decision making a process of negotiation and allow for jurisdictional vetos at both the senior official and Ministerial Council levels. The end result can be less than optimal outcomes in the policy framework in which My School is developed and implemented.
Much of the sensitivity around My School relates to the NAPLAN data and concerns about it being misinterpreted or used for unproductive purposes, such as development of “league tables”, which can draw inappropriate comparisons between schools. The web site has been designed to try to overcome these problems, often at the expense of useability.
A related issue is the interpretation of “similar” schools, which are based on schools having students with statistically similar background known as the Index of Community Socio-Educational Advantage (ICSEA). ACARA has put substantial effort into improving this index to make it more robust and stable over time. Nevertheless it is recognised that the ICSEA is not designed to capture the qualitative factors that influence parents and school communities’ perceptions about their school.
A key issue that flows from consideration of these matters is the primary purpose of My School and the key audience it is expected to serve. During 2008 and 2009 the Commonwealth Government pushed a transparency and accountability agenda through the Council of Australian Governments (COAG) and Ministerial Council processes as part of a broader educational reform agenda.
In making the decision to establish My School the then Ministerial Council agreed that there were
three key purposes for individual school performance reporting, namely:
1. To enable evaluation of school performance - to allow governments and researchers to analyse how well schools are performing by reference to nationally comparable data and to their own performance over time.
2. For public accountability (and transparency) purposes - to improve outcomes and equity for all students by using nationally comparable school performance data to build a substantive evidence base to support future improvements.
3. For better resource allocation – to enable identification of where resources are most needed to lift attainment as an aid to policy makers and to build pressure for change by placing parents and the broader community in the same position as education officials.
The current My School website says that the website “has two main purposes:
• It provides parents and students with information on each school – its view of itself and its mission, its staffing, its resources and its students’ characteristics and their performances.
• It provides schools and their communities with comparisons of their students’ performances in literacy and numeracy with those of students in other schools, most importantly those in schools that serve similar students. These comparisons provide information to support improvements in schools.” The Government’s Policy for Schools: Students First undertakes to improving schools and education outcomes through policies to deliver more say for parents and teachers, a sound national curriculum, improved quality teaching, and certainty over school funding. The key government policies relevant to this review are parent and community engagement; school autonomy;
Indigenous closing the gap targets; and development of an evidence base of performance data. The My School website has the capability to help advance this agenda.
My School represents a valuable, nationally consistent data set on Australian schools across all jurisdictions and all sectors. The systematic collection of new data, the refinement of its statistical underpinnings and further development of the system architecture has increased the potential value of the website over time. However, My School is a complex website with a lot of technical and statistical information that is not designed to be general user friendly. Consequently, the national data repository that My School now represents is probably underutilised.
There is some lack of clarity among stakeholders about the core purpose of My School and therefore its audience. The Commonwealth has a policy focus on transparency and accountability through publication of nationally consistent school level data. This is based on the absence of such data prior to My School and international evidence that clear accountability assists create a learning environment that encourages innovation and excellence. From a Commonwealth perspective, as a major funder and with both social policy and economic interests in educational outcomes, My School serves a valuable purpose.
For parents qualitative research results suggest that My School is perceived to be mainly about NAPLAN scores and academic performance, which is one aspect of how they make choices about schools. Nevertheless some parents use the website for helping choose a school, monitoring school performance and providing context for their own child’s achievement. Use of My School by principals and school system administrators is limited, in part because school systems are using NAPLAN and broader data to help drive school improvement. It would be useful to re-articulate the purpose of My School to focus on national transparency and accountability at the school level, while acknowledging its important benefits as a tool for educational improvement.
There is only limited data on the usage of My School, including by whom and for what purpose. In general usage of My School has been tending to decline but is still substantial. The available data shows that that there is spike in usage of My School at the time that new releases are made and to a lesser extent at the time NAPLAN tests are conducted and when the results are released. It could be expected that the main users at these times would be parents and school leaders.
The functionality of My School has been enhanced over time and the quality of data improved, especially the ICSEA methodology. Moreover, the My School work plan provides for the addition of further data sets in a phased manner. However, the website’s functionality and useability remains constrained by the complex Commonwealth, state and territory governance and policy framework in which ACARA operates. The My School design has been partly driven by the policy objective of making the collation and publication of league tables difficult, reflecting education union and other stakeholder concerns.
The available evidence suggests that the full functionality and utility of My School generally is not well understood, either by parents or teachers. As NAPLAN data relating to individual student performance is directly made available to parents and schools, the value added by My School relates to nationally consistent data for a school over time and comparative data for schools having students with statistically similar backgrounds and for all schools. Importantly, this includes where available student educational growth as presented in the NAPLAN student gain page.
My School website traffic data shows that the school profile, NAPLAN results in numbers and NAPLAN results in graphs pages together account for three-quarters of all page views. Other pages such as NAPLAN results in bands, VET in schools, local schools and school finances represent only a small proportions of total page views. The usage of the student gain page is small but appears to be increasing. This simple analysis indicates that most users are only interested in a small proportion of the information on offer or may find the website too complex to explore it more fully.
As a data rich website having a lot of information presented in graphical and tabular form with colour coding, statistical and educational terminology and explanatory notes and caveats, it is understandable that ordinary users may find the website challenging. Moreover, interpretation of the data can be difficult as, for example, the average NAPLAN results of students at a school can vary by year and by domain and margins of error overlap. A more sophisticated understanding of school performance than is offered by My School is required by school leaders in order to design educational interventions but the website can prompt important conversations.
With the 2015 release of My School a total of seven years of school level data will be publicly available. This means that for the first time results will be available for students who sat the first NAPLAN tests in year 3 in 2008 and have completed the full cycle of year 3, 5, 7 and 9 annual assessments. This data set will be very valuable to anyone interested in school performance and student gain over the 2008 to 2014 period. It also opens up the possibility of different ways of presenting data such as in time series form, which would be welcomed by parents.