Data for Accountability

Data has become the vehicle of choice for ensuring accountability. Government mandated reform is spearheaded by a focus on results, with demands for evidence firmly embedded (Fullan, 2000). Nations, states, provinces and school districts have implemented large-scale assessment systems, established indicators of effectiveness, set targets, created inspection or review programs, tied rewards and sanctions to results and many combinations of the above (Whitty et al., 1998; Leithwood et al., 1999). In the new educational orthodoxy (Hargreaves et al., 2001), large-scale assessment and testing has moved from being an instrument for decision-making about students to being the lever for holding schools accountable for results (Firestone et al., 1998). Leaders in states, districts and schools are required to report publicly about their efforts and demonstrate their successes.

Data for Improvement

School leaders are no longer resident experts about their schools. Instead, they are faced with the daunting task of anticipating the future and making conscious adaptations to their practices, in order to keep up and to be responsive to an ever-changing environment. To succeed in a rapidly changing and increasingly complex world, it is vital that schools grow, develop, adapt and take charge of change so that they can control their own futures (Stoll et al., 2003). Schools that are able to take charge of change, rather than being controlled by it are more effective and improve more rapidly than ones that are not (Rosenholtz, 1989; Stoll & Fink, 1996; Gray et al., 1999). There is not enough time for adaptation by trial and error or for experimentation with fads that inevitably lose their appeal. In the knowledge society, the sharing of data and information has become a critical part of decision-making. Very few institutions function without relying on data. Although the social process of sharing information and using it to produce knowledge is still not well understood, educators are recognising that they need to use data. Research studies, evaluations and routine data analyses offer mechanisms for streamlining and focusing planning and actions in schools.

THREE DIFFERENT REFORM CONTEXTS

Over the past decade, we have been fortunate to be able to watch and learn from reform efforts in a number of locales, through several research and evaluation projects. The data for this paper have emerged from these studies. The data for the National Literacy and Numeracy Strategies in England come from a four-year external ‘critical friend' evaluation that we have conducted for the English DfES (Earl et al., 2003). In Manitoba, we have been evaluating the implementation and impact of MSIP on the participating schools and on education in the province of Manitoba as a whole since 1995 (Earl & Lee, 1998; Earl et al., 2001). The data from Ontario come from several studies of implementation and impact of the reforms in Ontario secondary schools (Earl et al., 2002). This experience has given us considerable insight into the way that data are being considered and used in these vastly different contexts.

In this paper, we examine the tension that school leaders are feeling about data, by considering three very different contexts in which we have been working over the past several years:

• The National Literacy and Numeracy Strategies in England.

• The Manitoba School Improvement Program in Manitoba, Canada.

• Secondary School Reform in Ontario, Canada.

Using data is a major part of the reform in each of these large-scale school reform efforts and this theme has been a sub-focus of investigation in our projects. The interviews and surveys that have been done in each of these projects allow us to highlight the multiple dilemmas experienced by educational leaders as they negotiate their relationship with data about their schools, especially within contexts of school improvement that are directed, at least in part, from outside. (For a complete description of the methodologies used in these studies, refer to the original reports cited above.)

The context of each of these three locations is very different, both in the nature and origin of reforms and in the availability and use of data. The next sections give a brief overview of each reform and the availability and use of data.

The National Literacy and Numeracy Strategies in England

The National Literacy and Numeracy Strategies (NLS and NNS) represent a major government initiative to improve classroom practice and pupil learning in literacy and mathematics in primary schools across England. The Strategies, comprehensive in design and execution, have pulled together various policy strands to provide clear direction and support for change, with new roles, high quality materials and political support. National targets aimed to increase the percentage of 11-year-olds reaching the ‘expected level'—Level 4—in annual national assessments for English and mathematics. The strong accountability system established by the previous government was continued, with school-based targets and focused attention to literacy and numeracy teaching in OfSTED inspections. At the same time, the government added many opportunities for capacity building and created a national infrastructure to support the implementation and advancement of the Strategies.

The Strategies are an ambitious professional learning programme that has involved virtually all primary schools in England. Although the Strategies were not statutory, all 20,000 primary schools received materials and access to training and regular monitoring of school performance by the national government made the Strategies a high priority for most schools.

England arguably has more data and more sophisticated data about education than any other jurisdiction in the world. The National Assessment program has been in place since 1989 and the Key Stage Assessment results form a major element of policy decisions and accountability. Primary schools implementing NLS/NNS set targets for their progress on the Key Stage Assessments that are reported publicly and monitored by the LEA and the national government. They also receive detailed reports about their results in an annual ‘Autumn Package' that includes the individual school's performance and assessment report, as well as national summaries, value-added information and national benchmark information using free school meals data to create groups of ‘like' schools. Schools are also inspected regularly by OfSTED for compliance with directives and for quality of teaching, management and so on. The reports are posted on a public website. Local Education Authorities and headteachers have responsibility for monitoring the progress and success of the Strategies and are expected to use available data to guide their monitoring and to justify their decisions.

Manitoba School Improvement Program (MSIP)

MSIP is a Canadian non-profit, non-governmental school improvement initiative. It was established to improve the learning experiences and outcomes of secondary school students in participating schools by building schools' capacities. Over the past 10 years, a Canadian foundation and later MSIP as a non-profit company have supported school-based improvement projects designed to help students at-risk remain in school and fulfil their individual educational potential, by providing multi-year grants to schools and districts to support locally-defined school improvement efforts. The schools also receive professional and technical support from a small secretariat of consultants employed by MSIP.

Schools that receive MSIP funding are engaged in school-based improvement, focused on the needs of adolescent secondary school students (especially those at risk). As part of their commitment to MSIP they agree to include an evaluation process in their projects and to produce annual evaluation reports.

MSIP involves a relatively small number of secondary schools (approx. 30 since 1991) but this number represents about 10% of the secondary schools in Manitoba. These schools have actively pursued a range of initiatives, designed to bring about whole-school change.

MSIP has always believed that thoughtful reflection based on data helps build a school's capacity to sustain improvement (Lee, 1999). Even in the early days of the program, inquiry and use of data were highlighted as important elements of school improvement and advocated by MSIP staff. In an interim evaluation report of the implementation of MSIP, Fullan et al. (1995, p. 27) indicated that 'engaging in ongoing inquiry and reflection appears to be one of the key factors separating schools with deep impact from those whose project impact is less significant.' This belief was borne out in an evaluation of the overall program (Earl & Lee, 1998) which suggested that successful MSIP schools understand and embrace the evaluation process and regard evaluation and use of data as an invaluable tool for school planning.

Although the MSIP network places value on using data for decision-making, the data available to them are almost all locally developed in the schools, as part of their internal reflections and planning. The provincial assessment programme in Manitoba has focused largely on primary schools and the assessments that have been done in secondary have not been consistent over the years. Most districts, even when they collect data about schools for management purposes, do not prepare summary reports or return the data to the schools. Because the schools (or sometimes districts) all use different systems for capturing and recording data, it is difficult to extract comparable summaries of routine data across schools. No data are reported publicly.

Secondary School Reform (SSR) in Ontario, Canada

Secondary School Reform (SSR) in Ontario was introduced in 1997 as part of a major education bill – the Education Quality Improvement Act. The goals of SSR were to improve the accountability, effectiveness and quality of Ontario's school system. The reforms included secondary school reduced from five to four years, new and more challenging curriculum, two differentiated levels of courses, mandatory community service, specified subject and skill graduation outcomes, prior learning assessment, common report cards and a mandatory literacy test as a requirement for a secondary school diploma. At the same time, the government released a new funding model that removed taxation for education from municipalities and positioned it in the provincial purview; amalgamated school boards, established school advisory councils; mandated the amount of instructional time in a teacher's day; and set an average class size for districts across the province. As might be expected, the political context in Ontario that has accompanied large-scale reform has not been smooth or pleasant. From the early days, teachers and politicians have been at loggerheads. Teachers' staged a 2-week province-wide walkout in October 1997, in opposition to ‘'the heavy hand of government'. These reforms and the rancour that accompanied them influence all secondary schools in the province.

Having data available about schools is a relatively recent phenomenon in the province of Ontario, Canada. With the exception of a few sample assessments of students during the 70s and 80s, the province had almost no history of large-scale assessment and none with ‘'high stakes' for students, schools or districts. In 1995 the government created the Education Quality and Accountability Office (EQAO), an arms-length agency of government charged with collecting, evaluating and reporting information about educational quality. Large-scale assessment of reading, writing and mathematics began in 1997 in grades 3 and 6. Later EQAO introduced a mathematics assessment in grade 9 and a reading and writing test in grade 10. This grade 10 test is a high-takes graduation requirement for secondary students. Schools receive summaries of the performance of their students on each of these assessments and they are required to report their results publicly to the community. The results are also published in local newspapers.

LEADERSHIP DILEMMAS IN A DATA-RICH WORLD

The explicit connections between data and large-scale reforms make it impossible to avoid using data. At the same time, some leaders are becoming convinced that they need to pay attention to data to focus and clarify their decisions. In these three locations, leaders have expressed both their reservations and their hopes for the use of data in school planning and change. The three jurisdictions that we are watching are very different from one another and their actual use of data is quite variable as well, with most use in England and least in Ontario. The use of data in England and Ontario is a mandated part of a much larger centrally directed reform agenda. In Manitoba, the use of data is much more ‘'home grown' and localised in individual schools. Leaders in all three locations described their worry about not understanding data. Leaders in England and Ontario, the two locations embroiled in large-scale reform, were much more concerned than those in Manitoba about the power struggle between central and local control, and worried about who to trust and how to protect themselves and their schools. All of the groups were positive about how data helped them focus their thinking and provided them with insights that were useful in making decisions. They also all expressed concern about how to translate data into useable knowledge and communicate it to their constituents, especially when they were not in control of the communication strategy.