Teacher professional development is discussed as one of the key factors in improving classroom instruction. Thereby it is argued that conventional approaches to teacher professional development seem not to be meeting the needs of teachers and therefore are considered less effective than alternative approaches. Those alternative approaches often take on a situated perspective (Borko, 2004) and are linked to national educational reform efforts.

However, the question as to whether one particular approach is more effective than the other depends on scientifically based research. Educational researchers rely on evaluation tools to make evidence-based judgements. This symposium brings together researchers dealing with professional development approaches in different countries using different evaluation tools. Central questions being dealt with in the symposium are:

-     Which tools are …

-     … being used for evaluating teacher professional developement?

-     … adequate for different professional development approaches?

-     … being accepted by the teachers themselves?

-     … well suited to provide information on the effects of professional development on

-      teacher cooperation and quality development,

-      teachers’ views on teaching and learning,

-      changes in classroom practices (methods, used classroom materials, …) or

-      learning outcomes on the side of the students?

The researchers present an array of different evaluation tools that are currently beeing used to analyze those aspects. To name some, evaluators employ teacher and student questionaires, performance tests, classroom observations, or teacher portfolios.

In the symposium advantages and disadvantages of different tools will be discussed on the basis of empirical findings linked to different teacher professional development programs. Because teacher professional development is strongly dependend on the cultural context, the symposium brings together examples from four different countries, namely the USA, Austria, Switzerland and Germany.

Borko, H. (2004). Professional development and teacher learning: Mapping the terrain. Educational Researcher, 33(8), 3-15.

In this study the adaptive teaching competency of 49 primary and secondary school teachers and the learning achievement of their 890 students are examined. Ideally, teaching takes situated factors such as students’ preconditions into account. Teachers’ constant adjustment of planning and teaching to the individual learning processes of diverse students is what we mean by 'adaptive teaching competency'. At the heart of adaptive teaching competency is students’ learning for understanding.

To measure adaptive teaching competency new tools were developed. A vignette was employed to measure adaptive planning competency, a video test to measure adaptive implementation competency and a teacher science test to measure teachers’ subject knowledge. The theoretical structure of adaptive teaching competency is confirmed empirically employing confirmatory factor analysis. Results of multilevel analyses and structural equation modelling are presented. Primary findings show a correlation between adaptive teaching competency and the quality of teaching as perceived by the students. High adaptive teaching competency for example seems to correlate with a better fit of teaching methods on students’ preconditions, with higher student participation and higher teaching quality. These results can be interpreted as criterion-related validation of the construct adaptive teaching competency.

In addition, first analyses show that students of highly adaptive teachers demonstrate larger achievement gains than students of teachers with lower adaptive teaching competency. Teachers’ diagnostic planning competency seems to be especially relevant for student success.

Learning is a highly individual process and is basically determined by subjects' preconditions. Ideally, teachers take these situated factors into account through the adjustment of their planning and teaching to the individual learning processes of diverse students. This constant adjustment of planning and teaching to the learning processes of the students as individual learners is what we mean by Adaptive Teaching Competency.

In accordance with Weinert & Helmke (1996) the construct of adaptive teaching competency includes the following four dimensions:

(1) Subject Knowledge, (2) Diagnosis of Students’ Learning, (3) Methods of Teaching, and (4) Classroom Management.

Subject Knowledge stands for in-depth, differentiated, clearly structured, and transparent content knowledge of a specific topic. According to Piaget and Aebli such knowledge needs to be highly operational. Diagnosis of Students‘ Learning stands for the teacher‘s knowledge (a) of learning in the given domain and (b) of individual preconditions, characteristics, and special needs of each particular learner. The competency Methods of Teaching stands for a teacher‘s repertoire of teaching approaches, which includes (a) systematic instruction and (b) the facilitation of independent learning. Classroom Management stands for the teacher‘s competency to create conditions, which facilitate systematic and effective student learning.

To measure Adaptive Teaching Competency new tools were developed (i.e., vignette, video test, and teacher science test). First analyses of the data show that adaptive planning competency was significantly increased by an intervention that included content-focused coaching (West & Staub, 2003) and a two-day seminar.

Aims

The main objective of this study is to gain a deeper understanding of the effects of adaptive teaching competency on classroom teaching and student learning.

Research questions:

a)     Are the tools developed within this research project suitable to measure different components of adaptive teaching competency?

b)     Can the theoretical structure of adaptive teaching competency be confirmed empirically?

c)      What are the relationships between adaptive teaching competency and the students' perception of teaching and between adaptive teaching competency and teachers' beliefs?

d)     What impact does adaptive teaching competency have on student learning?

Methodology / research design

Following an information session, 26 primary (grades 4 and 5) and 23 secondary (grades 7 and 8) teachers from Swiss public schools volunteered to participate in this study with their students (n = 890).  All teachers were teaching science classes and two thirds of them had more than six years of teaching experience.

Structural equation modelling is used with latent variables to take the multifaceted structure of school achievement (Helmke & Weinert, 1997) into account. Our units of analyses include teachers and students nested within classrooms as well as situational factors within the classroom and individual preconditions of students which impact teaching and learning. This multilevel structure of the data requires the application of hierarchical linear models. With these analyses student and teacher variables can be investigated simultaneously and contextual effects of classes can be controlled for.

Learning success can only be determined in a longitudinal setting. To that effect students took tests in the given topic ‘germination of seeds’ before and after the taught unit. Students were given different test booklets for pre- and post-tests using multi-matrix sampling. To calculate test scores, item response theory (IRT) was used. The analysis also involves examining student learning outcome in different ability groups. The effects on student learning will also be examined for different ability groups.

Findings

To verify the construct validity of adaptive teaching competency a confirmatory factor analysis (CFA) was employed. The result shows that the data which was gathered through the vignette and the video test fit the theoretically proposed construct.

Primary findings show a correlation between adaptive teaching competency and the quality of teaching as perceived by the students. High adaptive teaching competency correlates with a better fit of teaching methods on students’ preconditions, with higher student participation and higher teaching quality. These results can be interpreted as criterion-related validation of the construct adaptive teaching competency.

In addition, first analyses on the classroom level show that students of highly adaptive teachers demonstrate larger achievement gains than students of teachers with lower adaptive teaching competency. Teachers’ diagnostic planning competency seems to be especially relevant.

Theoretical and educational significance

Findings of this study contribute to a better understanding of relationships between teaching competencies, the actual teaching (implementation) and student achievement. In particular, the construct adaptive teaching competency might help bridge the gap between teachers' knowledge and implementation (Mandl & Gerstenmaier, 2000).

Since diagnostic planning competencies are especially important for students’ learning, there should be a higher emphasis on planning. Strategies for diagnosing students’ preconditions such as prior knowledge, learning strategies, and motivational orientation should be improved. 

Since adaptive teaching competency is related to students’ learning gains it is an important concept for the training of teachers and their further education. Tools such as the video test could be very useful for training sessions.

References

Helmke, A. & Weinert, F. E. (1997). Bedingungsfaktoren schulischer Leistungen. In F.W. Weinert (Hrsg.), Enzyklopaedie der Psychologie, Bd. 3: Psychologie des Unterrichts und der Schule (S. 71-176). Goettingen; Bern; Toronto; Seattle: Hogrefe.

Mandl, H. & Gerstenmaier, J. (2000). Die Kluft zwischen Wissen und Handeln. Empirische und theoretische Loesungsansaetze. Goettingen: Hogrefe.

Weinert, F.E. & Helmke, A. (1996). Der gute Lehrer: Person, Funktion oder Fiktion? Zeitschrift fuer Paedagogik, Beiheft (34), 223-233.

The National Science Foundation (NSF) initiated the Local Systemic Change funded 88 multi-million dollar projects. The initiative’s primary goal was to improve instruction in science, mathematics, and technology through teacher professional development (PD) within whole schools or school districts. LSC projects were expected to align policy and practice, to engage in a range of activities to support reform, develop clearly defined, measurable outcomes for teaching, and an evaluation plan that provided formative and summative feedback.

This paper focuses on the evaluation of one of the LSC projects that provided 100 hours of PD to 2000 teachers across 8 school districts that serves over 55,000 students. The author of this paper was the Lead Evaluator and part of the national Core Evaluation. The contribution of this work to the field is how a system-wide evaluation can serve both the needs of local sites for formative and summative feedback and also provides stakeholders and policy-makers data on the overall impact of a national initiative.

All of the evaluation activities were driven by a set of core evaluation questions:

· What is the overall quality of the professional development activities?

· What is the extent of school and teacher involvement in LSC activities?

· What is the impact of the LSC professional development on teacher preparedness, attitudes, and beliefs about mathematics and science teaching and learning?

· What is the impact of the LSC on classroom practices in mathematics and science?

· To what extent are the district and school contexts becoming more supportive of the LSC vision for exemplary mathematics and science education?

· What is the extent of institutionalization of high quality professional development systems in the LSC districts?

· The project focused on in this paper also collected data on the impact of the PD on student learning.

In 1995, the National Science Foundation (NSF) initiated the Local Systemic Change through Teacher Enhancement program. The initiative’s primary goal was to improve instruction in science, mathematics, and technology through teacher professional development (PD) within whole schools or school districts.

The LSC initiative distinguishes itself from former NSF-supported teacher enhancement efforts in two important ways. First, it targets all teachers in a school district or multiple districts for professional development; each targeted teacher is to participate in a minimum of 100 hours of professional development. Second, the LSC emphasizes preparing teachers to implement district-designated mathematics and science instructional materials in their classes.

In addition to providing professional development for teachers, the LSC initiative promotes efforts to build a supportive environment for improving science, mathematics, and technology instruction. LSC projects were expected to align policy and practice within targeted districts, and to engage in a range of activities to support reform, develop clearly defined, measurable outcomes for teaching, and an evaluation plan that provided formative and summative feedback.

This paper focuses on the evaluation of one of the LSC projects that provided 100 hours of PD to 2000 teachers across 8 school districts that serves over 55,000 students. The author of this paper was the Lead Evaluator and part of the national Core Evaluation discussed below. The contribution of this work to the field is how a system-wide evaluation can serve both the needs of local sites for formative and summative feedback and also provides stakeholders and policy-makers data on the overall impact of a national initiative.

All of the evaluation activities were driven by a set of core evaluation questions:

What is the overall quality of the LSC professional development activities?

What is the extent of school and teacher involvement in LSC activities?

What is the impact of the LSC professional development on teacher preparedness, attitudes, and beliefs about mathematics and science teaching and learning?

What is the impact of the LSC on classroom practices in mathematics and science?

To what extent are the district and school contexts becoming more supportive of the LSC vision for exemplary mathematics and science education?

What is the extent of institutionalization of high quality professional development systems in the LSC districts?

In addition to this core set of questions the project discussed in this paper also collected data on the impact of the PD on student learning.

The Core Evaluation

NSF’s solicitation for the LSC initiative indicated the Foundation’s interest from the outset in providing a framework for collecting data from LSC projects to evaluate their efforts. NSF contracted with Horizon Research, Inc. (HRI) in Chapel Hill, North Carolina to develop a data collection framework, to provide technical assistance in implementing evaluation activities, and to prepare cross-site analyses of evaluation results.

Since the LSC’s inception, HRI collaborated with NSF staff, LSC Principal Investigators (PIs), and project evaluators on the design and implementation of a core evaluation system. The system includes the collection of baseline data during an LSC’s first year of funding, and a range of data collection activities during subsequent years.

Evaluation Tools employed

While LSC projects are bound by goals and expectations set by NSF, they also must develop a strategy that is responsive to local context and needs. Although the result has been a set of LSC initiatives that range considerably in design and implementation strategies, HRI created an evaluation system that allows information to be aggregated across these diverse LSC projects. Project evaluators are asked to collect data using standardized questionnaires and protocols designed to answer core evaluation questions, and to complete ratings on the quality of LSC professional development programs. Data collection activities completed by evaluators of each LSC project include the following:

• Observations of Professional Development Activities

The core evaluation requires evaluators to conduct observations of 5 – 8 professional development sessions each year, and to record their observations on standardized protocols.

• Classroom Observations

Ten teachers were randomly selected for the Core Evaluation observations (the project of this paper observed 20 teachers annually). These teachers, or their randomly selected back-ups, were to be observed in the spring, at the beginning, midway, and end of the project. 55 percent of the lessons observed were taught by teachers who had participated in at least 20 hours of LSC professional development. In all cases, the data were weighted to represent the total population of eligible teachers in the project.

• Teacher Questionnaires

Each project administered questionnaires to a random sample of 250 teachers. Teacher questionnaires asked respondents about attitudes, beliefs, preparedness to teach, instructional practices, and so on. Projects were expected to achieve an 80 percent response rate for the teacher questionnaires.

• Principal Questionnaires

Projects also administered questionnaires to the entire population of principals of targeted schools each year. For principal questionnaires, projects were expected to achieve a response rate of 100 percent.

• Teacher Interviews

Evaluators in each of the projects were asked to interview a sample of teachers who participated in at least 20 hours of LSC professional development activities.

• Interviews with LSC Principal Investigators

All evaluators interviewed the PI of their projects annually.

• Project Ratings

Each year, the evaluators and PIs submitted ratings of the quality of their LSC professional development programs. Ratings of the supportiveness of the context for the LSC vision and likelihood of institutionalization of the LSC reforms were submitted in projects’ Baseline Year, Year Two, and Final Year of data collection.

• Not required by the Core Evaluation but will be reported in this presentation is the impact of the project on student learning.

Results and Conclusions

In the presentation selected findings of the evaluation will be presented. The results will be discussed with reference to the outcomes of the professional development as well as to the benefits and costs of using a systemwide evaluation approach to meet the needs of both the systemwide and local needs

As a reaction to the poor results of Austrian high school students in the TIMSS achievement test, the IMST project was started. The so called “IMST Fund” is a national supporting system for the development of classroom instruction and schools. Its aim is to effectively stimulate innovations in the teaching of mathematics, science, technology and information and computer technology as well as in teacher training. The Fund provides financial, organisational, and evaluation support as well as consultation for teachers’ innovation projects.

This study investigates how the quality of students’ self-related cognitions and emotions change within the projects first year.

Of special interest is which conditions of the learning environment are responsible for the stability or respectively the change of self-related cognitions and emotions.

Deci and Ryan’s (2002) self-determination theory (SDT) provides the theoretical foundation of this paper. It allows a differentiated analysis of the qualities of learning motivation and also suggests that motivational processes are highly influenced by basic psychological needs for autonomy, competence and social relatedness. According to SDT it is supposed that the IMST projects develop powerful learning environments and foster self-related cognitions (e.g. self concepts) and emotions (less subject-related fear). Furthermore, self-determined and satisfied teachers should create supportive learning environments which positively influence students’ learning processes.

1. The IMST project

IMST - Innovations in Mathematics, Science and Technology Teaching is a national initiative financed by the Ministry of Education, Science and Culture for the areas of mathematics, science and technology, information and computer technology and related subjects. IMST was initiated after the Austrian participation in the Third International Mathematics and Science Study (TIMSS 1995). The first aim of IMST (1998-1999) was to examine the causes of the low performance of Austrian students in upper secondary schools. In 2000-2004, about 50 projects at upper secondary schools (and partially at other organisations, e.g. teacher education institutions) were supported (Krainer et al., 2002). In 2004, the project (now IMST3) was extended to all secondary schools. IMST3 comprises seven core measures. These measures are implemented by a wide network of persons and institutions. The “IMST Fund” is one of these measures in order to effectively stimulate innovations in the teaching of mathematics, science, technology and information and computer technology as well as in teacher training. Teachers are encouraged to submit fund-proposals for projects they want to realize in the following school year (Krainer, 2005; in press). The Fund provides financial, organisational, and evaluation support.

2. The study design

The main aim of the study is to examine the changes and stabilities of students’ self-related cognitions and emotions within one year of carrying out an IMST Fund project. Because these projects follow a bottom up-philosophy, innovations and interventions are of a heteronymous nature. This complicates the comparability of the contents of the projects and their effects on students’ learning processes. For this reason we created an exploratory study focussing on students’ perceptions of the learning environments and self related cognitions and emotions.

The second aim of the study is to investigate teachers’ motivation and their related working conditions at school.

Using structural equation modelling, the interdependence of working conditions, teachers’ motivation, students’ perceptions of the learning environment and students’ self-related cognitions and emotions can be examined. Working conditions at school are measured as perceived autonomy support, respectively restraints (pressure) for teachers (see Pelletier, Seguin-Levesque & Legault, 2002). This study design follows the idea that the general conditions of schools and the school system influence teachers’ motivation to work as well as the organisation’s climate for innovations. Furthermore, these factors should indirectly determine classroom instruction and students’ self-related cognitions and emotions.

The study is conceptualized as longitudinal study that was performed in the classes taking part in the IMST Fund. The first survey is operated in November/December 2006; the second will be executed in June 2007. The sample includes about 1400 students and 60 teachers.

3. Theory

The theoretical basis is a multidimensional perspective of motivation, the self-determination theory (SDT) of Deci and Ryan (2002a, 2002b). It is proposed that perceived support of basic psychological needs (support of autonomy, support of competence, and social relatedness) are associated with intrinsic motivation or self-determined forms of extrinsic motivation.

Accordingly, SDT proposes a taxonomy of types of regulation for extrinsic motivation that differ in the degree to which they represent autonomy (continuum of regulation from controlled to autonomous, from amotivated to intrinsically motivated). The approach of the SDT allows finer analyses for motivational processes than traditional conceptions do.

4. Research questions

The following research questions are expected to be answered by the investigation:

- Do students’ self-related cognitions and emotions remain stable or do they increase within the IMST project?

- Will students perceive a change in the quality of instruction at the end of the project year?

- If teachers perceive their working conditions as supportive and feel free from pressure, will they be highly motivated (self-determined) and will they create supportive learning environments for their students?

- If students’ basic needs in the classroom are satisfied, will they perceive themselves as self-determined?

- Do self-determined students show more interest, a higher content-related self-concept, and less fear?

The study seeks to make both a theoretical and practical contribution: First, we suppose to find further information about the relationship of the degree of self-determination between teachers and students. We expect additional facts about the deviation of the teachers’ and students’ mutual view concerning self- and externally described motivation (see Hanfstingl et al., 2006). Practically, we want to point out empirically the relevance of the consolidation to support students’ and teachers’ basic psychological needs.

5. References

Deci, E. L., & Ryan, R. M. (2002a). The paradox of achievement: The harder you push, the worse it gets. In J. Aronson (Ed.), Improving academic achievement: Impact of psychological factors on education. New York: Academic Press.

Deci, E. L., & Ryan, R. M. (2002b) (Eds.). Handbook on self-determination research. Rochester: University of Rochester Press.

Hanfstingl, B., Mueller, F. H., Benke, G., & Riemenschneider, I. G. (2006). Berufliche Selbstbestimmung von Lehrer/innen und dessen Auswirkung auf Unterrichtsgestaltung und Lernmotivation von Schueler/innen: Eine Analyse mit Strukturgleichungsmodellen auf der Basis von Partial Least Squares (PLS). Vortrag bei der 68. Tagung der Arbeitsgruppe Empirisch-Paedagogische Forschung (AEPF) der Deutschen Gesellschaft fuer Erziehungswissenschaft (DGfE), Sept. in Muenchen.

Krainer, K. (2005). Pupils, teachers and schools as mathematics learners. In C. Kynigos (Ed.), Mathematics Education as a field of Research in the Knowledge Society. Proceedings of the First GARME Conference (pp. 34-51). Athens, Greece: Hellenic Letters (Pubs).

Krainer, K. (in press). Die Programme IMST und SINUS: Reflexionen ueber Ansatz, Wirkungen und Weiterentwicklungen. In D. Hoettecke (Hrsg.), Naturwissenschaftliche Bildung im internationalen Vergleich. Gesellschaft fuer Didaktik der Chemie und Physik. Tagungsband der Jahrestagung 2006 in Bern. Muenster: LIT-Verlag.

Krainer, K., Doerfler, W., Jungwirth, H., Kuehnelt, H., Rauch, F., & Stern, T. (Eds.) (2002). Lernen im Aufbruch: Mathematik und Naturwissenschaften. Pilotprojekt IMST². Innsbruck: Studienverlag.

Pelletier, L., Seguin-Levesque, C., & Legault, L. (2002). Pressure from above and pressure from below as determinants of teachers' motivation and teaching behaviors. Journal of Educational Psychology, 94, 186-196.

Vallerand, R. J., Pelletier, L. G., Blais, M. R., Briere, N. M., Senecal, C., & Vallieres, E. F. (1992). The Academic Motivation scale. A measurement of intrinsic, extrinsic, and amotivation in education. Educational and Psychological Measurement, 52(4), 1003-1017.

In recent years, some endeavours have been made to disseminate research-based knowledge into the practical field of education. Innovation projects based on theories of situated learning use a so called “symbiotic approach” to point out the reciprocal dependency and profit of researchers and practitioners. The content of the innovation is constructed jointly by researchers and teachers. Thus, the teachers’ ability and experience are valued in the process of implementation, and it is of great importance that teachers consider themselves as responsible for the content and success of the project. In this paper we want to argue that the specific role teachers play in projects with an approach of symbiotic implementation lead to specific consequences in terms of design and methods to be used in these projects. Our findings stem from the project “Chemie im Kontext” (Chemistry in Context), a project dealing with a context- and inquiry-based curriculum development for lower and upper secondary schools. At the beginning of the project, we pursued a kind of traditional empirical research strategy: We used questionnaires and tests to assess the view of teachers and the effects of the lessons on student learning. A major finding of this first phase of the project was that this traditional research approach wasn’t appropriate to our symbiotic implementation strategy, as it didn’t fit to teacher’s role in the project. Thus, we changed our research methodology: We started a closer co-operation with teachers concerning the planning of research; we offered support and tools for teachers’ own research activities and we used different forms of interviews. These changes will be discussed with respect to the “appropriateness of research in implementation projects”.

In recent years, some endeavours have been made to overcome the gap between theoretical and practical knowledge in the field of education: In many countries projects have been conducted that aim at disseminating research-based knowledge into the practical field of education. Many of these projects are based on theories of constructivist or situated learning: This theoretical view of learning has been applied to the learning of teachers, yielding new forms of professional development as well as other forms of implementation.

            Innovation projects based on theories of situated learning use what is called a “symbiotic approach” to point out the reciprocal dependency and profit of researchers and practitioners. This implementation approach can be characterised by the following aspects: A) It is a co-operative and symbiotic process in which teachers and researchers work together for a significant duration in learning communities. B) The process can be characterised by constant development: Changes to, and revisions of, the innovation are not only allowed, but regarded as essential in adapting the innovation to different situations and conditions. Thus, the content of the innovation is constructed jointly by researchers and teachers. The teachers’ ability and experience are valued in the process of implementation, and it is of great importance that teachers consider themselves as responsible for the content and success of the project.

            As a consequence, teachers view themselves as an important and responsible part of the project and aim at influencing the process and the results of the project. In this paper we argue that the specific role teachers play in projects with an approach of symbiotic implementation leads to specific consequences in terms of design and methods to be used in these projects. Our findings stem from the project “Chemie im Kontext” (Chemistry in Context).

            “Chemie im Kontext” is a project dealing with curriculum development for lower and upper secondary schools (Parchmann, Graesel, Baer, Demuth & Ralle, 2006). The curriculum aims to foster the acquisition of applicable and transferable knowledge about chemistry and to raise the interest of students for chemical problems. Chemie im Kontext (ChiK) uses a conceptual framework based on three guiding principles: (A) The context-based teaching units are based on relevant and authentic topics and questions, which provide the guidelines for teaching. For the design of learning environments, different forms of contexts are used: everyday life situations, important social issues, and scientific and technical issues. (B) A variety of teaching and learning methods enable students to integrate their ideas and interests, and help them to construct their own knowledge. (C) Basic concepts as well as students´ abilities are developed through the investigation of relevant topics. The idea of ChiK is not to develop only one particular concept through one single context-based teaching unit, but rather to pick up the same basic concepts again and again in different units.

            ChiK offers a conceptual framework for teaching and learning chemistry that has to be adapted to specific constraints, e.g. the prescribed goals and contents of ministerial syllabi, laboratory equipment of schools, the interests of students, local partners, etc. The task of teachers is to use the framework and the “basic materials” to establish learning environments suited for teachers’ general conditions and their students. Hence, the implementation process integrates the framework, designed by science educators and teachers, and the different individual conditions and experiences of teachers and classes participating in the project.

            The paper gives an overview about methodology and methods of our research conducted in the project as well as about the changes of our research approach during the first three years. At the beginning of the project, we pursued a kind of research that is common in Educational Psychology: We used questionnaires and tests to assess the view of teachers and the effects of the lessons on student learning. Further, we asked the teachers to document their lessons with a documentation schema that was developed by researchers. A major finding of this first phase of the project was that the traditional research approach wasn’t appropriate to our symbiotic implementation strategy: Neither did teachers accept to fill out the questionnaires nor were they motivated to use the documentation schema. On the other hand, some of the teachers began to develop elaborated methods to assess the quality of their teaching and to document their lessons. Teachers were not so much interested in general findings for the whole project. Instead, they wanted to get a more precise insight how CHiK could help them to enhance their teaching within the given constraints in their specific school. Thus, we changed our research methodology: First, we started a closer co-operation with teachers concerning the planning of research; furthermore we offered support and tools for teachers’ own research activities. Secondly, we used interviews and telephone interviews instead of questionnaires to obtain relevant information about the success of the implementation of CHiK. These methods as well as the closeness to action research were more successful. These changes will be discussed with respect to the “appropriateness of research in implementation projects”: The symbiotic approach urged us to use a kind of research that resembles to the concept of action research which has been conducted in a wide array of projects in Educational Sciences in the 70ties. In this view, teachers (and their students) are not restricted to the delivery of data that helps researchers to develop empirical results.

References

Graesel, C., Fussangel, K. & Parchmann, I. (2006). Lerngemeinschaften in der Lehrerfortbildung: Kooperationserfahrungen und -ueberzeugungen von Lehrkraeften. [Professional development of teachers in learning communities: Teachers’ experiences and beliefs concerning collaboration.]Zeitschrift fuer Erziehungswissenschaft.

Parchmann, I., Graesel, C., Baer, A., Demuth, R. & Ralle, B. (2006). Chemie im Kontext - a symbiotic implementation of a context-based teaching and learning approach. International Journal of Science Education, 22, 1041-1062.

In this paper we refer to the use of the portfolio-method to support and evaluate teacher professional development. Our study draws on data from the German program “Increasing the Efficiency of Mathematics and Science instruction” (SINUS-Transfer). This program’s approach requires teachers to improve their teaching in a cooperative manner and with regard to didactical problem areas in classrooms. In order to support teachers working in the program we developed a customized portfolio concept. The so-called “subject department portfolio” requires teachers of one school to collaboratively maintain a portfolio. Colleagues keep their shared goals, further steps to engage in as well as examples of classroom material including comments and reflections in this portfolio. The instrument allows teachers to document and reflect on efforts to improve their teaching and to make their thoughts and developments accessible to others. Additionally, the portfolio has the function to yield important information on the success of the program. In this respect the subject department portfolio serves as a tool for evaluation on a program level.

A central prerequisite for a successful use of the portfolio with regard to those aspects, is seen in the teachers’ acceptance of the instrument. Therefore the main question we investigate in this paper is how teachers assess the portfolio.

Teacher as well as school principal questionaire surveys are conducted regularly within the evaluation of the program. There teachers were also asked to rate the portfolio instrument. Results of latent-class analyses show the acceptance of this tool differs widely between different groups of teachers. The study yields indicators on requirements that have to be met in order to effectively use the portfolio as a tool for evaluation. Findings will be discussed in the light of alternative forms of professional development as well as adequate ways of evaluating those approaches.

Theoretical background and aim of the study

The portfolio-method is more and more discussed as a tool to support teacher professional development (Craig, 2003; Orland-Barak, 2005; Tucker et al., 2003). It’s main function is seen in assisting the reflection and evaluation of classroom teaching. In this paper we look at the instrument portfolio not only in its role to support teacher learning but also in its function to serve as a tool for evaluating teacher professional development initiatives. We thereby refer to experiences from the German program “Increasing the Efficiency of Mathematics and Science instruction” (SINUS-Transfer) (Prenzel & Ostermeier, 2006). In the second of two phases of scaling-up, over 1.700 German secondary schools are currently taking part in this program. The program’s approach requires teachers to improve their teaching in a cooperative manner and with regard to didactical problem areas in science and mathematics teaching. Professional development takes place within the school (at the level of the subject department) as well as between schools. Teachers reflect critically on their teaching and on their students’ learning processes. They discover didactical problems, agree upon shared goals to work on and test possible solutions within the classroom.

To support teachers within this approach of collaborative and school based professional development the program makes use of the portfolio-method. We developed an instrument customized to the program’s aims that requires mathematics and/or science teachers of one school to collaboratively maintain a portfolio (Meentzen et al., 2006). In this so-called subject department portfolio colleagues keep their shared goals, further steps to engage in as well as examples of classroom material including comments and reflections. The portfolio allows for teachers to document and reflect on efforts to improve their teaching and to make their thoughts and developments accessible to others. Besides the idea to support the teachers’ learning processes the portfolio has the function to yield important data on the overall progress and success of the professional development program. In this respect the subject department portfolio serves as a tool for evaluation on a program level.

Research questions

Although there are several advantages to use the instrument in this way, some important questions arise in doing so: The instrument is new to most of the teachers and it is a rather fragile tool compared to commonly used methodes in educational research like for instance questionaires. To use the portfolio as an evaluation tool certain requirements have to be met: Teachers have to accept the instrument, understand it and employ it. The central question we will deal with in this paper is how teachers accept the instrument portfolio within the program: Particularly we are interested whether the portfolio is being valued as a helpful instrument in order to support collaborative quality development within the program.

Design and methods

Within the evaluation of the program teacher and school principal questionaire surveys are being conducted on a regular basis. The goal of those surveys is to obtain indicators on how the teachers participating in the program evaluate the professional development approach itself as well as the support structures that are supplied by the program coordinators. The teacher questionaire also contained questions concerning the value of the portfolio instrument. Particularly teachers were asked to assess whether the portfolio is adequate to support their work and to what extend it is suitable to foster processes of self-evaluation. Besides teachers could rate to what extend they appreciated specific support structures (e.g. instruction material, forms to help structure the work with the portfolio, etc.) supplied to help implement the instrument at their school. Our paper refers to a sample of secondary teachers (N=934) from the first phase of scaling-up within the program in the year of 2005.

Results

The results of latent-class analyses show four different groups of teachers with specific response patterns concerning their acceptance of the portfolio method. Those patterns range from a clear acceptance of the tool to an obvious rejection of the instrument. Analyses of Variance showed that those groups of teachers who were in favor of the portfolio method also had more positive ratings in other program related variables (e.g. support through local coordinators of school networks, collaboration with colleagues at school level, etc.).

Educational significance of our study

The findings indicate that teachers participating in the professional development program accept the tool portfolio in different ways. More than a third of the participating teachers seem to look rather critically on the instrument. Although having specific response patterns the other groups of teachers (65%) rate the portfolio much more positively. Those positive ratings go along with high ratings concerning their contentment with the program’s support structure and their collaborative work within the program. Altogehter, the findings point out that a crucial aspect seems to be the local support of teachers in implementing the instrument.

Our study yields indicators on requirements that have to be met in order to effectively use the portfolio as a tool for evaluation. The paper will discuss the findings in the light of alternative forms of professional development as well as adequate ways of evaluating those approaches.

References

Craig, C. J. (2003). What teachers come to know through school portfolio development. Teaching And Teacher Education, 19(8), 815-827.

Meentzen, U., Ostermeier, C., & Prenzel, M. (2006, 09.04.). Use of subject department portfolios in a professional development program to support teacher cooperation. Paper presented at the Poster presented at the Annual Meeting of the American Educational Research Association (AERA), San Francisco.

Orland-Barak, L. (2005). Portfolios as evidence of reflective practice: what remains 'untold'. Educational Research, 47(1), 25-44.

Prenzel, M., & Ostermeier, C. (2006). Improving mathematics and science instruction: A program for the professional development of teachers. In F. K. Oser, F. Achtenhagen & U. Renold (Eds.), Competence oriented teacher training. Old research demands and new pathways (pp. 79-96). Rotterdam: Sense Publishers.

Tucker, P. D., Stronge, J. H., Gareis, C. R., & Beers, C. S. (2003). The efficacy of portfolios for teacher evaluation and professional development: do they make a difference? Educational Administration Quarterly, 39(5), 572-602.