Becoming Accountable for the Impact of Graduates
on Students and Schools:
Making Operational the Shift From Teaching to Learning

A Concept Paper for Discussion

prepared by:
Roger Pankratz

Western Kentucky University

for a symposium
at the Annual Meeting of
The American Association of Colleges
for Teacher Education

Washington, DC
February 24-27, 1999

 

Becoming Accountable for the Impact of Graduates
 on Students and Schools:
Making Operational the Shift from Teaching to Learning

The Challenge to Focus on Teacher Performance and Student Learning  

Not long ago, I watched a group of faculty present a plan for a school-university partnership program to their provost prior to submitting the proposal to a foundation for support.  The designers of the plan described the rich field experiences for teacher candidates, the faculty mentoring program, electronic portfolios, increased interactions between practioners and professors, and the potential for professional publications.  After reviewing the plan, the provost looked up from the proposal document and said, “This looks very interesting, but what does the new partnership program have to do with student learning?  Suppose three years from now when the program is fully operational and a reporter from CNN or ABC comes to see what you have achieved.  You show her the many fine features of your program and then she turns to you and says, ‘What evidence do you have that all children are learning at high levels?’  What will you tell her?’ ” A response from one faculty member was that it should be obvious from all the learning activities in which students are engaged.

While student involvement and activity may have been a recognized indicator of learning a decade ago, it falls far short of acceptability in Kentucky today with nine years of statewide performance assessment and school accountability.  A recent report from the Council of Chief State School Officers indicated that 48 states have adopted some type of statewide testing program.  Thirty-six states issue a “report card” to parents and the public on the performance of schools; and in 16 states, schools whose students perform poorly are taken over.

The provost of every teacher preparation institution in Kentucky is interested in student learning because they are being asked to provide evidence that their graduates are prepared to facilitate learning at high levels for all students, not just those who are the most academically able.  The business community, citizen groups, and policy makers in Kentucky want to know why large numbers of students and schools are not keeping pace with the high standards set by the state and the extent to which teachers are being prepared for high performing schools.  Also, provosts of teacher preparation institutions in Kentucky are interested in student learning because school administrators want to know that the new teachers they hire will be able to contribute positively to the performance of the students they teach. Like athletic coaches, school principals want evidence that new teachers they employ are team players who have a track record of high performance and can facilitate learning in all students.

Another reason provosts in teacher preparation institutions in Kentucky and across the nation are interested in teacher performance related to student learning is because new standards have been adopted for NCATE 2000.  Under these new standards, the following guidelines apply to all institutions planning continuing accreditation.

“Institutions will use available measures that show candidates’ performance in relation to the standards (e.g., state licensure tests, essays, or projects in subject matter mastery) and will develop measures in other areas (e.g., demonstrating teaching proficiency during the clinical education experience and showing ability to have a positive effect on student learning).  This information will be summarized for each program with examples of assessment tasks, scoring rubrics, and candidates’ work.”1

Those responsible for teacher preparation programs know that the documentation of inputs and processes that have been the foundation for unit and program accreditation in the past will be insufficient to meet the new performance standards, and evidence of teacher performance related to student learning will be required for continuing accreditation.

In November of 1997, representatives of the Renaissance Group2 institutions considered alternative program initiatives that would most likely encourage member institutions to examine their programs and develop strategies to demonstrate accountability for the performance of their graduates.  It was agreed that a focus on teacher performance that positively impacts student learning would best promote that system of accountability.  As a result, a number of member institutions have agreed to examine and implement programs to collect and publish data on the impact of their graduates on students and schools and to strengthen their program to ensure higher teacher performance.  It is this quest for evidence of impact of graduates on students and schools that is the focus of this discussion paper.

The Paradigm Shift From Teaching to Learning

Need for a Change in Focus

According to Barr and Tagg3, the fundamental problem with the focus on instruction rather than learning is “that our dominant paradigm mistakes a means for an end.”  To hold the belief that the purpose of colleges or schools is to provide instruction is like saying that General Motors’ primary business is to operate assembly lines.  The primary problem resulting from focusing on instruction rather than learning is that it is incompatible with standards-based teaching and learning.  Accepting accountability for the learning of all students requires attending to a complex set of processes and conditions associated with teaching and learning, not just instruction.  Focusing mainly on instruction works only for the spectrum of learners that can most profit from the instruction that happens to be offered but may have less than the desired impact on the rest of the intended learners.  A learning paradigm accepts responsibility for positively impacting all students.

A second reason why the instruction paradigm is inadequate for our teacher preparation programs or our P-12 schools is that with this focus, it is impossible to increase outcomes without increasing resources.  More instruction requires more resources.  More learning invites the reconfiguration of resources.  Over the past decade, business and industry have faced the reality of becoming more efficient or losing out to the competition.  Teacher education and P-12 education are facing the same dilemma.  Only if we approach educating our future populations from a learning perspective will we be able to become efficient and afford our national dreams of a better life for all.

A third reason that continuing to focus on instruction rather than learning fails to support standards-based teaching and learning is that primary attention is given to only one aspect of the teaching/learning process, “the means.”  On the other hand, a focus on learning, or the product, encourages both the teacher and the learner to try out and adjust various aspects of the learning process to achieve the best results.  Thus, the learning paradigm is much more concerned with the whole of a learning system than only one aspect, instruction.  It forces us to look at all the factors that affect learning, including the student’s abilities, background, and prior knowledge; the context of schooling; the complexity of the learning task; and the learning resources available.

Fundamental Differences in Two Approaches

Barr and Tagg4  have described the key differences in the instruction vs. learning paradigm for undergraduate education in college.  These differences are described under the categories of “mission and purposes,” “critical for success,” and “teaching and learning structures.”  Schalock and Myton5 described many of these same differences when they presented contrasting factors that characterize traditional schooling and “standards-based teaching and learning.”  They report fundamental shifts in how schools are organized, the nature of instruction, and in assessment, record keeping, and reporting.  Common to both of these descriptions are the following differences in the two approaches to teaching and learning.

 

Defining Attribute	Focus of Instruction	Focus on Learning
•                     mission or purpose	offer instruction	produce learning
•                     expectations	different for different students	high standards for all
•                     progress in the system	grade to grade accumulation of course completions	performance related to defined standards
•                     nature of the learning environment	the class	varies based on the learner, learning target, and learning context
•                     nature of instruction	same for all	varies based on the learner, learning target, and learning context
•                     time for teaching and learning	same for all	flexible
•                     nature of assessment	-local -single form -teacher’s objectives	-public -multiple forms -state or national standards
•                     program evaluation	end of course evaluation	comprehensive assessment of all components of the system

Implications for Teacher Preparation Programs       

Based on the contrast of the two approaches described above relative to key attributes, it is obvious that fundamental differences in beliefs, structures, processes, and behaviors on the part of university and school leaders and practitioners will be required to make the proposed shift to a focus on learning.  However, some beliefs and processes, because of their importance and difficulty to change, will become the greatest challenge for leaders of reform in teacher preparation programs.  Six of these challenges are presented below.

Accepting Accountability for Student Performance.  The focus on instruction has often absolved the teacher of responsibility for student learning.  Public school teachers and university faculty are quick to blame low performance on students, their parents, and colleagues who are former teachers of their students.  Accepting the responsibility to positively impact learning empowers the teacher to alter and improve the learning environment for the learner.  However, this stance is often very threatening to teachers who believe that most factors related to low performance of students are beyond their control.

Focusing on Learning Standards.  College professors traditionally have experienced a lot of autonomy and academic freedom about what they taught.  The concept of state or nationally defined standards and being told what is the most important for students to learn is often difficult for college professors and teachers to accept.  Both have enjoyed the freedom to teach their favorite topics using their favorite strategies without regard for the needs of students.  College professors and school practitioners need to be shown that defining content standards does not take away their freedom to design instruction.

Placing More Emphasis on Assessment of Learning Progress.    Standards-based teaching and learning requires frequent assessment, diagnosis, and communication of learning progress.  Also, teachers need multiple tools of formal and informal assessments to assure continuous progress of all children.  Interviews with more than 20,000 new teachers in Kentucky show that more than four out of ten beginning teachers say they were poorly prepared for standards-based alternative assessments6.  Also, a review of teacher education programs in ten Renaissance Group institutions showed a lack of emphasis on assessment of student learning in teacher and administrator preparation programs7.

Engaging Students and Teachers in In-depth Learning Tasks.  The practice of covering a broad set of topics in a superficial manner is a common practice in P-12 education.  Focusing on a few key concepts in depth is the exception rather than the rule in America, as shown by the Third International Mathematics and Science Study8.  In Kentucky, less than two percent of middle school students are “proficient” in science that requires students to write open response questions about relationships and science inquiry processes.  Getting university faculty to model the learning environments we need in P-12 schools is highly essential if we expect standards-based teaching and learning at high levels for all students.  This challenge will be especially difficult in university environments where academic lectures have been the established way of life.

Collaborating with Colleagues, Parents, and Community Resources.  Standards-based learning does not assume that all or even most learning takes place in the classroom or under the direct instruction of the teacher.  Believing that this can be the case and developing meaningful relationships with parents and other school and community resources is foreign to many school practitioners and most college faculty.  Yet, these collaborative relationships must be developed and new teachers trained in how to utilize a much wider repertoire of learning resources.

Reflecting on and Learning From Experience.  Teachers who have gone through the National Board Certification process often talk about it as a life changing experience because they are required to analyze and evaluate their teaching.  This gives them a life-long tool to improve on their teaching performance.  Standards-based teaching and learning provide regular feedback to the student on their performance and feedback to preparation programs on their effectiveness.  National Board Certified teachers report their experience was unlike anything they ever experienced in their preparation or professional development programs.  In standards-based programs, reflection and evaluation must become an integral part of professional life both on campus and in schools.

Strategies for Measuring and Collecting Information

on Teachers’ Positive Effects on Student Learning

Standards-based teaching and learning, as in instruction-based systems, requires teacher candidates to show high performance levels in the knowledge of the content they teach and pedagogy.  However, a standards-based program also requires evidence that graduates can positively impact student learning.  The Program Standards for Elementary Teacher Preparation document recently developed by NCATE describes four major attributes for performance-based teacher preparation9.

• Teacher candidate knowledge

• Teaching performances (methodology)

• Teacher dispositions (values and commitments)

• Positive effects on student learning

It is the fourth attribute that has been omitted from instruction-based systems and is of major concern to the Renaissance Group institutions.  Assessment coordinators from member Renaissance Group institutions have formed a task force to develop and test strategies for determining the extent to which their graduates are having positive effects on the K-12 students they teach.  Renaissance Group institutions agree that all members must adopt multiple measures for assessing teacher candidate performance that include all four attributes described in the NCATE Program  Standards.  However, they realize that the fourth attribute, effects on student learning, has not been addressed in the past and should be a developmental priority.

Assessment coordinators at several member institutions have conducted focus groups with their teacher graduates and with teacher education faculty regarding ideas and strategies for determining the impact of graduates on students and schools.  Also, the assessment coordinators have explored various practices under development and being implemented across the nation that attempt to measure teacher impact on students.  A repertoire of promising and potential strategies for assessing student impact has been assembled for the consideration of member institutions as they each develop their own performance evaluation program of graduates.

The strategies and/or ideas collected that are related to assessing teacher impact on student learning have been organized into five categories for further discussion and consideration by the Renaissance Group assessment coordinators: surveys, student work samples, teacher work samples, student achievement data, and teacher performance links to achievement data.  Each of these five categories is described below.

Surveys About Teacher Impact

A review of surveys that have been conducted by member Renaissance Group institutions confirms that most surveys are mail-out instruments to graduates with very general questions about the graduate’s satisfaction with their preparation program.  The return rate is usually low (less than 30%) and overall satisfaction with programs quite high.

In 1996 and 1997, the Kentucky Institute for Education Research conducted telephone random sampled interviews with 2200 first, second, and third year teachers in Kentucky about their initial teacher preparation.  With some variation among responses of graduates from different institutions, six of ten said they were extremely well prepared, and less than one in ten said they were poorly prepared.  However, when new teachers were asked more specifically about their preparation for specific performance areas such as design of instruction, use of technology for instruction, or strategies for assessing student learning, they reported being less prepared and showed more variation among institutions.  When asked about preparation for specific program initiatives mandated in all schools by Kentucky’s Education Reform Act of 1990, such as the use of alternative assessments, less than half said they were well prepared10.

The lessons to be learned from the two-year statewide surveys in Kentucky were that the more specific and focused questions yielded more discriminating and useful information.  Also, random stratified telephone interviews yielded quite different information than mail-out surveys with low return rates.

While surveys are not the most desirable method to obtain information about teachers’ impact on students, they are very likely to continue to be used by teacher preparation institutions due to the lower cost and ease of conducting surveys.  Therefore, the Renaissance Group coordinators are including surveys and interviews about teachers’ impact on students as one possible source of data.  However, consideration should be given to multiple sources of data beyond the self report of the graduate to include perceptions of school administrators, other teachers, parents, and the students of the graduate.  Also, questions about impact on specific learning targets and random sampling of respondents should be considered.

Student Work Samples and Feedback on Student Work

Teacher exhibits required for National Board and for INTASC assessments require progressive examples of student work over time and the feedback provided to the students to help them improve their performances.  Portfolios in most student teaching and teacher intern programs require exhibits of student work.  Student work samples and teacher feedback exhibits can be useful sources of information about a candidate’s ability to facilitate student learning.  They provide evidence of how teachers are able to support learning for individual students, but not the whole class.  National Board and INTASC assessments direct the teacher to select different types of students for their exhibits; however, the choice is up to the teacher.  Thus, students who may have the most difficult time learning may not be selected for student work samples.  For student work samples to provide meaningful information across teacher candidates, well designed common scoring guides and instructions for collecting student work are essential.

Teacher Work Sampling11                 

Teacher work sampling, as the name implies, focuses on a sample of teacher work but also collects samples of pupil work.  According to its authors and developers, “Work sampling resembles what teacher educators typically require of student teachers as they assume full responsibility in the classroom.  It represents a strategy that directly connects teacher performance in a meaningful way.”  The process for this methodology was developed over the past decade at the University of Western Oregon and has been adopted by the Oregon Teacher Standards and Practices Commission as the primary means for teacher candidates to provide evidence that they have met Oregon’s teacher performance standards for licensure12.

While there are some variations across institutions, Western Oregon University’s program requires teacher candidates to complete ten teaching tasks, each having a required product that has a performance measure.  The ten teaching tasks include:

1. Describing a unit of study

2. Mapping the classroom context

3. Identifying learning outcomes

4. Developing assessments for outcomes

5. Administering pre-instruction assessments

6. Developing a design for instruction and assessment for all pupils

7.  Implementing the instructional plan

8. Administering the post-instructional assessment

9. Summarizing, interpreting, and reporting the growth of each pupil and selected groups of pupils in the class

10. Reflecting and evaluating the teaching and learning process for the instructional unit

At Western Oregon, a foundation for the knowledge, skills, and processes required for the ten tasks is provided in the early years of the program.  The set of ten tasks is completed for two separate units of two to five weeks of instruction during their student teaching experience.  The work sample methodology provides direct evidence of a teacher candidate’s effect on student learning in a relatively short time period and clearly connects the elements of standards-based teaching and learning.

Critics of the work sample methodology question what teacher candidates can be expected to accomplish with students in two to five weeks at the student teaching level.  The developers of the Oregon Teacher Work Sample Model have worked hard over the past five years to respond to their critics and strengthen the weaker components of their program.  The authors do not advocate that the data from the teacher work samples should be the only indicators of teacher performance.  However, the Oregon program represents the nation’s most fully developed model for providing evidence of a teacher candidate’s effects on student learning.

The Oregon Teacher Work Sampling Methodology is being studied by NCATE, AACTE, and the Renaissance Group as a model that holds promise for providing evidence of teacher impact on student learning. 

Collecting and Attributing Standardized Student

Test Scores to the Effects of Teacher Graduates on Learning

In the summer of 1998, The Teachers College at the University of Nebraska-Lincoln13, in honor if its 90th anniversary, made a pact with the state of Nebraska that it would implement a plan to be accountable for how well their graduates teach and how well their teachers’ students learn.  While the Teachers College announced it would collect data from multiple sources and at different points in their teacher graduates’ preparation programs and beginning years of teaching, in the second year of their teaching career, Teachers College graduates will be graded by their schools on the success of their K-12 students.  Data for the evaluation of teacher effectiveness will come from standardized tests already in place in those schools.

Although the detailed processes for attributing student test scores to the ability of specific graduates to produce learning have not been worked out, the Teachers College pact represents a significant “first” institutional initiative that accepts responsibility for student learning in public schools.

Because 48 states now have some type of required testing programs in schools, other teacher preparation institutions will attempt to follow the University of Nebraska’s lead to collect achievement data on the students of their graduates.  This strategy has its challenges in attributing a student’s scores to a specific teacher when it is not a one-teacher, self-contained classroom.  Also, since not all grades or disciplines are tested, some type of sampling design will need to be considered.

Linking Indicators of Teacher Performance

To Student Achievement Results

In states that have a history of statewide testing of K-12 students and have adopted some type of teacher evaluation program, educators or policy makers may want to investigate the relationship between indicators of teacher performance measured by teacher assessment instruments and student achievement scores.  Establishing relationships between teacher performance indicators and student achievement scores provides an indirect method of showing that teacher graduates are having a positive impact on student learning.  For example, in the state of Connecticut, the Connecticut Mastery Test14 has been administered to students in grades 4, 6, and 8 since 1986 to assess reading, writing, and math skills statewide.  In addition, this state has developed the Connecticut Competency Instrument to teachers in their first year of teaching and a formal, portfolio-based assessment process during year two for a newly appointed teacher.  The Connecticut Department of Education has already established relationships between teacher participation in the state’s Beginning Educators Support and Training (BEST) Program15 and student achievement measured by the Connecticut Mastery Test16.  Now, the Department of Education is looking to link teacher scores on their Teacher Competency Instrument and the portfolio-based assessment process with student achievement.  Again, the challenges of this strategy to linking teacher performance with student achievement includes the limited testing of 4th, 6th, and 8th grades and only reading, writing, and math skills.

In Alabama, the State Board of Education has established policies and procedures requiring the development of a professional education personnel evaluation system.  School systems may develop their own or use a system developed by the state which, through observation and documentation, evaluates teachers in five performance areas17.  The state of Alabama also has a statewide testing program for all public schools.  Researchers at the University of Alabama, Birmingham, are investigating the relationship between teacher performance factors on the state’s personnel evaluation instrument and student achievement as one method of linking teacher performance to student learning in Alabama.

A major challenge with this strategy for collecting data on teacher impact on student learning is how to interpret results when some performance factors show positive relationships and other factors show no relationship, or even negative relationships, to student achievement.  Also, it is highly likely that teacher performance factors may be related to student learning differently in one subject area compared to another.

The next section of this paper presents some of the general challenges that must be addressed with all strategies that may be considered for gathering data on teacher impact on student learning.

Issues and Challenges to Programs that Attempt to Link Teacher

Performance with Student Learning

Every non-traditional program has its set of critics and challenges to its validity, and performance-based teaching and learning is no exception.  The attempt to link teacher performance to student learning is not new. However, claiming that it can be done or might be done in high-stakes settings or on a statewide scale was unheard of a decade ago.  Any practitioner or research group that plans to have its work accepted by the professional community needs to consider the issues and challenges being directed at performance-based teaching and learning friends and foe.

Following are six issues and/or challenges that have been debated by critics and supporters of performance-based teaching and learning.  The six questions are raised to inform and promote further discussion on each issue.

Issue 1.  What are reasonable expectations for student teachers, first-year interns, or new teacher graduates with respect to impact on student learning?

The state of Oregon requires that teacher candidates show they can produce student learning before they can be licensed to teach.  Several other states (e.g., Connecticut, Alabama, Washington, Nebraska) are attempting to link new teacher performance with student achievement.

A number of researchers and teacher educators have expressed skepticism that beginning teachers cannot be expected to have the skills and experience to produce real measurable gains in student achievement.  How convincing is the data to support or refute the claims on both sides of this issue?  

Issue 2.  How much time is needed for teachers to show real progress in student achievement in a high stakes system?

In the Oregon system, through pre-post assessments, student teachers are expected to show positive gains in learning within a five-week period.  On the other hand, teachers in Kentucky are given two years to show significant gains on statewide test scores.  The University of Nebraska at Lincoln is planning to look at data after the second year of teaching.  What can be expected in a given time period?  Are the conditions different for short-term progress than for long-term gains?  Can short-term gains be a valid measure of teacher performance?

Issue 3.  To what extent do student teachers and even beginning teachers have enough control of contextual factors in the school to affect student achievement?

There has always been skepticism among the professional community that while teachers facilitate learning, many factors that affect student learning are beyond the teacher’s control; and individual teachers cannot be held directly accountable for student gains.  In Kentucky, the  schools, not individual teachers, are held accountable for student learning.  Others in the professional community have strong beliefs that teachers, like athletic coaches, as professionals should be able to work with contextual factors and any students to show improvement of performance.  The concept of no exceptions and no excuses had gained support especially with some policy makers and members of the business community.  The Oregon program expects student teachers who use another teacher’s students and classroom to show evidence of learning with students assigned to them.  Is it fair to expect student growth in all classrooms?  Under what conditions can student teachers and beginning teachers be expected to be accountable for student learning?

Issue 4.  Under what conditions can student achievement be attributed to the performance of a student teacher or beginning teacher?

One teacher is usually not the only contributor to student learning.  In team teaching situations, this is especially true.  Students in middle and high school are influenced by as many as five or six teachers each day.  Some students receive more help from their parents and tutors than others.  How does one attribute a given teacher’s influence on the learning of students in situations that are not one-teacher, self -contained classrooms?  How valid are the strategies being advanced that compute the contribution of several different teachers to a student’s academic growth?18

Issue 5.  How does one set standards of teacher performance related to student learning under conditions where different schools have different standards and use different tests?

In the Oregon Work Sample Methodology, guidelines have been developed over the years to insure that student teachers use teaching outcomes and assessment processes that meet prescribed standards and are somewhat equivalent.  In many states, there are common standards in the basic discipline areas and statewide tests that provide some common measure of student achievement.  However, if institutions across the nation were to attempt to assess the impact of their graduates on student learning, they would find their graduates in schools with quite different standards and different testing systems.  Given different standards and different assessments, what strategies and techniques are available or being used to gather data and make judgments about the impact of graduates on student learning?

Issue 6.  How much of any teacher’s performance related to student achievement can be attributed to the teacher’s preparation program?

The University of Nebraska is planning to collect achievement data on second-year graduates.  In Kentucky, consideration is being given to look at student data of graduates over three years.  But even if institutions can show that their graduates are having a positive impact on student learning, how much of a teacher’s success can be attributed to their preparation program; how much can be attributed to innate academic and professional abilities, and how much can be attributed to professional development after graduation?  Can teacher preparation programs of the future be judged by the performance of students of graduates?

Benefits of a Standards-Based Approach to Teaching

and Learning for Teacher Preparation Programs

Given the above issues and challenges to linking teacher performance to student learning, one might ask, “What are the benefits of this approach to teacher preparation programs with all of the obstacles to overcome?”  Three major values to the focus on student learning are presented below.

A Standards-Based Model provides a framework to focus on the knowledge, skills, and performances that connect teaching and learning.

With the instruction-based model, teacher education programs have focused mostly on teaching strategies and have short changed assessment of student learning and meaningful feedback to students.  Also, not enough attention has been given to the alignment of learning targets, curriculum, and assessment and the teacher performances needed to make the connection across the complex set of teaching processes for this alignment.  The focus on learning forces the teacher candidate and the teacher educator to look more broadly at all aspects of the teaching/learning process.

A Standards-Based Model provides a framework for improving student learning, teacher performance, and teacher preparation programs.

Placing the learning of all students at high levels as the central focus requires practitioners and teacher educators to examine and evaluate all factors and processes that contribute to student learning.  This broader view is more likely to provide teachers with clues to how student learning can be improved and teacher educators with how the performance of teacher candidates can be improved.  Teachers who have experienced the National Boards and the INTASC assessment process almost without exception report that the greatest benefit to them is the requirement to reflect and evaluate their own teaching processes.  This is a powerful tool for continuous improvement.

A Standards-Based approach provides an accountability model that can be attractive to both educators and the public.

Making P-12 student learning a central focus in teacher preparation programs is not only educationally sound, but it has the potential to gain back the support of schools and the public for teacher preparation programs that has been eroded in the 90's decade of school reform.  In the October 1998 AACTE Briefs, David Imig described the “disconnect” between schools, colleges, and departments of education and the real-world of schooling as viewed by practitioners, policy makers, and the public.  Serious attention to a standards-based approach to teaching and learning provides an excellent opportunity for institutions that prepare teachers to convince school practitioners and the public that we all have one common objective--to facilitate the learning of all students at high levels of performance.