The UPRM TPP has selected the evaluation of completer preparation impact on P-12 student learning and development growth as its target area for improvement. Since 2006, the UPRM TPP has systematically collected information about the quality of its teacher candidates, but has not conducted studies in program completer classrooms to measure their effectiveness as teachers. The UPRM TPP needs to acquire valid, reliable data in order to optimize its preparation of future teachers in terms of enabling them to foster P-12 student learning and growth. The National Research Council (2010, Evidence 4.1.3) collected evidence about the quality of teacher preparation, recognizing that although teacher education is necessary for ensuring the quality of the teaching force, it is neither the only condition nor a sufficient one. With this in mind, the Council acknowledged many gaps still exist in the research regarding teacher education and its effect on student learning.
The objectives of the project for self-evaluation and optimization of the UPRM TPP are to:
1) measure alumni effectiveness in their classrooms;
2) evaluate and adapt alternative methods for collecting information regarding alumni impact and program effectiveness;
3) review information collected about recent alumni teachers;
4) use information gathered to evaluate strengths and weaknesses of the program in terms of the effectiveness of its alumni teachers;
5) use collected data to identify gaps between the profile of the teacher candidate and subsequent alumni teacher classroom performance;
6) refine and improve systematic data collection and analysis.
To attain these objectives, the Mixed Method Research will address seven principal research questions that will shed light on the impact of teacher preparation on P-12 student learning and development growth as the area for improvement:
1. What is the impact of UPRM TPP preparation on alumni’s P-12 students learning and development?
2. How does UPRM TPP alumni classroom instruction align with professional standards such as InTASC and PRDE Teacher Professional Standards?
3. How satisfied are UPRM TPP alumni with the relevance and effectiveness of their preparation?
4. How satisfied are the employers with the relevance and effectiveness UPRM TPP alumni and their preparation?
5. How satisfied are K-12 students with the classroom instruction they receive from UPRM alumni teachers?
6. What impact are UPRM alumni teachers having on the academic development of their P-12 students?
7. Do P-12 students perceive their UPRM alumni teachers as fair, ethical, challenging, and helpful?
The Mixed Methods Research Plan will use qualitative and quantitative methodologies to address the research questions about the quality of UPRM TPP alumni performance in terms of national InTASC and state PRDE professional standards. Data will be collected from key school stakeholders that comprise the UPRM alumni teaching environment: the alumnus, the alumnus’ students, and the alumnus’ immediate supervisor.
The qualitative case study methodology part of the plan will use multiple means to collect information from and about 8 to 10 alumni teachers as a way to explore process tracing that links causes and outcomes while allowing data triangulation. The qualitative data will provide opportunities to formulate hypothesis and generate new questions about the classroom effectiveness of alumni with respect to applying the skills, knowledge, and dispositions that the UPRM TPP programs sought to provide.
The Danielson Framework Teacher Observation Rubric (4.2.1) and the Alumni and Employer Interview protocols will be piloted with participants as part of the case study. These instruments and protocols were selected in part for their validity and reliability as demonstrated in teacher evaluation projects such as the MET study and will be adapted as needed for future research. The Flowers and Hancock’s Alumni Interview Protocol will be adapted for employer interviews.
UPRM TPP faculty members will be trained to conduct interviews and classroom observations to assure inter-rater reliability and to be as free as possible from bias. The classroom observations will be correlated with alumni self-reports and measures of student achievement versus learning goals.
As a means to complement the information gathered through case studies, two TPP created surveys (alumni and employer) will be used both with the case study population and a larger population. The alumni and employer surveys were created using pertinent information collected from previous surveys, but redesigning them to align more closely with InTASC (2013) and PRDE professional standards, as well as CAEP standards.
Validated item versions of the elementary and secondary Tripod Student Surveys will be used to gather data from alumni students, depending on the grade level taught by the alumnus. At least two different groups of each alumni teacher’s students will be asked to complete a Tripod Student Survey. The validity and reliability of the Tripod Student Surveys are documented in Asking Students about Teaching: Student Perception Surveys and Their Implementation, MET Project Policy and Practice Brief by the Bill & Melinda Gates Foundation, 2012*.
The Plan will include four schools that have at least two alumni teachers each. The schools selected will vary in the following aspects 1) zone: urban, rural; 2) type: public, private; and 3) level: elementary school, middle school, high school. The alumni teachers in the study will represent the majority of the UPRM licensure areas: Agricultural Education, Art, Biology, General Science, Business Education, Physical Education, Spanish, Social Studies, Physics, History, English, Mathematics, and Chemistry. A range of 8 to 10 teachers will be the target.
At least two different groups of each alumni teacher’s students will be included in the study as well as their school directors to produce a more comprehensive view of the alumni school ecosystem and to triangulate the data.
The TPP created alumni survey will be e-mailed to all UPRM TPP alumni completing teaching practice over the four years prior to the last (2012-2015).
The first round of employer surveys will be e-mailed to school directors at UPRM TPP centers. A second round of employer surveys will be e-mailed to directors at schools identified in responses from working alumni. The surveys will be administered with the Qualtrics© online survey system.
*See information of the MET project in http://www.metproject.org/downloads/Asking_Students_Practitioner_Brief.pdf.
See page 4 of SI uploaded plan attached – Table 1. Assessment Plan – Goal alignment to assessment instrument with CAEP Standard 4 element by study participant.
Information obtained from direct observations, teaching-learning artifacts and teacher self-reports as well as data from alumni students and employers will be compared, correlated, and triangulated to form a comprehensive picture of the teachers’ effectiveness with respect to student learning. The proposed analysis will include comparisons among alumni in diverse licensure areas as well as by school type. With the proposed mixed methods data collection and subsequent analysis the UPRM TPP anticipates ascertaining how its recent alumni are impacting their students learning and growth. The proposed plan will help the UPRM TPP identify the strengths and weaknesses of the teachers it has prepared within their school ecosystem. Identifying these strengths and weaknesses will permit the UPRM TPP to direct future efforts at better preparing its candidates to meet student academic development needs in Puerto Rico’s classrooms.
During the implementation in 2016-17, the research process itself will undergo periodic formative evaluation to make adjustments that maintain data and data cycle consistency for continuous program improvement over the next seven years. The UPRM TPP will share and discuss the collected information with faculty and stakeholders. Annual faculty data days and Assessor Committee meetings will be held to present the data with an initial analysis in order to make data driven decisions to optimize the program. Suggestions and comments from research participants will be given deliberate attention in the data analysis and in planning improvements.
The in-depth look at alumni performance provided by the proposed research, taking into consideration their school ecosystem is expected to produce and foster future educational research. The proposed research should prove fruitful territory for UPRM to contribute to the knowledge base of higher education reforms. The results of the mixed methods study will be published in peer reviewed journals, presented in professional education conferences, and shared with interested parties on the http://uprm.edu/eppcaep webpage.
The budget includes $68,386.33 for compensation and release time for 8 faculty members. The budget for three student assistants is $8,700.00 and $10,000 for administrative personnel compensation. The budget for marginal benefits for the above personnel is $8,516.54. The budget for training costs including lodging and food is $21,681.53. The budget for resources, materials and equipment is $6,500.00. The budget also includes $1,500.00 for participant stipends. The total budget for the first year 2016-17 comes to a grand total of $125,284.40.