In June 2016, the Puerto Rico Department of Education (PRDE) discussed their 2015-2016 teacher effectiveness pilot study and their plans to scale up the study with a group of UPR Teacher Preparation Program representatives [4.1.1]. The pilot study tested the evaluation instruments and methodology with 20% of the PRDE teaching force. The 2016-2017 scaled up study promises to collect direct measures of student learning and development addressing diverse subjects and grades with P-12 impact and growth data. Despite personal and formal requests [4.1.2], the UPRM TPP has yet to receive a copy of the federal report based on that study. At the time of submitting the UPRM Self Study the PRDE has not finished the proposed full-scale study. The UPRM TPP will use whatever information from the pilot study or the subsequent scale up that the PRDE makes available.
In the absence of state data regarding teacher effectiveness, the UPRM TPP has developed a Mixed Methods Research Plan [4.1.3] to collect the type of information needed to analyze the program’s strengths and weaknesses in terms of the impact of the in-service teachers. The Mixed Methods Research Plan will use qualitative and quantitative methodology to answer a list of central research questions addressing aspects of alumni performance quality as it pertains to national InTASC and state PRDE professional standards. Data will be collected from key school stakeholders that comprise the UPRM alumni-teaching environment: the alumnus, the alumnus’ students, and the alumnus’ immediate supervisor.
The case study will address seven principal research questions:
1. What is the impact of UPRM TPP preparation on alumni’s P-12 students learning and development?
2. How does UPRM TPP alumni classroom instruction align with professional standards such as InTASC and PRDE Teacher Professional Standards?
3. How satisfied are UPRM TPP alumni with the relevance and effectiveness of their preparation?
4. How satisfied are the employers with the relevance and effectiveness of UPRM TPP alumni and their preparation?
5. How satisfied are K-12 students with the classroom instruction they receive from UPRM alumni teachers?
6. What impact are UPRM alumni teachers having on the academic development of their P-12 students?
7. Do P-12 students perceive their UPRM alumni teachers as fair, ethical, challenging, and helpful?
The qualitative case study methodology part of the plan will use multiple means to collect information from 8 to 10 alumni cases as a way to explore process tracing that links causes and outcomes while allowing data triangulation. The qualitative data will provide opportunities to formulate hypothesis and generate new questions to study alumni classroom effectiveness with respect to applying the skills, knowledge, and dispositions that the UPRM TPP programs sought to provide.
The Danielson Framework Teacher Observation Rubric [in 4.1.3 pages 20-50] and the Alumni [4.4.1] and Employer Interview [4.3.1] protocols will be piloted with the participants as part of the case study. These instruments and protocols were selected for their proven validity and reliability in teacher evaluation projects like the MET study and will be adapted for future research. The Flowers and Hancock’s Alumni Interview Protocol will be adapted for employer interviews to maintain consistency in the questions needed for triangulation.
UPRM TTP faculty members will be trained to conduct interviews and classroom observations that are as free as possible from bias and to assure inter-rater reliability. The classroom observations will be correlated with alumni self-reports and student achievement measures versus learning goals.
As a means to complement the information gathered through case studies, two TPP created surveys (alumni and employer) will be used both with the case study population and a larger population. The surveys were created using pertinent information collected from previous surveys, but redesigned to align more closely with InTASC and PRDE professional standards, as well as CAEP standards. Both surveys were presented to the UPRM CAEP Steering Committee to verify content validity and alignment with professional and CAEP standards. Recommendations gleaned from CAEP’s webinars about using surveys were also considered for both. After incorporating the recommendations from committee members regarding format and content accuracy, pilot tests were conducted to verify the clarity of multiple choice items and open questions.
Validated item versions of the elementary and secondary Tripod Student Surveys [4.2.2] will be used to gather data from alumni students, depending on the grade level taught by the alumnus. At least two different groups of the alumni teacher’s students will be asked to complete a Tripod Student Survey. The validity and reliability of the Tripod Student Surveys are documented in Asking Students about Teaching: Student Perception Surveys and Their Implementation, MET Project Policy and Practice Brief by the Bill & Melinda Gates Foundation, 2012.
To maximize the utility of information from small samples in case studies an “information – oriented selection” approach with a maximum variation of cases is recommended in order to augment the possibilities of obtaining information about the significance of various circumstances for process tracing and outcomes. Following this recommendation, the Plan will include four schools that have at least 2 alumni teachers each. The schools selected will vary in the following aspects 1) zone: urban, rural; 2) type: public, private; and 3) level: elementary school, middle school, high school. The alumni teachers in the study will represent the majority of the UPRM licensure areas: Agricultural Education, Art, Biology, General Science, Business Education, Physical Education, Spanish, Social Studies, Physics, History, English, Mathematics, and Chemistry. A range of 8 to 10 teachers will be the target.
At least two different groups of each alumni teacher’s students will be included in the study as well as their school directors to produce a more comprehensive view of the alumni school ecosystem and to triangulate data.
The TPP created alumni survey will be e-mailed to all UPRM TPP alumni completing teaching practice over the five years prior to the last (2012-2016). In Puerto Rico, candidates who take and pass the teacher certification exams in early March are certified by the Puerto Rico Department of Education in November, are then allowed to apply for teaching positions in January, and may be hired for the following school year some fourteen months after graduating. Hence few, if any, UPRM TPP alumni completing teaching practice in the previous academic year will have a teaching position with the PRDE.
The first round of employer surveys will be e-mailed to school directors at UPRM Teacher Preparation Practice centers. A second round of employer surveys will be e-mailed to directors at schools identified in responses from working alumni. The surveys will be administered using the Qualtrics© online survey system. If survey response rates are less than 20%, survey copies will be printed, delivered in persona.
The direct measures in the aforementioned approach will include observing the alumni in the classroom and an alumni prepared analysis of student learning using pre-post teacher created tests aligned to content standards. The indirect measures will include alumni interviews and surveys; interviews and surveys of alumni immediate supervisors (school directors); and alumni student surveys. All the instruments and protocols have been prepared in both English and Spanish so that participants may use the version of their preference.
A three-day workshop retreat will be held during the 2016-2017 fall semester to train faculty to use the ATWS rubrics, surveys, observation instruments, and interview protocols in a professional manner. The Chalk and Wire® online data collection system will be used to record data and to generate reports from the ATWS rubric, the Danielson’s Observation Rubric, and the Flowers & Hancock’s interview rubrics. The Qualtrics® online survey system will be used to collect and organize data gathered from the Tripod Student Survey, the employer, and alumni TPP surveys as well as to generate reports from same.
The Mixed Method Research Plan outlined above has been submitted to the UPRM IRB. The Plan includes participant and parental consent forms for all parties.
Information obtained from direct observations, teaching-learning artifacts and teacher self-reports as well as data from alumni students and employers will be compared, correlated, and triangulated to form a comprehensive picture of the teachers’ effectiveness with respect to student learning. The proposed analysis will include comparisons among alumni in diverse licensure areas as well as by school type. With the proposed mixed methods data collection and subsequent analysis the UPRM TPP anticipates ascertaining how its recent alumni are impacting their students learning and growth. The proposed plan will help the UPRM TPP identify the strengths and weaknesses of the teachers it prepares within their school ecosystem. Identifying these strengths and weaknesses will permit the UPRM TPP to direct future efforts at better preparing its candidates to meet student academic development needs in Puerto Rico’s classrooms.
The research process itself will undergo periodic formative evaluation to make adjustments that maintain the consistent data and data cycles needed for continuous program improvement over the next seven years. The UPRM TPP will share and discuss the collected information with faculty and stakeholders. Annual faculty data days and Assessor Committee meetings will be held to present the data with an initial analysis in order to make data driven decisions to optimize the program. Suggestions and comments from research participants will be given deliberate attention in the data analysis, faculty data days, and in planning improvements.
The in-depth look at alumni performance provided by the proposed research, taking into consideration their school ecosystem, is expected to produce and foster future educational research. The proposed research should prove fruitful territory for UPRM to contribute to the knowledge base of higher education reforms. The results of the mixed methods study will be published in peer-reviewed journals, presented in professional education conferences, and shared with interested parties on the http://uprm.edu/eppcaep webpage.
The UPRM TPP conducted pilot studies on employer and completer satisfaction surveys in May 2015. The data obtained from both studies served to suggest changes in the structure, instructions and purpose, items unambiguity, language clarity, and a better alignment with the national (InTASC) and state standards (PRDE-TPS). Specific questions on promotion, retention and specialty area were added to the revised and improved surveys [4.3.1]. The changes in the revised versions are directed at improving their validity and reliability as well as allowing the disaggregation by specialty area. Despite multiple emails to school directors to encourage response, the past Employer Satisfaction Survey had a very low response rate, only three school directors answered it [4.3.3]. The collected data was insufficient to make valid generalizations. The results were used primarily to pilot the survey structure and questions and to test the Qualtrics© online data collection process. However analysis of the data collected as well as interpretations for future actions are presented in detail in 4.3.3 and 4.4.3.