Based on the analysis of the disaggregated data, how have the results of specialty licensure area or SPA evidence been used to inform decision making and improve instruction and candidate learning outcomes?
The Teacher Preparation Program (TPP) and the Agricultural Education Department (AgED) use Puerto Rico’s annual Teacher Certification Test results (PCMAS) to evaluate how well their respective programs are preparing candidates to meet UPRM and Puerto Rico Department of Education (DEPR) teacher requirements, particularly the requirements regarding content and pedagogical knowledge. The Teacher Certification Test (PCMAS) results are submitted to the Puerto Rico Title II reports on teacher preparation programs. The DEPR bases 80% of its rating of institutional teacher preparation programs on five years of Teacher Certification Test results. The latest (2014) UPRM TPP rating by the DEPR was excellent (see 1.1.4a). The next rating will be issued in 2017. The UPRM assessment board uses PCMAS results annually to identify strengths and weaknesses in the existing curricula and to make data driven decisions regarding course content and program requirements. The specific content courses required for each of the specialty licensure areas were revised in response to an analysis of PCMAS content test results from 2011 to 2014 for the methodology and student teaching courses [see http://uprm.edu/p/eppcaep/brochures_ppm for revised curricular requirements by licensure area].
The TPP and AgED also use the Teacher Certification Survey (PCMAS Survey) to assess candidate perception of their respective preparation, programs, courses, faculty, and clinical experience. The Teacher Certification Survey results are included in the annual institutional report sent to UPRM. The report provides aggregated responses for UPRM Teacher Certification Test takers and for all Teacher Certification Test takers in Puerto Rico. The survey has confirmed candidate satisfaction with their preparation, especially with regard to content knowledge and pedagogical content knowledge [see 3.6.1].
In addition to the PCMAS Survey conducted by College Board, the TPP and AgED conduct various surveys. These surveys have proven informative and, more importantly, helpful designing initiatives to improve program operations and quality. The surveys include: End of Program Surveys, Cooperating Teacher Surveys, and Practice Center Principal Surveys. Some results were similar regardless of candidate licensure area. One example of changes made in response to findings from these surveys is the remodeling of physical facilities and integrating more technology in courses. In the 2014 End of Program Survey, candidates made specific recommendations for improving the available technology resources. The alumni survey of 2015 repeated similar concerns regarding technology resources and integrating technology in coursework [see 4.4.3].
The 2014 End of Program Survey and 2015 Alumni Survey paralleled 2014 clinical faculty reports that teacher candidate technology use during teaching practice was limited and weak in terms of instructional effectiveness. Since UPRM teacher candidates had performed relatively well with respect to the use of technology on the basic teacher certification battery (PCMAS – see 1.1.4), this was attributed in large part to the limited availability of computers, projectors, internet connections, and educational technology in the teaching practice center classrooms. Even though such limitations were and continue to be a common problem in the Puerto Rico public school system, the 2015 surveys showed that UPRM teacher candidates were aware of the need to further develop their educational technology skills to respond to student learning needs. Subsequently the TPP remodeled its facilities to provide an additional computer center and to add “smart boards” and data displays to several classrooms.
The TPP used the Resource Center for Investigation and Educational Services (CRUISE in Spanish http://uprm.edu/cruise) and the Resource Center for Distance Education (CREAD in Spanish) to address candidate concerns regarding resources for planning and preparing for their teaching practice classes. CRUISE received substantial external funding from conducting professional development for PRDE teachers in 2013 that has served to sponsor TPP educational research and acquiring classroom technology. CRUISE continues to submit research and professional development proposals and to provide candidates with technology resources that they can use during their teaching practice including laptops, projectors, iPods and SwivlsTM . CREAD (http://cread9.wixsite.com/creaduprm) provides technology assistance and appropriate tools so future teachers can create and use electronic educational modules during their teaching practices and their demonstration classes.
As well as providing more education technology facilities and equipment, the TPP requires teacher candidates to create and maintain a web-based Electronic Portfolio in their methodology and teaching practice courses as well as the “Use of the Micro-computer in the Classroom” course. The latter course, EDPE 3129, requires teacher candidates to create and post a complete lesson in their blog that addresses a standard in their teaching specialty and includes a test as an assessment strategy [see 1.5.1 for assignment rubric and data].
In addition, the TPP has conducted technology integration skills workshops targeting candidates in methodology and teaching practice courses. The TPP is monitoring technology use in methodology and practicum courses to determine if teacher candidate educational technology use and skills improve during the 2016-17 academic year.
The PRDE distributed the new PR Core Content Standards in 2014. The PR Core Content Standards are college and career-ready standards based on Common Core. The Science Standards were aligned with NGSS. Lesson plan requirements, instruments, and rubrics in TPP courses were revised to align with the new PR Core Content Standards. In fall 2014 the TPP faculty decided to change the Classroom Observation Rubric to align with the 2013 InTASC standards, the revised PRDE Professional Standards as well as with the indicators in the 2014 PR Core Content Standards and new PRDE lesson planning guidelines (Carta Curricular 6-2014-2015). After reviewing several instruments and rubrics, the TPP translated and adapted the Utah Valley University School of Education’s “Continuum of Instructional Practice” with their permission. That was the beginning of the new “Observation Instrument for Teaching Practice Improvement” replacing the former “Classroom Observation Instrument”. The translation from English to Spanish was followed by a back translation from Spanish to English, and then it was distributed to clinical faculty and cooperating teachers in each licensure specialty for content validity. A questionnaire was e-mailed to clinical faculty, cooperating teachers, and school directors to collect suggestions and recommendations about the new instrument. Clinical faculty met to align the instrument to the various standards and to UPRM student learning outcomes. After a pilot test during the 2014-15-fall semester, changes were made to clarify the language clarification. During the 2014-15 spring semester, all the TPP specialty areas used the “Observation Instrument for Teaching Practice Improvement”. The new 28-item instrument was universally acclaimed as an improvement over the 72 and 78-item predecessors. The instrument has improved teacher candidate feedback in quantity and quality for all TPP specialty areas. Two years of disaggregated data have shown there are differences in item scoring by area that require discussion and follow up. A clinical faculty “Data Day Retreat” is scheduled for fall 2016 and this issue will be given high priority in order to improve candidate-learning outcomes.
The TPP, the AgED, and licensure area content departments use candidate transcript data including: content and education courses previously taken, grades obtained, overall GPA, and specialized GPA to plan course offerings tailored to each licensure area. The same data are used to counsel students during enrollment. Candidate transcript data allow education and content departments to identify course offerings needed. The courses needed by teacher candidates and the course demand by non-candidates are taken into account for allocating teaching resources. In this regard, early identification of Methodology and the Practice Teaching students is critical in order to recruit the needed adjunct university and clinical faculty.
The departments and faculty regularly use candidate faculty evaluation for decision-making. Student evaluations are taken into account at UPRM for teaching assignments, tenure, and promotion. Faculty use student teaching evaluations to monitor and improve their classroom performance. The departments use aggregated evaluation data to plan faculty (university and clinical) professional development every semester. Candidate evaluations of cooperating teachers are a major factor in subsequent teaching practice placements.
The TPP and the AgED use field and clinical experience supervisory data such as Teaching Practice Center (school site) attendance hours and UPRM teacher preparation seminar and workshop attendance to assure both candidates and clinical faculty meet their responsibilities. Teaching practice grades take school, seminar, and workshop attendance into account.
Candidate statistics and demographics such as content programs, year of curricular sequence enrollment, academic status, number enrolled, retention, and teacher certification exam results are used to complete annual institutional reports that affect the allocation of funds. In the 2015-2016 spring semester, there were 117 teacher candidates and potential teacher candidates with 18 credits or more in Education courses from 39 UPRM undergraduate programs, 10 graduate programs, and 2 non-degree programs.
Based on the analysis of specialty licensure area data, how have individual licensure areas used data for change?
The classroom observation and teacher certification exam data collected showed that UPRM candidates had developed sufficient knowledge, skills, and dispositions to be effective in the classroom. However, comparing the data across specialty licensure areas revealed a weakness in communications skill development among non-language teacher candidates. The analysis of Teacher Candidate Work Sample results showed the same disparity in communication skills across specialty licensure areas. Subsequent discussion brought out that whereas language teacher candidates are routinely required to revise written assignments throughout their academic career, non-language teacher candidates rarely have similar requirements or opportunities. Decision: require all potential teacher candidates to revise poorly written education foundation course assignments and refer candidates needing additional help to the College of Arts & Sciences Writing Center with their written course assignments. The expected summary passing rate is 93% for UPRM candidates in 2016 (pending revision by UPRM and CollegeBoard®).
Historically, the highest score on 1 to 4 of the five Puerto Rico Specialization Teacher Certification exams has been made by a UPRM teacher candidate each year. In March 2016, UPRM teacher candidates made the highest scores on the Mathematics, English, and Spanish Specialization Exams. Last year two UPRM teacher candidates tied for the top score on the Mathematics Specialization exam. Given that UPRM teacher candidates account for 5 – 20% of the test takers for any one specialization exam, this attests to the breadth and depth of UPRM teacher candidate preparation in licensure area content.
With a 100% passing rate on Puerto Rico’s science teacher specialization exam six of the past seven years, UPRM’s science teacher candidates have clearly been well prepared. In Puerto Rico’s high schools, the science teachers are responsible for laboratory safety as well as other aspects of school safety. Given this particular responsibility, the TPP offers a series of modules for future science teachers in the Theory and Methodology of Science Teaching course.
The UPRM Spanish teacher candidates have compiled a 100% passing rate on Puerto Rico’s Spanish teacher specialization exam five of the past seven years including the highest score in 2016. With only 11 candidates over the past three years and an average margin of 40 points above the minimum passing score on the general high school teacher exam have revealed no particular deficiencies. The areas of concern for UPRM Spanish teacher candidates are two of the three that apply to all UPRM candidates: how to develop critical thinking among students and using educational technology.
Over the years, UPRM English teacher candidates have also performed well (90% passing rate) on Puerto Rico’s English teacher specialization exam. However, an analysis of English teacher candidates’ classroom performances raised concerns about the unique requirements of teaching English to Puerto Rico’s native Spanish speakers. In 2011-12, the Perspectives on Teaching English as a Second Language (INGL 5010) became a co-requisite for the Teaching English Methodology course.
The success rates for UPRM candidates on the English teacher certification exam has not changed significantly, from the year 2014 to 2016, twenty-seven of twenty-nine candidates passed the exam on the first attempt, two passed on the second attempt, and two failed on a single attempt. One of those two failures was a UPRM candidate who graduated prior to 2010. The change was made to better prepare candidates to address the needs of Puerto Rico’s students and of mainland students for whom English is a second language, an area of increasing demand.
UPRM mathematics teacher candidates have also performed very well on Puerto Rico’s mathematics teacher specialization exam making the top score seven of the past seven years and 90% approving the exam every year. It was a comparison of the mathematics content required by the UPRM Math Education program with that recommended for teachers by the National Council of Teachers of Mathematics that led to significant program changes. The 139 credit, virtual double major in mathematics and education was revised to include Discrete Mathematics, Mathematics History, the Use of Microcomputers in the Classroom, the Use of Technology in Teaching Mathematics, and the Seminar on the Nature and Needs of Exceptional Children (see http://math.uprm.edu/academic/programs/bachelor-math.html for complete program). Further analysis revealed that some math teacher candidates enrolled in their Teaching Practice before completing a course in Geometry. The TPP now requires 18 credits in specific math courses including Geometry before enrolling in the mathematics methodology course.
UPRM social studies teacher candidates were performing poorly on Puerto Rico’s social studies teacher specialization exam with less than 70% approving the exam from 2010 to 2013. A transcript analysis revealed that successful candidates had taken specific courses that unsuccessful candidates had not. Based on this analysis, the UPRM Social Sciences Department and the TPP established specific content courses in each of the five social science bachelor’s programs to require for enrolling in the social sciences methodology course. In 2014, the approval rate for UPRM social science candidates jumped to 89%. In fact, over the past three years 47 of 56 (84%) UPRM social science and history teacher candidates have passed their certification exam. During that time three UPRM social science teacher alumni took the exam a total of 7 times giving the appearance of lower success rates. The TPP will continue analyzing the results.
There is no specialization exam for Physical Education licensure. However candidates must practice teaching in both high school and elementary school and pass both high school and elementary professional teacher certification exams. To prepare for the unique challenges of teaching Physical Education, since 2014 UPRM candidates develop a biomechanics skill analysis project (EDFI 4106), take two courses in Motor Learning (EDFI 4005 and EDFI 4179) before enrolling in the Physical Education methodology course.
During practice teaching prior to 2014, supervisors and cooperating teachers alike found that teacher candidate technology use during teaching practice was limited and weak in terms of instructional effectiveness. Since UPRM teacher candidates had performed relatively well with respect to the use of technology on the basic teacher certification battery (PCMAS – see 1.1.4), this was attributed in large part to the limited availability of computers, projectors, internet connections, and educational technology in the teaching practice center classrooms. Even though such limitations are a common problem in the Puerto Rico public school system, UPRM teacher candidates need to develop skills in the use of educational technology as well as the disposition to learn how to use technology as it develops to respond to student learning needs. One of the ways the UPRM Teacher Preparation Program has addressed the development of educational technology skills by its teacher candidates is by requiring them to create and maintain a web-based Electronic Portfolio in the methodology and teaching practice courses as well as the “Use of the Micro-computer in the Classroom” course. The latter course, EDPE 3129, requires teacher candidates to create and post a complete lesson in their blog that addresses a standard in their teaching specialty and includes a test as an assessment strategy (see 1.5.X for assignment rubric). With respect to the lack of technology in teaching practice centers, the TTP has acquired laptops, projectors, SwivlsTM and iPods. These are loaned to candidates to record classes during their teaching practice. The TTP is conducting technology integration skills workshops targeting candidates in methodology and teaching practice courses. The Resource for Research and Educational University Center (CRUISE) and the Resource Center for Online Education (CREAD) have partnered to acquire educational equipment and to train assistants to help teacher candidates prepare technology enhanced lessons since 2015. The TPP is monitoring technology use to determine if teacher candidate educational technology use and skills improve during the 2016-17 academic year.
For Program Review with Feedback only: How does the specialty licensure area data align with and provide evidence for meeting the state-selected standards?
The Puerto Rico Teacher Certification Specialization exam (PCMAS) results are an important source of specialty licensure area data. Specialization exams are required for licensure to teach Science, English, Spanish, Social Studies/History and Mathematics. The content of these specialization exams is based on an analysis of the curricular requirements at Puerto Rico’s higher education institutions and the definition of a competent teacher in the material to be taught as expressed in official PRDE documents (translated from the Boletín Especialidad PCMAS -https://latam.collegeboard.org/slides/slide/boletin-especialidades-pcmas-7). The CollegeBoard® Bulletin includes an outline of the topics to be addressed in each specialization exam. In 2015, the Puerto Rico Department of Education worked jointly with CollegeBoard® to re-conceptualize and revise the Teacher Certification Exams so that the exams would serve the needs of Puerto Rico’s PK-12 students and to evaluate all of the competences that a teacher candidate must develop to be effective in the classroom (see 1.1.4b). By design, the PCMAS is consistent with the Puerto Rico Professional Standards for Teachers (PR-PST see http://www.de.gobierno.pr/files/Estandares_Profesionales_de_los_Maestros_de_Puerto_Rico_2008.pdf or 1.1.1c for translation). Hence exam results provide the UPRM TPP with direct measures of how well its teacher candidates are meeting the PR-PST. Each of the specialization exam results provides score breakdowns in 3 to 6 sub-areas that allow the UPRM TPP to detect content and pedagogical topics that may need attention in a particular licensure area.
The UPRM TPP’s shared values and beliefs framework is consistent with the Puerto Rico Department of Education professional standards, as well as with institutional Student Learning Outcomes, national standards (CAEP), and performance standards (InTASC 2013). Based on current teaching-learning theory, validated neuroscience learning research, and wisdom of practice; the UPRM TPP established ten teacher candidate proficiencies that align with the standards mentioned [see alignments in 1.1.1a and 1.1.1b]. In order to assure that its candidates develop the knowledge, the skills, and the dispositions to be effective teachers in the PR school system; UPRM TPP teacher candidates’ development of these ten proficiencies is measured throughout the program particularly at transition points (see URPM TPP Assessment System Procedures guide in 5.1.1) with key course grades and key rubrics. Two key rubrics are the Observation Instrument for Teaching Practice Improvement and the Teacher Candidate Work Sample. These rubrics address all ten UPRM TPP proficiencies and consequently the eleven standards in the PR-PST. Every education course goal and objective is aligned to the ten candidate proficiencies which in turn are aligned to PR-PST as well as all the aforementioned set of institutional and national standards. Each education course addresses one or more of the ten UPRM TPP proficiencies (see http://uprm.edu/p/eppcaep/syllabi for syllabi details).
Data derived from PCMAS results, Classroom Observation instruments, electronic portfolios, content course credits and grades are disaggregated by licensure area and analyzed against the candidate proficiencies to determine what aspects need attention. The results of this analysis are compared with aggregated teacher candidate data derived from TCWS, education course instruments, and education course grades to uncover patterns as well as possible courses of action to address areas of concern.
* Program Review Option (per state partnership agreement): CAEP Program Review with Feedback (State-selected standards)